For norm ETFs, it is made up of a basket of stocks. In the case of bonds, is that how bonds ETFs are like as well? If that is the case, isn't it problematic because bonds are constantly maturing?

Yes, it is formed by different bonds who have different maturity dates.

A more direct answer to whether its problematic is "yes, but not because bonds are constantly maturing." The real issue is balancing the return and the duration of a bond portfolio, that is, the sensitvity of the portfolio to interest rate cahnges. The use of different maturities is necessary to acheive the desired yield of the portfolio while maintaining an palatable duration