Is it only because LTC doesn't have as many users as BTC? Or do the faster confirmation times also help with lowering fees?
Litecoin's blockspace supply is greater:
Litecoin's blockspace demand is much lower:
Bitcoin is operating with demand exhausting the supply continuously, thus finding actually a price for blockspace. In the past months we've seen feerates move between 1 and 540 satoshi/byte. Now that the latest hype has cooled down slightly, we can observe a daily and weekly cycle, under the week going from ~8 to ~400 satoshi/byte, dropping down to as low as 2 satoshi/byte on weekends.
On the other hand, Litecoin is operating with demand at approximately 1% of the supply. Miners appear to be observing a hard coded minimum fee of 100 litoshi/B.