Are there any significant hurdles preventing us from using multiple penalty transactions, where the nth penalty transaction spends HTLCs [n*483; (n+1)*483[ from the commitment transaction? At first sight, it seems like quite a straightforward upgrade.
If the only goal is to increase the maximum number of HTLC outputs, the above solution doesn't have any hurdles of which I'm aware. I believe it's something LN devs have discussed before, for example in this post I believe it's what's being called "commitment fan out".
However, it's probably worth noting that most LN implementations currently limit the maximum number of pending HTLCs well below 483 for various reasons, such as:
- LN fee ransom attack, leading CLN and Eclair to lower their max HTLCs to 30. See also the PR where LND made this configurable.
- Mass closure attacks (described in the original LN paper) where Mallory can get a large number of channels to all be forced closed at the same time, making them compete against each other for block space, raising fees massively and possibly leading to some channels not closing on time. More HTLC outputs per channel means (in the worst case) more data per channel so fewer channels need to be forced closed for the attack to become viable.
- Channel jamming attacks, which isn't made relatively worse by having more HTLC outputs available, but just makes the absolute number of blocked HTLCs higher
Another consideration is that it's possible for two peers to have more than 483 HTLC slots between them by simply opening a second channel between themselves. When forwarding a payment, they can then use whichever channel they prefer, not necessarily the one requested by the payment creator (e.g. they can use a different one of their channels for liquidity reasons). This has most of the advantages of the nested-HTLCs you propose (and all of the downsides described here previously) and doesn't require any changes to the protocol.
In short, I think the approach you describe is possible but there isn't high demand for more than 483 slots per channel, and what demand exists is being met using alternative approaches.