Automatic force closure and on chain settlement of expired HTLC

Question

After roughly a month of mostly smooth operation a lightning node (LND 0.16.2-beta) automatically initiated a force closure of a channel. Two days later an expired HTLC was settled on-chain. In both instances, high fees (~2-3 times the going rate) were automatically paid from the on-chain wallet of the node. Everything happened without any user intervention whatsoever. Here are the two transactions:

https://mempool.space/tx/71c18287c075bf3eb9a5083b98701faa87071266e5f0fe33577885fbb1180901 https://mempool.space/tx/2730193b26c2c2be70d20bb1e3f0e0d4eefed5283086320ae2f361af0f58aadf

The two transactions seem to be related in the sense that the expired HTLC was caused by the force closure.

So, I'm trying to find out what caused the force closure. As far as I know the lightning network protocol is designed to have the bad actors bear the cost of their behavior and make everyone else whole, so I suspect that the node was misbehaving somehow?

• Could it be a connection issue (the node is only reachable via TOR)?
• How can the high fees for force closure/expired HTLC be avoided in the future, is this configurable anywhere?

If you need more info (logs, etc.), I'd be happy to ask the operator to provide them.

Answer 1

According to https://massmux.substack.com/p/why-i-see-many-force-close-on-my force closes would happen automatically when there is an expired HTLC:

Most forced close are due to expired HTLC. A transaction tries to go through your node, ends up in pending status, your node goes offline, after the HTLC expires (please see below the default hours for each implementation) the channel goes into force close. More precisely:

LND has by default 40

CLN has by default 34

Eclair by default 144

so I believe you have this theory:

The two transactions seem to be related in the sense that the expired HTLC was caused by the force closure.

backwards, the expired HTLC likely caused the automatic force closure as per the default settings in the relevant lightning implementation.

HTLC relay failure leading to node downtime seems to be the cause, but it is hard to say what specifically about your node setup and channels may have been vulnerable to this situation.

Also like you mentioned, it is possible the remote channel was misbehaving in some way. This is why until you can be certain that you are avoiding all related vulnerabilities you should not open channels with lots of unknown people especially not with questionable reputation.

Edit:

Also relevant from documentation: Source: HTLC Consolidated

Answer 2

It appears that there is an issue in LND 0.16.2:

https://github.com/lightningnetwork/lnd/issues/7682

It should be fixed in 0.16.3.

BTW, pro tip for routing node operators running LND 0.16.2: Regularly (e.g. once a day) look at your pending HTLCs. If one will expire before your next check, disconnect the peer of the affected channel with lncli disconnect. This should prevent expiry, see this issue for more info.