c-lightning: single tx multi-channel open stuck on CHANNELD_AWAITING_LOCKIN:We've confirmed funding, they haven't yet

Question

I've tried to open multiple Lightning Network channels using a single Bitcoin transaction, with these steps:

1. For every peer in PEERS (n=25) run:

lightning-cli connect ${peer}
lightning-cli fundchannel_start ${peer.id} ${amount}

2. Save all returned ADDRESSES
3. Construct a Bitcoin Transaction spending ${amount} to each of returned ${ADDRESSES}:

bitcoin-cli createrawtransaction …
bitcoin-cli signrawtransactionwithkey …
bitcoin-cli sendrawtransaction …

4. Note returned txid
5. Complete all channels with:

lightning-cli fundchannel_compete ${peer.id} ${txid} ${corresponding vout}

6. Wait for channels to open, which never happens…

My regnet tests worked, but on mainnet things got stuck…

Relevant transaction.

(Redacted) output of lightning-cli listpeers for involved peers.

Among all of the channels that got stuck, there's one where both ends are under my control (c-lightning <----> lnd). I'm able to test things on it:

$ lightning-cli listpeers 03935a378993d0b55056801b11957aaecb9f85f34b64245f864c22a2d25001de74 | jq
{
  "peers": [
    {
      "id": "03935a378993d0b55056801b11957aaecb9f85f34b64245f864c22a2d25001de74",
      "connected": true,
      "netaddr": [
        "203.150.177.97:52226"
      ],
      "globalfeatures": "",
      "localfeatures": "81",
      "channels": [
        {
          "state": "CHANNELD_AWAITING_LOCKIN",
          "scratch_txid": "873bbef47e5c6dda481d62cf5b67d83170448a81e770162aedf1437d6ce16ef9",
          "owner": "lightning_channeld",
          "short_channel_id": "589813x1972x24",
          "direction": 0,
          "channel_id": "2197bec830e4c2124cc78d96f882c2f8e216c5aebf66002d0d1b9d5a833a6406",
          "funding_txid": "1e643a835a9d1b0d2d0066bfaec516e2f8c282f8968dc74c12c2e430c8be9721",
          "private": false,
          "funding_allocation_msat": {
            "03935a378993d0b55056801b11957aaecb9f85f34b64245f864c22a2d25001de74": 0,
            "032260c3b64b471b7eb0630b4af5d07ca94ff4e759573cbbe1bfb25845c375ed6e": 309735000
          },
          "funding_msat": {
            "03935a378993d0b55056801b11957aaecb9f85f34b64245f864c22a2d25001de74": "0msat",
            "032260c3b64b471b7eb0630b4af5d07ca94ff4e759573cbbe1bfb25845c375ed6e": "309735000msat"
          },
          "msatoshi_to_us": 309735000,
          "to_us_msat": "309735000msat",
          "msatoshi_to_us_min": 309735000,
          "min_to_us_msat": "309735000msat",
          "msatoshi_to_us_max": 309735000,
          "max_to_us_msat": "309735000msat",
          "msatoshi_total": 309735000,
          "total_msat": "309735000msat",
          "dust_limit_satoshis": 546,
          "dust_limit_msat": "546000msat",
          "max_htlc_value_in_flight_msat": 18446744073709552000,
          "max_total_htlc_in_msat": "18446744073709551615msat",
          "their_channel_reserve_satoshis": 3098,
          "their_reserve_msat": "3098000msat",
          "our_channel_reserve_satoshis": 3097,
          "our_reserve_msat": "3097000msat",
          "spendable_msatoshi": 300877000,
          "spendable_msat": "300877000msat",
          "htlc_minimum_msat": 0,
          "minimum_htlc_in_msat": "0msat",
          "their_to_self_delay": 144,
          "our_to_self_delay": 144,
          "max_accepted_htlcs": 483,
          "status": [
            "CHANNELD_AWAITING_LOCKIN:Reconnected, and reestablished.",
            "CHANNELD_AWAITING_LOCKIN:We've confirmed funding, they haven't yet."
          ],
          "in_payments_offered": 0,
          "in_msatoshi_offered": 0,
          "in_offered_msat": "0msat",
          "in_payments_fulfilled": 0,
          "in_msatoshi_fulfilled": 0,
          "in_fulfilled_msat": "0msat",
          "out_payments_offered": 0,
          "out_msatoshi_offered": 0,
          "out_offered_msat": "0msat",
          "out_payments_fulfilled": 0,
          "out_msatoshi_fulfilled": 0,
          "out_fulfilled_msat": "0msat",
          "htlcs": []
        }
      ]
    }
  ]
}

It seems like, for whatever reason, other peers have failed to confirm funding.

And the question obviously is: how does one resolve it? :) Any help appreciated!

edit: Attempts to force-close channels result with this (full-er log):

sendrawtx exit 26, gave error code: -26?error message:?non-mandatory-script-verify-flag (Witness program hash mismatch) (code 64)?

tl;dr: Tried opening multiple channels using a single btc tx. Channels got stuck. What do?

Answer 1

I wrote the fundchannel_start/fundchannel_complete pathway. You should not broadcast the signed funding transaction until after you have successfully called fundchannel_complete for each of the peers. This is because fundchannel_complete secures the commitment transactions that are necessary to spend the funding transaction output for that channel. By broadcasting the transaction before you've 'completed' the channel open, there's no guarantee that you'll be able to spend those funds.

If you get an error in any of the funchannel_completes then your entire funding transaction will have to be discarded/cancelled. There's not good support at the moment for calling fundchannel_cancel after fundchannel_complete has been called; this needs to be fixed. There's an open issue for it on c-lightning's Github repo.

It's highly likely that the issue you're seeing with the LND/c-lightning interop is related to the fact that you published the funding transaction before notifying them about it. You might try re-scanning the blockchain with the LND node and see if it resolves the error.