Bitcoind not publishing blocks via ZMQ after reorganization

Question

On the testnet, I am running a bitcoin core deamon (bitcoind) and am using ZMQ to receive updates about transactions and blocks (subscribed to both 'rawblock' and 'rawtx').

I was curious as to how the ZMQ would handle chain-reorganizations, so I left it running overnight, logging the hash of every block I am notified about.

The next morning, using the getchaintips RPC, I can see that a fork did occur, as seen below:

[
  {
    "height": 1664502,
    "hash": "00000000000000315e8e11c0babc400475f3a9bfc21678f32ca8f0f878f7e807",
    "branchlen": 0,
    "status": "active"
  },
  {
    "height": 1664491,
    "hash": "00000000002d324cbfe1261d506f85e9baf61d30e762e342f4f44429d4d6e1fb",
    "branchlen": 1,
    "status": "valid-fork"
  }
]

According to my own logs however, I only received one block at height 1664491, which was the (now stale) block "...d4d6e1fb", as seen below:

Received block 000000000001a2a613d4c66e78233f4d9fbfcf8e08502507c39c95460120cbda at height 1664490
Received block 00000000002d324cbfe1261d506f85e9baf61d30e762e342f4f44429d4d6e1fb at height 1664491
Received block 000000000000001b8f4e6ba531411eae489a6dc5e9c9c07ef9e2288f3f623014 at height 1664492
Received block 00000000000000436dfb1f1bb687bca8b905a994bcaece65ede27170fd453462 at height 1664493
Received block 00000000002753e7d21158cb6bcb8093779c7a3e7868762161edf7bedf50ce3c at height 1664494
Received block 00000000000000f18a226b32ae8399a3faf3a4c4155e153e7067f02e31c02385 at height 1664495
Received block 00000000000000ff309ae5442d7e5a271fb22cd7cac97d95d8a47a3d9d135c18 at height 1664496
Received block 000000000000606889000756962c02619a11158559d6f060daf68cef25cd0726 at height 1664497
Received block 000000000016c716d24a9dbd115807be235bf74e4fd4dec33d2e84a55a3c3a8a at height 1664498
Received block 000000000000012f5a3553526f751cb385a630cad7c285ba84a5468e8085213a at height 1664499
Received block 0000000000000119e1f42b0d345c59eb9bbf7dc9ed7591b3716ebbc26a9930b9 at height 1664500
Received block 00000000000000022b963361b5b33dcc733f1b9a20adff75336ce48d29be2227 at height 1664501

Curious, I looked up the hash of the now-correct block at height 1664491, which is actually 000000000001d39f8f818382632fd4c4929a34da542103d37ebfba85980d6496.

Using the RPC command getblock, I can get the information about this block from my bitcoind daemon, which means that it knows about it. However, I was never notified about this block through ZMQ, which is somewhat concerning.

Is this expected behaviour? During a chain-reorganization, can I only expect ZMQ to publish the tip of the new chain, leaving me to figure out that a reorganization has occurred and then having to manually retrieve the new blocks?

Is this behaviour documented somewhere? I am not entirely confident in relying on my own experimentation, to figure out how to handle this.

Answer 1

Is this expected behaviour? During a chain-reorganization, can I only expect ZMQ to publish the tip of the new chain

Yes, a ZMQ notification is emitted any time the active tip of the chain changes.

If you have a chain B1-B2-B3, and there is a 1-deep reorg that rewrites B3, you end up with B1-B2-B3'-B4. The tip changes here from B3 to B4; it does not go back to B2 in between.

leaving me to figure out that a reorganization has occurred and then having to manually retrieve the new blocks?

You have to do that anyway. ZMQ is not a reliable protocol, so you must be able to recover from missing messages (and in general, with a node going down temporarily). Dealing with this means discovering that the latest tip is a reorganization compared to what you had before.

This is true in general for ZMQ, you can use it as a notification mechanism to perform an update right when it happens, but you need infrastructure for fetching to recover too.

For more information, see the ZMQ documentation for Bitcoin Core at https://github.com/bitcoin/bitcoin/blob/master/doc/zmq.md.