How to save the mempool information immediately before the node receives a block?

Question

I want to use Bitcoin RPC API 'getrawmempool' to save the snapshot of the mempool immediately before my full node receives a block. This requires modification of the Bitcoin source code. How should I modify it? Who can help me very kindly?

Answer 1

You can use zmqpubsequence to get updates of all transactions added/removed to/from mempool and blocks received.

Example:

Save zmqpubsequence=tcp://127.0.0.1:28332 in bitcoin.conf

Run below python code:

import binascii
import asyncio
import zmq
import zmq.asyncio
import signal
import struct
import sys

if (sys.version_info.major, sys.version_info.minor) < (3, 5):
    print("This example only works with Python 3.5 and greater")
    sys.exit(1)

port = 28332

class ZMQHandler():
    def __init__(self):
        self.loop = asyncio.get_event_loop()
        self.zmqContext = zmq.asyncio.Context()

        self.zmqSubSocket = self.zmqContext.socket(zmq.SUB)
        self.zmqSubSocket.setsockopt(zmq.RCVHWM, 0)
        self.zmqSubSocket.setsockopt_string(zmq.SUBSCRIBE, "sequence")
        self.zmqSubSocket.connect("tcp://127.0.0.1:%i" % port)

    async def handle(self) :
        topic, body, seq = await self.zmqSubSocket.recv_multipart()
        sequence = "Unknown"
        if topic == b"sequence":
            hash = binascii.hexlify(body[:32])
            label = chr(body[32])
            mempool_sequence = None if len(body) != 32+1+8 else struct.unpack("<Q", body[32+1:])[0]
            print('- SEQUENCE ('+sequence+') -')
            print(hash, label, mempool_sequence)
        # schedule ourselves to receive the next message
        asyncio.ensure_future(self.handle())

    def start(self):
        self.loop.add_signal_handler(signal.SIGINT, self.stop)
        self.loop.create_task(self.handle())
        self.loop.run_forever()

    def stop(self):
        self.loop.stop()
        self.zmqContext.destroy()

daemon = ZMQHandler()
daemon.start()

Source: https://github.com/bitcoin/bitcoin/blob/master/contrib/zmq/zmq_sub.py

python3 zmqtest.py

b'0000000000000020dd401c162d8514e7cc13a2aa123fbdaab05e117defe4fa45' C None
- SEQUENCE (Unknown) -
b'034c21d4ea43da1640baeed6a5b5da7d9581f946187840326a55a3fff652f896' A 1
- SEQUENCE (Unknown) -
b'507ec9296156c3925c6e2852284eeb07838480202d80eeb0718fc271e5c5a7e6' A 2
- SEQUENCE (Unknown) -
b'8da15cc34a0a16ef4160648a387afb98a212f4d93844c0b837ce156f3d40b70e' A 3
- SEQUENCE (Unknown) -
b'47134355caf0539bf1bb3c9496eb212318ed64f26bf8f7654026a7254fe8a3b1' A 4
- SEQUENCE (Unknown) -
b'044a56e511b1ff097c528cee053c8b4d0b97fe4237308f00d9714eca99ed7d58' A 5
- SEQUENCE (Unknown) -
b'a6edd6322b02e95a10890a811c6d0a8899d30874c8e2b02a017d785ee4ee5c80' A 6
- SEQUENCE (Unknown) -
b'4790472baba9c5860805fa3258d85d50bfa607f1ce8437ef2c4a3c6b503a8b86' A 7
- SEQUENCE (Unknown) -
b'ca82da8ee0e982995f7dd0a98580c102ae4adc4f8edf29ca64b98b1f3f1be751' A 8
- SEQUENCE (Unknown) -
b'1bcde3da8835980b8b884ba58bd36884fb5265fa6a06aabaabc8cf7517ce58ee' A 9
- SEQUENCE (Unknown) -
b'5a0d816ff22037c04d10597496c05bce9e72c6b379e33e24196902c64e2429dc' A 10
- SEQUENCE (Unknown) -
b'd0ddd43dafefb4230a3f71bd026bf7cf5e740d23163f95b809620fefeca9b0eb' A 11
- SEQUENCE (Unknown) -
b'a186fd5560d6def8eace6e9aa4a370c49cfcdbfb7883988697e9d008d1c81f73' A 12
- SEQUENCE (Unknown) -
b'40f790b2823b55ef21544f284e3381dcccd387c44512a2518edc0d64b7c78ff8' A 13
- SEQUENCE (Unknown) -
b'24fef1ea3a3058ec23015ae01788a9386fd937646dcc0ebc30d4078d4fd0ab83' A 14
- SEQUENCE (Unknown) -
b'245661a7e1a099b636541e80933720040b947efd8d63e96bae6a627ed5024d31' A 15
- SEQUENCE (Unknown) -
b'200e1379db62a299c4df69b2543ac9acc538cddd52bbc2000a40878a246ad5d2' A 16
- SEQUENCE (Unknown) -
b'37d8b02e67d30583ebfc41861c64a8bf0e10daec25366976ed002f8fa76e8300' A 17
- SEQUENCE (Unknown) -
b'677f74c4412d317921528809e983ccd57d1d387760614a99606d1d651bbdcb76' A 18
- SEQUENCE (Unknown) -
b'5c490f86c11c372d2cfd395020eb2a3726970f64829c7254f99520c8e9e9fa02' A 19
- SEQUENCE (Unknown) -
b'a176e21826a217cda3d3ef59393a9ba293ab7ca238f17b3a945a0928e172221d' A 20
- SEQUENCE (Unknown) -
b'7bc33a81ee5c5cd5083ef11e6a9d0d50fbe081fad0bf9d2f4c980f228a32f5a7' A 21
- SEQUENCE (Unknown) -
b'18f5d3c96a66a75d20f8f19c4a1d1eff28488610d8ac1241865e5d7b7a94bfbc' A 22
- SEQUENCE (Unknown) -
b'd14f474c19d0814c23f742f5bf7372dda29f872c072bbb7e25e97409020e4299' A 23
- SEQUENCE (Unknown) -
b'00000000000000046fbd9d36c40562094bf0049ead63e5311ac515123baf5789' C None

How to save the mempool information immediately before the node receives a block?

In the above results, Block 00000000000000046fbd9d36c40562094bf0049ead63e5311ac515123baf5789 was received and 23 transactions were added to mempool before it.

If a transaction is removed from mempool, python script will print:

b'TX_ID' R SEQ_NUMBER

Hope this helps although its not necessary that transactions between 2 blocks printed by the zmq python script will be same as getrawmempool results at that moment. The script will only print additiion and removal of transactions in sequence.