Synchronize mempool between 3 nodes

Question

Three bitcore nodes (version of bitcoind is quite old - 0.12.1) are deployed to the servers on different cloud platforms.

All instances have the same bitcoin.conf files (except of rpcallowip, rpcuser and rpcpassword):

server=1
whitelist=127.0.0.1

txindex=1
addressindex=1
timestampindex=1
spentindex=1

zmqpubrawtx=tcp://127.0.0.1:28332
zmqpubhashblock=tcp://127.0.0.1:28332

rpcthreads=8
rpcworkqueue=32
rpcservertimeout=60
rpcallowip=127.0.0.1
rpcuser=...
rpcpassword=...
uacomment=bitcore

The problem is all of them have considerably different mempools:

• first has 33689 txs in mempool
• second has 58219 txs
• third node has 39540 txs
• blockchain.info has 63598 txs

I understand, that mempool is node-specific and there is no "Single Bitcoin Network Mempool". However I need synchronization at least between these three nodes (though it would be better to be as close to blockchain.info as possible).

If there is transaction, that exists in blokchain.info's mempool, but doesn't exist in all three nodes' mempools - it is ok. But as soon as this transaction get to one of these three nodes, it should appear on others as soon as possible.

I've tried using addnode option to force nodes connect to each other, but it didn't helped.

May be I am missing something, or at least there is a way to find the reason of so big differences in mempool ?

---

Aim of consistent mempool.

We have mobile client and application server, all looking into these nodes (some kind of load balancing). User can be logged into one account on different devices. When server sees transaction in mempool, it should update user's status.

When mempools differ it is possible, that:

1. User will see different transactions on different devices, which may mess him
2. User will see transaction, and server will miss it. In such situation user expects, that status will be updated, but server wouldn't do it

So it would be very nice, to have (almost) equal mempools.

---

So far.

I've tried:

• addnode (to connect to specific nodes)
• connect (to connect only to specific nodes), to give it a try, thanks to @whatever

This options were described here, on bitcoin.se, and (alongside with other possible options for bitcoind) in bitcoin wiki. But there is still problem with big difference in mempool's size.

Answer 1

You can force mempool synchronization by running a script which connects to the RPC ports, fetches transactions from one node's mempool, and sends it to the other nodes. Note that because this uses RPC, it will be fairly slow and will still result in some mempool desynchronization. You will probably want to run this script periodically. Here is an example in python using python-bitcoinrpc.

Note: I have not tested this

from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException

# rpc_user and rpc_password are set in the bitcoin.conf file
nodes = []
NODE1_URL = "10.10.10.1:8332"
NODE2_URL = "10.10.10.2:8332"
NODE3_URL = "10.10.10.3:8332"
nodes.append(AuthServiceProxy("http://%s:%s@%s"%(rpc_user, rpc_password, NODE1_URL)))
nodes.append(AuthServiceProxy("http://%s:%s@%s"%(rpc_user, rpc_password, NODE2_URL)))
nodes.append(AuthServiceProxy("http://%s:%s@%s"%(rpc_user, rpc_password, NODE3_URL)))

for node in nodes:
    txs = node.getrawmempool()
    for tx in txs:
        rawtx = node.getrawtransaction(tx)
        for b_node in nodes:
            b_node.sendrawtransaction(rawtx)

What this does is it fetches a list of transaction ids in a node's mempool. Then it fetches each transaction and broadcasts them through the other nodes. It does this for each node. This, in theory, ensures that all of the nodes will have the same transactions in their mempool.

Answer 2

I've noticed 2 things:

1. If I started second node, when first had 1000 txs in mempool, then second node will have 1000 txs less (almost) constantly till mempool clearing. This means that all newly broadcasted txs hit all (three) mempools, but newer nodes cannot catch up with older nodes.
2. Periodically mempools decrease abruptly (from 70 thousands to 40 thousands for example). Nodes remove expired transactions or reach their maxmempool size and clean mempool. After this event, amount of transactions is roughly the same on all three nodes (difference is less than 100 txs, comparable to several thousands in my case).

---

Old transactions seems not to be broadcasted by nodes. And it looks right, because if nodes will broadcast all their mempool transactions all the time, network will be bloated by unnecessary information. So new node can catch up with old one only when all transaction, that existed before new node's start, will be deleted from mempools (by either way). However may be I am mistaken and there is a way to force new node fetch all transactions from mempool of older node.

(Almost) All new transactions will be in mempools of (almost) all running nodes. Mempools sooner or later will become roughly the same in size. The less time there is between the start of your node - the better (difference will be smaller).

So if you need to sync mempools, use addnode=IP:PORT option in your bitcoin.conf file (just to be sure that your nodes are talking to each other) and wait a little bit (approximately 72 hours) and mempools will be as close as possible to each other. However as far as I understood, there is always the possibility that some transactions will be missing in some mempools and there is no way to fix it.

Answer 3

You could set 2 nodes to connect to 1 node and nothing else.

Answer 4

You can use the answer by @Andrew above but instead of using the script use mempoolcp command line utility. https://github.com/dev7ba/mempoolcp

Mempoolpc takes into account the dependencies between transactions and the fact that you can't send a child tx before a parent tx, or a parent tx before a grandparent tx... because otherwise, the sent transactions could be denied by the receiving node. The script above does not have this into consideration.