3

I'm trying to scale getrawtransaction calls from bitcoind, without manually parsing the .dat files.

I've tried:

  • rpcworkqueue=512 (no difference)
  • rpcthreads=128 (no difference)
  • par=8 (no difference)
  • passing the blockhash with every call to avoid searching the txindex (big difference)
  • using JSON rpc batching (big difference)
  • having two separate instances on the same pc and rotating requests (double speed, but costs 200GB)

It seems that a bitcoind will only ever devote 1 core of the cpu to JSON RPC, so when it gets to 100% the only way to scale is with another instance. Unfortunately my server has 24 cores at 2.1GHZ.

Does anyone know if it's possible to have two instances of bitcoind running in the same folder. I.E. one instance is connected to the network and the other instance is running in a read only mode.

Or is is possible to allocate multiple cores(not threads) to JSON rpc

Any help is appreciated, thank you.

Improve this question

2 Answers 2

Reset to default
3

Or is is possible to allocate multiple cores(not threads) to JSON rpc

No. The problem is not to do with thread or core allocation. Rather Bitcoin Core makes use of locks to keep the chain state consistent when multiple threads may access it. In this case, you will need to access the chain state to get transactions so multiple getrawtransaction calls cannot really be run in parallel (locks aren't held for the entire call, but they are held for the parts that matter). So adding more cores or threads won't help at all.

The only thing that does help is to use the blockhash parameter. Without it, getrawtransaction will wait until Bitcoin Core isn't in the middle of processing multiple blocks, and the check and wait for this takes some time. However if you specify the blockhash, it will know immediately which block to look in and quickly look it up, no blocking or waiting for blocks to be processed. So this will be faster than without.

Otherwise there's nothing else that you can do.

Improve this answer
1
  • Hi Andrew, thank you for your insight. So even if the block is from 2014, there still needs to be a read lock to maintain chain state?
    – Roderick Obrist
    Commented Jul 16, 2019 at 11:33
0

At this point, your best bet is probably launching a bunch of 1 core pods on Kubernetes with copies of the chain lol

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.