6

I am data mining the blockchain using JSON/RPC on a bitcoind full node with txindex (0.9.1).

I have 64 RPC connections simultaneously going to bitcoind from a remote client. My workload is mostly getrawtransaction and decoderawtransaction. I have bumped rpcthreads to 64.

What I am observing though is that bitcoind is taking a very large amount of time (from 30 seconds to 2-3 minutes) to respond to each call.

I have verified that there are about 66 connections opened on the RPC port, and the bitcoind process has 73 threads, so that seems to confirm that both my load and the rpcthreads setting are correct.

If I query something like "help", I get the same slow response time, so it seems to be related to the RPC server rather than actually accessing the data from disk/memory.

Am I missing a setting to make it scale?

Improve this question
3
  • 2
    What is the cpu load on the server? How about the memory usage; are you swapping? Bitcoind is not particularly optimized for performance so it's conceivable that it is just that slow.
    – Nate Eldredge
    Commented Apr 29, 2014 at 13:47
  • CPU is ok, but the memory usage is high, I see quite a bit of swapping. I will allocate more RAM to the machine and see if that makes a difference.
    – Flavien
    Commented Apr 29, 2014 at 13:51
  • 1
    When you have a use case that is very different from what everyone else uses a piece of software for, it's not unusual to see pathological performance behavior. You may want to profile the running instance and see if there's any obvious optimizations that help with your particular use case.
    – David Schwartz
    Commented Apr 29, 2014 at 20:55

1 Answer 1

Reset to default
5

Bitcoind can be quite memory hungry and bandwidth hungry. If you're swapping a lot, definitely add more memory.

You may also consider using an external API like:

http://dev.blockcypher.com/#blockchain-api

Bitcoind itself is actually quite hard to scale out.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.