3

I am currently working on a script that parses every BTC transaction from the beginning of the blockchain. My goal is to record summaries about each transaction in database, after synthesizing it.

While working on this, I observed that the first time I am using getrawtransaction, for 1000 transactions (i.e., 1000 RPCs), it takes between 20 and 30 seconds.

The second time I do the RPCs, for the same 1000 transactions, it takes less than 1 second.

My conclusion is that there is some caching mechanism set up, somewhere. I do not know if it is caching at the OS layer or at the software layer, though, and cannot manage to find any documentation on this. Incidentally, I would like to be able to manage this mechanism to allow me to make requests more quickly.

Does anyone have an idea?

Thank you very much.

1 Answer 1

4

I'm pretty sure you're observing the disk cache. When you call getrawtransaction, Bitcoin Core first needs to check the transaction index (enabled by -txindex) and then load the transaction from its blk*.dat file (and rev*.dat file if you're also requesting prevouts), all of which are operations that access the disk. As far as I know, Bitcoin Core itself doesn't have a cache that would help in this situation.

Side note, if you want to process the entire blockchain with an external program, I'd recommend using getblock instead of getrawtransaction.

3
  • 2
    The database system Bitcoin Core uses for the txindex is LevelDB, which does have a small in-memory cache in user space. Still, I'd expect that the majority of what OP is observing is indeed the operating system's disk cache. Commented Oct 23, 2023 at 14:36
  • Thank you very much for your answer! The disk cache theory makes sense, and I noted your advice about getblock.
    – Lev
    Commented Oct 24, 2023 at 8:44
  • @vojtěch-strnad just to let you know that using getblock and parsing the transactions from there, instead of using getrawtransaction, has divided the time of my program by 3. Many thanks again!
    – Lev
    Commented Oct 24, 2023 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.