2

I am trying to parse out the bitcoin data using Big Data and wanted to know more about the blk files themselves. I was wondering how does bitcoin split the transactions across the files. I know there is a 128 MiB limit on the files. Does that mean it is possible for transaction data to be split across a blk ? For instance the merkle root of the transaction is in one file and the output index is in another ? Or is block data retained in a single file ?

Improve this question

2 Answers 2

Reset to default
3

Blocks are never split across files: the whole block, from magic to length, to header, to transaction data, is always stored in a single file, continuously.

However you may need to take into account:

  • Blocks in the blk*.dat files are stored in the order they are received from the network.l, which may not be the blocks' logical order.
  • It is possible that a block is in file N, and its descendant block is in file N-1.
  • The files contain all blocks ever received, including ones that are not part of the main chain but were reorged out.
  • There can be "gaps" of zeroes or garbage data between blocks in the blk*.dat files.
Improve this answer
2
  • Thanks for your answer! Just one follow up, is there anyway we can pre-sort the blk files in anyway ? Because I'd like to process them into a CSV on individual nodes on a distributed environment and then find out the input address by looking backwards if that makes sense
    – Rohan Maheshwari
    Aug 7, 2022 at 3:53
  • 2
    Bitcoin Core has a linearize script: github.com/bitcoin/bitcoin/tree/master/contrib/linearize. This can be used to construct a single file with all blocks in the active chain (no reorged forks), in order, without gaps.
    – Pieter Wuille
    Aug 7, 2022 at 12:12
0

As a kind of addendum to the first bullet-point in Pieter's answer, I had written a program to parse blkxxxxx.dat files and, in my case at least, the blocks are very obviously not in logical order. Just as Pieter asserted.

Notice the day number in the timestamps in the first ten blocks:

C> blockchain -file blk02100.dat -blocks -longstats
Block      1, 2020-06-03 03:48, target 171297F6,   2273 transactions
Block      2, 2020-06-03 04:29, target 171297F6,   2768 transactions
Block      3, 2020-06-04 02:00, target 171297F6,   2177 transactions
Block      4, 2020-06-05 12:37, target 17147F35,   2243 transactions
Block      5, 2020-06-01 07:15, target 171297F6,   3150 transactions
Block      6, 2020-06-02 21:48, target 171297F6,   2911 transactions
Block      7, 2020-06-02 21:49, target 171297F6,   2722 transactions
Block      8, 2020-06-05 16:23, target 17147F35,    182 transactions
Block      9, 2020-06-03 05:34, target 171297F6,   2002 transactions
Block     10, 2020-06-03 04:36, target 171297F6,   1958 transactions
...
Block    111, 2020-06-06 03:39, target 17147F35,    259 transactions

Statistics for blk02100.dat (2020-06-01 to 2020-06-06)

   111 blocks
222519 transactions (2004.7 per block)
109173 Segwit tx    (49.1%)
622026 inputs       (mean 2.8, max 1030 per transaction)
614553 outputs      (mean 2.8, max 3799 per transaction)

Processed file in 322.5756ms

I guess this scattered order is because my node was receiving blocks from multiple peers and concurrently receiving them from fairly disjoint ranges.

I can confirm that every blkxxxxx.dat file I tried, started and ended at a block boundary. There were no partial blocks.

Footnote: My program isn't extensively tested so maybe take the above with a pinch of salt. I believe the timestamps to be parsed correctly.

Improve this answer
1
  • Yes ! I have found the same to be true and the python parser i am currently using parser also has the option to obtain blocks in an ordered/ unordered form.
    – Rohan Maheshwari
    Aug 7, 2022 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.