I am trying to parse out the bitcoin data using Big Data and wanted to know more about the blk files themselves. I was wondering how does bitcoin split the transactions across the files. I know there is a 128 MiB limit on the files. Does that mean it is possible for transaction data to be split across a blk ? For instance the merkle root of the transaction is in one file and the output index is in another ? Or is block data retained in a single file ?
Blocks are never split across files: the whole block, from magic to length, to header, to transaction data, is always stored in a single file, continuously.
However you may need to take into account:
- Blocks in the blk*.dat files are stored in the order they are received from the network.l, which may not be the blocks' logical order.
- It is possible that a block is in file N, and its descendant block is in file N-1.
- The files contain all blocks ever received, including ones that are not part of the main chain but were reorged out.
- There can be "gaps" of zeroes or garbage data between blocks in the blk*.dat files.
Thanks for your answer! Just one follow up, is there anyway we can pre-sort the blk files in anyway ? Because I'd like to process them into a CSV on individual nodes on a distributed environment and then find out the input address by looking backwards if that makes sense Aug 7, 2022 at 3:53
2Bitcoin Core has a linearize script: github.com/bitcoin/bitcoin/tree/master/contrib/linearize. This can be used to construct a single file with all blocks in the active chain (no reorged forks), in order, without gaps. Aug 7, 2022 at 12:12
As a kind of addendum to the first bullet-point in Pieter's answer, I had written a program to parse
blkxxxxx.dat files and, in my case at least, the blocks are very obviously not in logical order. Just as Pieter asserted.
Notice the day number in the timestamps in the first ten blocks:
C> blockchain -file blk02100.dat -blocks -longstats Block 1, 2020-06-03 03:48, target 171297F6, 2273 transactions Block 2, 2020-06-03 04:29, target 171297F6, 2768 transactions Block 3, 2020-06-04 02:00, target 171297F6, 2177 transactions Block 4, 2020-06-05 12:37, target 17147F35, 2243 transactions Block 5, 2020-06-01 07:15, target 171297F6, 3150 transactions Block 6, 2020-06-02 21:48, target 171297F6, 2911 transactions Block 7, 2020-06-02 21:49, target 171297F6, 2722 transactions Block 8, 2020-06-05 16:23, target 17147F35, 182 transactions Block 9, 2020-06-03 05:34, target 171297F6, 2002 transactions Block 10, 2020-06-03 04:36, target 171297F6, 1958 transactions ... Block 111, 2020-06-06 03:39, target 17147F35, 259 transactions Statistics for blk02100.dat (2020-06-01 to 2020-06-06) 111 blocks 222519 transactions (2004.7 per block) 109173 Segwit tx (49.1%) 622026 inputs (mean 2.8, max 1030 per transaction) 614553 outputs (mean 2.8, max 3799 per transaction) Processed file in 322.5756ms
I guess this scattered order is because my node was receiving blocks from multiple peers and concurrently receiving them from fairly disjoint ranges.
I can confirm that every
blkxxxxx.dat file I tried, started and ended at a block boundary. There were no partial blocks.
Footnote: My program isn't extensively tested so maybe take the above with a pinch of salt. I believe the timestamps to be parsed correctly.
Yes ! I have found the same to be true and the python parser i am currently using parser also has the option to obtain blocks in an ordered/ unordered form. Aug 7, 2022 at 3:54