How does bitcoind store blocks?

Question

I want to parse blockchain manually and extract the UTXO set (for various reasons). Is there any way to understand how the blocks are stored without going through the C++ code?

Some specs will be great.

Answer 1

There is a lot of information available, just not consolidated. After searching for sometime, I found the best answer from this link.

The format of each file is:

4 bytes: Magic bytes
4 bytes: Encodes size N of next upcoming block.
N bytes: Block encoded in standard format (with header)
--
above repeats

Quotes from the original link:

4 | 4 | 80 | TxData | 4 | 4 | 80 | TxData | 4 | 4 | 80 | TxData | ...

• First 4 bytes: magic bytes (identifying which network you are on)
• Second 4 bytes: the number of bytes of the remaining block
• Next 80 bytes: block header itself
• Next NumBlockBytes - 80 bytes: Transaction data in this block [ numTx | Tx1 | Tx2 | Tx3 | ... ]

What I don't understand is why are the magic bytes repeated? We could very well have used:

4 | 4 | 80 | TxData | 4 | 80 | TxData | 4 | 80 | TxData | ...

Answer 2

I want to parse blockchain manually

If you mean blockchain is a RAW on-disk data, which contains all information about the blocks and transactions (../blocks/blk*.dat files), so you can try blockchain parser. This is a standalone script that can parse that files. No installation needed, no dependencies.

Is there any way to understand how the blocks are stored without going through the C++ code?

The blockchain parser by the link can give you the way to understand how the bocks are stored in a simple intuitive view.

extract the UTXO set

For extraction of the UTXOs (for constructing UTXO set) you need to understand that this data did not stored permanently on disk.

There are two ways for building the UTXO set:

1. Parse the chainstate frozen dump (../chainstate/*.ldb files)
2. Construct UTXO set from parsed RAW data (../blocks/blk*.dat files)

The first way is faster, but not so usefull for deep understanding of how blockchain really works. The second way is what I'am trying to do now. This also called blockchain verification by third-parties scripting. I made the script for UTXO set building from RAW dumps, but my implementation on Python is very very slow.

I hope my answer may help you in your explorations.