I want to parse blockchain manually and extract the UTXO set (for various reasons). Is there any way to understand how the blocks are stored without going through the C++ code?
Some specs will be great.
There is a lot of information available, just not consolidated. After searching for sometime, I found the best answer from this link.
The format of each file is:
4 bytes: Magic bytes 4 bytes: Encodes size N of next upcoming block. N bytes: Block encoded in standard format (with header) -- above repeats
Quotes from the original link:
4 | 4 | 80 | TxData | 4 | 4 | 80 | TxData | 4 | 4 | 80 | TxData | ...
What I don't understand is why are the magic bytes repeated? We could very well have used:
4 | 4 | 80 | TxData | 4 | 80 | TxData | 4 | 80 | TxData | ...
I want to parse blockchain manually
If you mean
blockchain is a RAW on-disk data, which contains all information about the blocks and transactions (../blocks/blk*.dat files), so you can try blockchain parser. This is a standalone script that can parse that files. No installation needed, no dependencies.
Is there any way to understand how the blocks are stored without going through the C++ code?
The blockchain parser by the link can give you the way to understand how the bocks are stored in a simple intuitive view.
extract the UTXO set
For extraction of the UTXOs (for constructing UTXO set) you need to understand that this data did not stored permanently on disk.
There are two ways for building the UTXO set:
The first way is faster, but not so usefull for deep understanding of how blockchain really works.
The second way is what I'am trying to do now. This also called
blockchain verification by third-parties scripting. I made the script for UTXO set building from RAW dumps, but my implementation on Python is very very slow.
I hope my answer may help you in your explorations.