How exactly do I decode the block file info in leveldb?

Question

Base on the links https://github.com/bitcoin/bitcoin/blob/fcbc8bfa6d10cac4f16699d6e6e68fb6eb98acd0/src/main.h#L392 and What are the keys used in the blockchain levelDB (ie what are the key:value pairs)? I should simply read in the results as varints. So I wrote some Python code shown below. But I did not get the results I expected.

Below is my code to decode the file information with file number 0. I parse through my blk00000.dat and found that the first block was the genesis block and the last block was block 177 with blockhash 00000000480c9799dd2b35009ff1833f69690f76c26a9bea7e00cd0042f67db1.

nBlocks looks to be correct with 134 blocks which was how many blocks I found in blk00000.dat. nHeightFirst looks right with the value being 0, but I nHeightLast is not 177. nSize and nUndoSize dont look right since it doesn't match the file size with I found on the terminal doing

ls -l  ~/.bitcoin/blocks/blk00000.dat

and

ls -l  ~/.bitcoin/blocks/rev00000.dat

Also the nTimeFirst does not match the timestamp of the genesis block. And nTimeLast does not match the timestamp of block 177, which I found to be the block with the highest timestamp in blk00000.dat. What am doing wrong?

import plyvel
blockIndexDB = plyvel.DB('/home/chris/.bitcoin/blocks/index')
result = blockIndexDB.get(b'f\x00\x00\x00\x00')

# result value from leveldb
# result = b'\x86\xa8%\xbe\xfe\xf4E\x88\xa5\xa7}\x00\x86\xa8r\x83\xc9\xfd\xd5)\x83\xec\xd3\xa1N'
# hexString = b'86a825befef44588a5a77d0086a87283c9fdd52983ecd3a14e'

nBlocks = int.from_bytes(result[0:1], byteorder='little')
print('nBlocks', nBlocks)
print(nBlocks == 134)

s = int.from_bytes(result[1:2], byteorder='little')
# print('s', s)
nSize = int.from_bytes(result[2:6], byteorder='little')
print('nSize', nSize)
print(nSize == 134216389) # blk00000.dat file size

s = int.from_bytes(result[6:7], byteorder='little')
# print('s', s)
nUndoSize = int.from_bytes(result[7:11], byteorder='little')
print('nUndoSize', nUndoSize)
print(nUndoSize == 19502205) # rev00000.dat file size

nHeightFirst = int.from_bytes(result[11:12], byteorder='little')
print('nHeightFirst', nHeightFirst)
print(nHeightFirst == 0)
nHeightLast = int.from_bytes(result[12:13], byteorder='little')
print('nHeightLast', nHeightLast)
print(nHeightLast == 177)

s = int.from_bytes(result[15:16], byteorder='little')
# print('s', s)
nTimeFirst = int.from_bytes(result[16:20], byteorder='little')
print('nTimeFirst', nTimeFirst)
print(nTimeFirst == 1231006505) # block 0 timestamp

s = int.from_bytes(result[20:21], byteorder='little')
# print('s', s)
nTimeLast = int.from_bytes(result[21:], byteorder='little')
print('nTimeLast', nTimeLast)
# block 177 is the last block in blk00000.dat file
print(nTimeLast == 1231736557) # block 177 timestamp

Answer 1

You are deserializing every value as an integer but that is incorrect. The values are serialized as variable length integers with high bits signalling the length of the integer which are dropped for the actual integer value itself. So you need to fix your code to deserialize the bytes as varints instead of just integers with fixed lengths.

The following code was able to deserialize the entry correctly.

#! /usr/bin/env python3

import binascii

def get_max(long=False):
    if long:
        return 18446744073709551615
    else:
        return 4294967295

def read_var_int(s, pos, long=False):
    n = 0
    while True:
        chData = s[pos]
        pos += 1
        if n > (get_max(long) >> 7):
            raise IOError("ReadVarInt(): size too large");
        n = (n << 7) | (chData & 0x7F)
        if chData & 0x80:
            if n == get_max(long):
                raise IOError("ReadVarInt(): size too large");
            n += 1
        else:
            return n, pos

entry = binascii.unhexlify(b'86a825befef44588a5a77d0086a87283c9fdd52983ecd3a14e')
pos = 0

nBlocks, pos = read_var_int(entry, pos)
print('nBlocks: {}'.format(nBlocks))
nSize, pos = read_var_int(entry, pos)
print('nSize: {}'.format(nSize))
nUndoSize, pos = read_var_int(entry, pos)
print('nUndoSize: {}'.format(nUndoSize))
nHeightFirst, pos = read_var_int(entry, pos)
print('nHeightFirst: {}'.format(nHeightFirst))
nHeightLast, pos = read_var_int(entry, pos)
print('nHeightLast: {}'.format(nHeightLast))
nTimeFirst, pos = read_var_int(entry, pos)
print('nTimeFirst: {}'.format(nTimeFirst))
nTimeLast, pos = read_var_int(entry, pos)
print('nTimeLast: {}'.format(nTimeLast))

The result was:

nBlocks: 119973
nSize: 134216389
nUndoSize: 19502205
nHeightFirst: 0
nHeightLast: 120050
nTimeFirst: 1231006505
nTimeLast: 1303712078

The nBlocks and nHeightLast are expected because early blocks were extremely small yet the block file is fairly large. I think that your calculation of 134 blocks in the block file is wrong. With 134 blocks, the blocks would have to be more than 1 MB in size which is simply untrue for early blocks.