Format of a block key's contents in bitcoind's LevelDB?

Question

I am trying to use bitcoind's LevelDB block index files (that are stored in blocks/index) to find the file (e.g. blk00029.dat) and byte position of specfic blocks.

Thanks to questions What is the database for? and What are the keys used in the blockchain levelDB (ie what are the key:value pairs)? I know where to look, however the contents of the database key for a block puzzles me. Even with the help of the source code I cannot figure out it's format.

For example for the block at height 100,000, which resides under database key b\x06\xe53\xfd\x1a\xda\x869\x1f?l42\x04\xb0\xd2x\xd4\xaa\xec\x1c\x0b \xaa'\xba\x03\x00\x00\x00\x00\x00 (block's hash string converted to bytes in little endian order), I am getting the following content which I hexlified for readability:

87c628858c201d04009be99e5883d1e92c0100000050120119172a610421a6c3011dd330d9df07b63616c2cc1f1cd00200000000006657a9252aacd5c0b2940996ecff952228c3067cc38d4885efb5a4ac4247e9f337221b4d4c86041b0f2b5710

The first 3 bytes (87c628) seem to be the same for every block that I get from the database. I'm guessing it is some version identifier?

The last 80 bytes (or 160 characters in hex) are the block header: 010000005 ... f2b5710. They match what you get by executing bitcoin-cli getblock 000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506 false.

The rest is a mystery to me. I know that block 100,000 had 4 transactions, but the byte that holds the value 4 has 129 for block #1, which had only a single transaction.

Where can I find out about the format of, at least, the block keys in LevelDB?

Could it be that the format of each key depends on the version of bitcoind that serialized the contents? If so, how do I figure out which format to use?

Answer 1

This is the Bitcoin Core v0.15.1 source code snippet that defines the serialization of CDiskBlockIndex. The various fields are:

• A varint version number of the client who wrote the record (for future extensibility). This is the 87c628 you're seeing.
• A varint block height.
• A varint status with various flags.
• A varint determining the number of transactions in the block.
• If the status field has bits 8 or 16 set, a varint determining which file (blk*.dat and rev*.dat) the block is stored in.
• If the status field has bit 8 set, a varint determining the offset into the blk*.dat file where the start of the block is.
• if the status field has bit 16 set, a varint determining the offset into the rev*.dat file where the start of the block's undo data is.
• The block header

Answer 2

This topic had me searching quite a lot as this post is almost all there is as form of documentation (thank Pieter for that). The missing piece, at least from my perspective, is that those fields are encoded using base 128 / Varint. To make things more difficult, this is not the same Varint as in sizes representations in serialized Bitcoin block (as explained here).

So, in an attempt to bring some light on this topic, here are 2 pieces of code, one in python, and the other in golang, that will decode and read those fields using a base128 decoder function. The first one is adapted from the book "The Undocumented Internals of Bitcoin, Ethereum and blockchains".
The golang version is my own adaptation from this python code.

python version:

import leveldb
import sys

def reverse(hash):
    return "".join(reversed([hash[i:i+2] for i in range(0, len(hash), 2)]))

def base128(ans, off):
    n = 0
    while True:
        ch = ord(ans[off:off+1])
        print ch
        off = off + 1
        n = (n << 7) | (ch & 0x7f)
        if ch & 0x80 == 128:
            n = n + 1
        else:
            return(n, off)

hash = "62" + reverse(sys.argv[1])
hash = hash.decode('hex')
db = leveldb.LevelDB('./testindex')
ans = db.Get(hash)

v, o = base128(ans, 0)
print v, o
v, o = base128(ans, o)
print v, o
# etc ...

golang version:

package main

import (
    "encoding/hex"
    "fmt"
    "log"
    "os"

    "github.com/syndtr/goleveldb/leveldb"
)

func reverse(s []byte) []byte {
    for i, j := 0, len(s)-1; i < j; i, j = i+1, j-1 {
        s[i], s[j] = s[j], s[i]
    }
    return s
}

// binary.Uvarint is supposed to do this, except it doesn't.
// And it is not the same varint as in bitcoin serialized varints.
// this is ported from "The Undocumented Internals of The Bitcoin
// and Ethereum Blockchains" python code
func base128(b []byte, offset uint64) (uint64, uint64) {
    for n := 0; ; n++ {
        ch := int(b[offset])
        offset++
        n = (n << 7) | (ch & 0x7f)
        if ch&0x80 != 128 {
            return uint64(n), offset
        }
    }
    return 0, 0
}

func main() {
    db, err := leveldb.OpenFile("./testindex", nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    h, err := hex.DecodeString(os.Args[1])
    if err != nil {
        log.Fatal(err)
    }

    b := append([]byte("b"), reverse(h)...)

    data, err := db.Get(b, nil)
    if err != nil {
        log.Fatal(err)
    }

    v, o := base128(data, 0)
    fmt.Printf("%d %d\n", v, o)
    v, o = base128(data, o)
    fmt.Printf("%d %d\n", v, o)
    // etc...
}

HTH

Answer 3

Ok, I have produced a standalone code snippet that can decode the block index records stored in the levelDB database. The challenge here, is that bitcoin core appears to use an extremely non-standard variable length integer representation. It does not match 'varint128' which is used by Google protobufs and other projects. It's very similar, but there is a real difference.

In this code snippet you pass the raw data stored in the levelDB record and it will populate a data structure with the results.

You can find the code reference here:

https://github.com/jratcliff63367/bitcoinstats/blob/main/include/CBlockIndex.h