4

I am trying to use bitcoind's LevelDB block index files (that are stored in blocks/index) to find the file (e.g. blk00029.dat) and byte position of specfic blocks.

Thanks to questions What is the database for? and What are the keys used in the blockchain levelDB (ie what are the key:value pairs)? I know where to look, however the contents of the database key for a block puzzles me. Even with the help of the source code I cannot figure out it's format.

For example for the block at height 100,000, which resides under database key b\x06\xe53\xfd\x1a\xda\x869\x1f?l42\x04\xb0\xd2x\xd4\xaa\xec\x1c\x0b \xaa'\xba\x03\x00\x00\x00\x00\x00 (block's hash string converted to bytes in little endian order), I am getting the following content which I hexlified for readability:

87c628858c201d04009be99e5883d1e92c0100000050120119172a610421a6c3011dd330d9df07b63616c2cc1f1cd00200000000006657a9252aacd5c0b2940996ecff952228c3067cc38d4885efb5a4ac4247e9f337221b4d4c86041b0f2b5710

The first 3 bytes (87c628) seem to be the same for every block that I get from the database. I'm guessing it is some version identifier?

The last 80 bytes (or 160 characters in hex) are the block header: 010000005 ... f2b5710. They match what you get by executing bitcoin-cli getblock 000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506 false.

The rest is a mystery to me. I know that block 100,000 had 4 transactions, but the byte that holds the value 4 has 129 for block #1, which had only a single transaction.

Where can I find out about the format of, at least, the block keys in LevelDB?

Could it be that the format of each key depends on the version of bitcoind that serialized the contents? If so, how do I figure out which format to use?

  • LevelDB has its own database format and compression stuff that may make reading the data directly from disk as binary data to be difficult. You're probably running into some of that stuff. – Andrew Chow Jan 4 '18 at 22:42
  • You're right, that's why I am reading the LevelDB using a Python library (plyvel). That seems to be working correctly because I can retrieve records for blocks (bX*32 records) and files (fXXXX). I want to use the info stored in these records to seek through the .blk files directly. – Arjen Jan 5 '18 at 8:55
  • Arjen can you please publish your code for retrieving the records using plyvel? – Adam Rosenthal Aug 18 '18 at 18:59
4

This is the Bitcoin Core v0.15.1 source code snippet that defines the serialization of CDiskBlockIndex. The various fields are:

  • A varint version number of the client who wrote the record (for future extensibility). This is the 87c628 you're seeing.
  • A varint block height.
  • A varint status with various flags.
  • A varint determining the number of transactions in the block.
  • If the status field has bits 8 or 16 set, a varint determining which file (blk*.dat and rev*.dat) the block is stored in.
  • If the status field has bit 8 set, a varint determining the offset into the blk*.dat file where the start of the block is.
  • if the status field has bit 16 set, a varint determining the offset into the rev*.dat file where the start of the block's undo data is.
  • The block header
  • I've now managed to decode the first five fields: version, height, status, num of txs and block number. I'm getting incorrect values for the data position, but I now know that I'm on the right track. Thank you for taking the time to answer. – Arjen Jan 6 '18 at 14:06
  • 1
    I was a bit too fast: I had a bug in my VarInt parser. All is good now. Thanks again! – Arjen Jan 6 '18 at 14:39
0

This topic had me searching quite a lot as this post is almost all there is as form of documentation (thank Pieter for that). The missing piece, at least from my perspective, is that those fields are encoded using base 128 / Varint. To make things more difficult, this is not the same Varint as in sizes representations in serialized Bitcoin block (as explained here).

So, in an attempt to bring some light on this topic, here are 2 pieces of code, one in python, and the other in golang, that will decode and read those fields using a base128 decoder function. The first one is adapted from the book "The Undocumented Internals of Bitcoin, Ethereum and blockchains".
The golang version is my own adaptation from this python code.

python version:

import leveldb
import sys

def reverse(hash):
    return "".join(reversed([hash[i:i+2] for i in range(0, len(hash), 2)]))

def base128(ans, off):
    n = 0
    while True:
        ch = ord(ans[off:off+1])
        print ch
        off = off + 1
        n = (n << 7) | (ch & 0x7f)
        if ch & 0x80 == 128:
            n = n + 1
        else:
            return(n, off)

hash = "62" + reverse(sys.argv[1])
hash = hash.decode('hex')
db = leveldb.LevelDB('./testindex')
ans = db.Get(hash)

v, o = base128(ans, 0)
print v, o
v, o = base128(ans, o)
print v, o
# etc ...

golang version:

package main

import (
    "encoding/hex"
    "fmt"
    "log"
    "os"

    "github.com/syndtr/goleveldb/leveldb"
)

func reverse(s []byte) []byte {
    for i, j := 0, len(s)-1; i < j; i, j = i+1, j-1 {
        s[i], s[j] = s[j], s[i]
    }
    return s
}

// binary.Uvarint is supposed to do this, except it doesn't.
// And it is not the same varint as in bitcoin serialized varints.
// this is ported from "The Undocumented Internals of The Bitcoin
// and Ethereum Blockchains" python code
func base128(b []byte, offset uint64) (uint64, uint64) {
    for n := 0; ; n++ {
        ch := int(b[offset])
        offset++
        n = (n << 7) | (ch & 0x7f)
        if ch&0x80 != 128 {
            return uint64(n), offset
        }
    }
    return 0, 0
}

func main() {
    db, err := leveldb.OpenFile("./testindex", nil)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    h, err := hex.DecodeString(os.Args[1])
    if err != nil {
        log.Fatal(err)
    }

    b := append([]byte("b"), reverse(h)...)

    data, err := db.Get(b, nil)
    if err != nil {
        log.Fatal(err)
    }

    v, o := base128(data, 0)
    fmt.Printf("%d %d\n", v, o)
    v, o = base128(data, o)
    fmt.Printf("%d %d\n", v, o)
    // etc...
}

HTH

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.