Python - Querying bitcoin's block header using leveldb

Question

I am facing a stupid problem. I am trying to read bitcoin/blocks/index leveldb using plyvel. I've set up a regtest and i know there is a block with the hash :

27ce8199dee22f96e4f392dec29ef207fcd8a3c9e43b7d58cb40a5ba01af674b

I am trying to query it using plyvel.

Note that I already read the topics :

How does Bitcoin read from/write to LevelDB

What are the keys used in the blockchain levelDB (ie what are the key:value pairs)?

https://en.bitcoin.it/wiki/Protocol_documentation

Then, I know I have to swap the endianness so it becomes :

MYVAR = "4b67af01baa540cb587d3be4c9a3d8fc07f29ec2de92f3e4962fe2de9981ce27"

Thus, I should query something like :

b + (byte) MYVAR 

However I do not get how. I guess I missunderstand something in encoding. I tried :

db.get(b'b4b67af01baa540cb587d3be4c9a3d8fc07f29ec2de92f3e4962fe2de9981ce27')
db.get(b'b'+b'4b67af01baa540cb587d3be4c9a3d8fc07f29ec2de92f3e4962fe2de9981ce27')

..

Then, I opened the bitcoin/blocks/index leveldb using a gui explorer and I noticed the key for this block is :

\x62\x4b\x67\xaf\x01\xba\xa5\x40\xcb\x58\x7d\x3b\xe4\xc9\xa3\xd8\xfc\x07\xf2\x9e\xc2\xde\x92\xf3\xe4\x96\x2f\xe2\xde\x99\x81\xce\x27

which is \x62 corresponding to the "b" + the hash with \x (escaping caracter ? I know here is my misunderstanding on hex encoding)

and obviously

db.get(b'\x62\x4b\x67\xaf\x01\xba\xa5\x40\xcb\x58\x7d\x3b\xe4\xc9\xa3\xd8\xfc\x07\xf2\x9e\xc2\xde\x92\xf3\xe4\x96\x2f\xe2\xde\x99\x81\xce\x27')

perfectly works

I also tried these methods :

https://stackoverflow.com/questions/5649407/hexadecimal-string-to-byte-array-in-python

No one of them works and I really don't get why :

bytes.fromhex(hash)
bytearray.fromhex(hash)

Thank you in advance

Answer 1

This answer is a little stale. But here it is anyway.

You need to watch for the endianness of the bytes. How we look at the block id (block header hash) is bigendian, most significant bit first, and how bitcoin-core manages everything behind the scenes is littleendian, least significant bit first.

Here is a simple script to access the entire leveldb block index.

import re
from plyvel import DB

# Note for POSIX users, typically we have to escape spaces, 
# e.g. "Application\ Support". However this is not needed here so you 
# can write "Application Support" instead.
BITCOIN_DATADIR: str ="/absolute/path/to/your/bitcoin/datadir/"
db = DB(BITCOIN_DATADIR + "blocks/index")
keys = list(db.iterator(include_value=False))
regtest = re.compile(br"^b")
block_id_keys = list(filter(regtest.search, keys))

We can access the data structure and see this:

>>> bytes.hex(block_id_keys[-1][::-1][:-1])
'0000000000429f3501d60512df277b97f539becd18a7541b433e5110b2feffff'

https://blockstream.info/block/0000000000429f3501d60512df277b97f539becd18a7541b433e5110b2feffff

Going through the [-1][::-1][:-1], the [-1] is the last item in the list, the [::-1] flips the byte order from littleendian to bigendian, and the [:-1] drops the b"b".

To see the raw key, we can see the entire structure:

>>> block_id_keys[-1]
b"b\xff\xff\xfe\xb2\x10Q>C\x1bT\xa7\x18\xcd\xbe9\xf5\x97{'\xdf\x12\x05\xd6\x015\x9fB\x00\x00\x00\x00\x00"

Note the key begins with b"b".

To get the leveldb key from a bigendian hex blockid, you could do the following:

>>> hex_data = "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f"
>>> test_key = b"b" + bytes.fromhex(hex_data)[::-1]
>>> db.get(test_key)
b'\x8b\x99@\x00\x0b\x01\x00\x08\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00;\xa3\xed\xfdz{\x12\xb2z\xc7,>gv\x8fa\x7f\xc8\x1b\xc3\x88\x8aQ2:\x9f\xb8\xaaK\x1e^J)\xab_I\xff\xff\x00\x1d\x1d\xac+|'

Also see https://bitcoin.stackexchange.com/a/29418/124452

Answer 2

The leveldb databases are serialized for optimization and do not work well with attempting to manually parse the data. You can more easily get details on a specific blockhash using the built-in JSON-RPC API and use the getblock call.