Reading blk.dat files using pybtc does not yield transactions

Question

I am trying to parse the blk.dat files using pybtc but I can't seem to get this right.

       f = open(block_filename, "rb")
        lines = f.readlines()
        for block in tqdm(lines):
            block = block.strip().hex()
            for i in range(len(block)):
                tmp_block = block[i:]
                try:
                    bt = pybtc.Block(tmp_block, format="raw")
                    tx = bt["tx"]
                    if (len(tx.keys()) > 0):
                        print("keys", tx["vOut"].keys())

Can anyone show me how to properly parse the blk.dat files using this library? Most of the time I get the following errors:

Int too large to convert
Empty tx dictionary

Answer 1

Can anyone show me how to properly parse the blk.dat files using this library?

I can't, using that library, but the following observations might help:

lines = f.readlines()

I'd expect something more like bindata = f.read() You opened the file for reading binary data (br) not for reading lines of text.

for block in tqdm(lines):

Isn't tqdm a progress-bar library? I don't understand what you are trying to do with this line. It isn't clear to me that this would extract a block from blockfile data. Since you are having problems it would be appropriate to add comments to the code explaining the intent of each line.

  block = block.strip().hex()

The string method strip() removes leading and trailing whitespace. Blockfiles contain binary data which shouldn't have string functions of this sort applied to them. For example 0x20 and 0x09 etc can occur as part of data without signifying whitespace (space or tab), removing these byte values could damage the data. Since you don't show your imports we don't know what data type your tqdm returns and therefore don't know what its methods do.

You don't show your imports which might help work out what this code does.

The method .hex() presumably was intended to convert the block data into a hexadecimal string. This would be OK as the the author's example suggests this is what pybtc's Block function expects.

However so far as I can tell, Python strings don't have a .hex() method. Python integers do. This might explain the Int too large to convert error you get, I don't know Python well enough to say for sure.

        for i in range(len(block)):
            tmp_block = block[i:]
            try:
                bt = pybtc.Block(tmp_block, format="raw")

This code seems to be searching for the start of a block and discarding bytes until it finds a starting point where the library succeeds in decoding a block. I think this isn't necessary. The blk*.dat files contain a fixed 4-byte network indication 0xD9B4BEF9 before each block. I would just check for that value and exit with a fatal error if it is not present. But note that the data may also contain this sequence in places other than the start of a block. You can check for it but not search for it.