4

I'm trying to write a parser for blk*.dat files from scratch. Right now, I can read blocks, extract the header fields and calculate the correct block hash based on the header fields.

I just tried to run the parser through all blk-files and it failed after successfully reading the first ~300 blk-files. When inspecting the failing file with a hex-editor, I found two "magic byte fields", only separated by 4 other bytes.

enter image description here

Since the mandatory block-header is supposed to be 80 bytes in length, I'm now quite confused.

I found three probably related questions:

  1. blk file error when reading - did something change in the format?
  2. How can you tell if you're at the end of an incomplete blk*.dat file?
  3. Zeros in blk00*.dat files

My guess, based on these other questions, is the following:

  • This is an 'incomplete block'
  • This happens when a block can only be partially downloaded, or written to disk.
  • Deleting the affected blk files and re-downloading parts of the blockchain doesn't make sense, since this can be expected to happen again.
  • The parser should be able to skip these blocks.
  • The overall blk file is not missing actual data and is not corrupt.
  • The 'incomplete block' will be downloaded and written again further down in the blk file (or to an other blk-file).

Which of my assumptions are correct? .. which are not? Am I missing something?

Improve this question
10
  • 1
    The node was probably killed while writing to disk.
    – Claris
    Commented Feb 14, 2018 at 20:45
  • 1
    Yeah. My files are in a strange state due to many forced node restarts during IBD. The software is fine with this because there’s an external database of pointers to the blocks on disk which can just skip the corruption.
    – Claris
    Commented Feb 15, 2018 at 18:07
  • 1
    Unlikely to occur by chance, perhaps, but there is nothing preventing anyone from intentionally inserting those 4 bytes in a transaction (via OP_RETURN, for instance). A miner who was feeling ornery could also twiddle the coinbase transaction until those bytes appeared in the Merkle root; it would take something on the order of 100 million tries, which is not much. In that case you'd see those bytes in the block header.
    – Nate Eldredge
    Commented Feb 15, 2018 at 18:30
  • 1
    Given your additional description, I agree that this is probably not the issue here, but it shows that your code definitely needs to handle this case properly.
    – Nate Eldredge
    Commented Feb 15, 2018 at 18:31
  • 2
    @NateEldredge I couldn't resist to search for it. :) As you already guessed, somebody did it. The following block from 2012 seems to contain such a case. blockchain.info/block/… This further stresses your suggestion to handle these cases properly.
    – forgemo
    Commented Feb 15, 2018 at 20:17

1 Answer 1

Reset to default
2

I'm going to answer my own question. These are my conclusions based on the comments from Nade Eldrege and eponymous.

Assumption 1: This is an 'incomplete block'

Yes, this is an incomplete block. More specifically, it is not part of the blk file format.

Assumption 2: This happens when a block can only be partially downloaded, or written to disk.

Yes, evidence strongly suggests that this happens when the bitcoind is interrupted / killed while writing to disk. If restarted, it just continues to download and write new blocks to the blk file. It doesn't clean up incomplete block.

Assumption 3: Deleting the affected blk files and re-downloading parts of the blockchain doesn't make sense, since this can be expected to happen again.

Yes and no. Yes, because deleting and re-downloading the blk files without interruptions solves the issue. When writing a blk-parser, however, it should be able to handle these incomplete blocks. For example, by skipping to the next valid block.

Assumption 4: The parser should be able to skip these blocks.

Yes. See #3

Assumption 5: The overall blk file is not missing actual data and is not corrupt.

I'm not sure, but since the bitcoind just continues with downloading blocks and works without any issues, one can assume that the incomplete block is just re-downloaded afterwards.

Assumption 6: The 'incomplete block' will be downloaded and written again further down in the blk file (or to an other blk-file).

It seems so. See #5

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.