I can't parse the UTXO from the chainstate

Question

I have a problem with reading tx out from the Bitcoin Core LevelDB chainstate storage. I know that the structure of the current version of the chainstate is key => value where the key is a prefix (0x43) | tx_id | coin_n (https://github.com/bitcoin/bitcoin/blob/0d509bab45d292caeaf34600e57b5928757c6005/src/txdb.cpp#L39) and the value is obfuscated by the obfuscation key through the Coin XOR obfuscation_key value of the Coin.

Ref: https://github.com/bitcoin/bitcoin/blob/0d509bab45d292caeaf34600e57b5928757c6005/src/txdb.cpp#L68

So, i have successfully read the prefix, obfuscation key and the txId with the output_n

key:
    prefix: [1]byte

(COutPoint)
    txID: [32]byte
    outN: VARINT

Ref: https://github.com/bitcoin/bitcoin/blob/0d509bab45d292caeaf34600e57b5928757c6005/src/primitives/transaction.h#L28

The Coin structure (https://github.com/bitcoin/bitcoin/blob/0d509bab45d292caeaf34600e57b5928757c6005/src/coins.h#L24)

code: VARINT((coinbase ? 1 : 0) | (height << 1))
txOut:  CTxOut (via TxOutCompression)

Now i stucked on the reading the code. In some values, i have a valid code with true block height and coinbase boolean. But in some values i have invalid data where the block height is not the same with the transaction's block height.

How you can reproduce this to help me?

My input (LevelDB view):

Key                                                                     | Value
0e006f62667573636174655f6b6579                                          | 08f05564aa8317fe24
42                                                                      | bde0dff15a88e0eb3d71c37fd986c4cdfd8916bd0b8d0b13343afef66829284d
4300c4626bf6dc028730d3a1f3951d398a71a164d8a898965d11bb8a699091144c00    | 8f6778aa979ddbbc669c187cbf11ad2e30ca53025f3f3b961e
43047ae5966a25bdc32ab4a258c57e28ffa763fc273c08b38fe85839e99e71402500    | ef6778aa979ddbbc669c187cbf11ad2e30ca53025f3f3b961e
430e3afb85bb66cc8a5edfd96f4aad1e65e4e6cf38f1dcc584c525d0b25b89599c00    | 706e56b68303740168c3add6552bf877fa95fb9d2bcbd6e142bb

When i am reading the first values and deobfuscate them, i am getting true block height and coinbase. But when i am reading the last value, i am getting this:

~ 706e56b68303740168c3add6552bf877fa95fb9d2bcbd6e142bb XOR f05564aa8317fe24
~ 803b321c00148a259896c97cd63c06530ac09f37a8dc28c5b2ee

So, when i am trying to read the uvarint from this, i am getting an invalid code with invalid block height:

package main

import (
    "bytes"
    "encoding/binary"
    "encoding/hex"
    "log"
)

func main() {
    utxo, _ := hex.DecodeString("803b321c00148a259896c97cd63c06530ac09f37a8dc28c5b2ee")
    utxoReader := bytes.NewReader(utxo)

    code, _ := binary.ReadUvarint(utxoReader)
    log.Println("code", code)
    
    blockHeight := code >> 1
    log.Println("blockHeight", blockHeight)
    
    isCoinbase := code|1 == 1
    log.Println("isCoinbase", isCoinbase)
}

Result

2024/04/03 15:24:06 code 7552
2024/04/03 15:24:06 blockHeight 3776
2024/04/03 15:24:06 isCoinbase false

Expected result: blockHeight must be 93, coinbase false.

Other information. My deobfuscate function code:

func deobfuscateValue(obfuscationKey, value []byte) []byte {
    deobfuscatedValue := make([]byte, len(value))

    copy(deobfuscatedValue, value)

    for i, c := range value {
        deobfuscatedValue[i] = c ^ obfuscationKey[i%len(obfuscationKey)]
    }

    return deobfuscatedValue
}

I am running the private regtest network on the Bitcoin Core v0.21 (i have tried on the 24 version and have the same results). I am use bitcoin-testnet-box docker image to test my program. Also, i have tested on the mainnet, but have the same problem too. My blockchain now have only 105 blocks.

Thank you for helping. I think I missed something simple and just don't know something about the chainstate or C++ realization or Golang :)

Answer 1

I got it :_)

I used binary.ReadUvarint which not the same as ReadVarint in the Bitcoin Core (https://github.com/bitcoin/bitcoin/blob/0d509bab45d292caeaf34600e57b5928757c6005/src/serialize.h#L465)

So far I have not been able to explain the reason for this behavior, but i found the right code on the btcd (https://github.com/btcsuite/btcd/blob/ae5533602c46f4ea14a4da725c19cb3db7d5eb88/blockchain/compress.go#L93) and adapted it to my use case.

func DeserializeVLQ(buf io.ByteReader) (uint64, int, error) {
    var n uint64
    var size int

    for {
        val, err := buf.ReadByte()
        if err != nil {
            if err == io.EOF {
                break // End of stream is a natural break
            }
            return 0, 0, err // Return any error that occurs
        }

        size++
        n = (n << 7) | uint64(val&0x7F)
        if val&0x80 != 0x80 {
            break
        }
        n++
    }

    return n, size, nil
}