6

While trying to understand the mining process and checking in python..... i am puzzled with which info from/to pool are in Little Endian, which one needs reversing in building the block header + hashing and which one not. ??

For example ... Let's start with connecting to solo.ckpool.org response to mining.authorize gives me ..... job_id,prevhash,coinb1,coinb2,merkle_branch,version,nbits,ntime,clean_jobs =

{'id': None, 'method': 'mining.notify', 'params': ['59bc8dfc0000029f',
        '0eba48f47bc0ab4bb35b230849868bf1d79aeb19006eed460000000000000000',
        '01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff35039a6807000404d5bc5904fc873d140c',
        '0a636b706f6f6c112f736f6c6f2e636b706f6f6c2e6f72672fffffffff03c4626c4e000000001976a914fc6123f4bfd3a840b4387ab90e9801e98fb17cf888acb1caca00000000001976a914f4cbe6c6bb3a8535c963169c22963d3a20e7686988ac0000000000000000266a24aa21a9edab3b87f3963e55a23f3cea5e4435bf6637328fd5e329bb8fdac7d31c3a01d26200000000',
        ['c251d18568d2cdb07c376d5f3304a2161e3d09d289d1a7e53737199171d73ae6',
         '4f9bcac3ec00fc8238db04b7758a1b673bdf38b93e48159678b60b51517dd563',
         '2aadad2c9c8e6df5c71264524b090cdf89830f12949527f7872c243bc1dc552c',
         '8c5edf39c0d85c9853308eef3ce6f9b71c8d6584ca4120d411bc75987bb2f254',
         '48751c2ae0ddc05f5d504a3d6039e7d5afb3d5f2a6aaac93bd5822e5f8efc328',
         '3e9b5c6b986c2ddc09bf8604eb2bd2a3b49b46f11042c282331a206dadb4296e',
         'c65f502b11b325abc0fac847e8fc5cd300169160e7244b29dd5acc3d0e4b8605',
         'afe7d2287020c4291f609a772adbeafc952a8de0ced38a7346acb6c7103bec6d',
         '0caa545ae63de39d47baea100ea15a18061235d7e1a372fbca88edf9ca5ac797',
         'c5bfff15b4dd64e7faac98f1bba118fb11990e4ff146cb45c7f4af6b93e2ea03',
         'a8805201e8152c1f5835c9aa024752fb34aeb7fa55dd3af0ae6109918f328dbb'],
        '20000000',
        '1801310b',
        '59bcd504',
        True]}

Based on the pool response....I can build the coinbase just by

coinbase = coinb1 + extranonce1 + extranonce2 + coinb2 
coinbase_hash_bin = hashlib.sha256(hashlib.sha256(binascii.unhexlify(coinbase)).digest()).digest()

and same to include in merkle tree with hashing to get merkle_root. I am not sure if i need to reverse the bits here ?

But to build a block header ... should i do this..

header = version + prevhash + merkle_root + ntime + nbits + nonce

or this..

header = binascii.hexlify(binascii.unhexlify(version)[::-1]).decode('utf-8') + \
prevhash + \
merkle_root + \
binascii.hexlify(binascii.unhexlify(ntime)[::-1]).decode('utf-8') + \
binascii.hexlify(binascii.unhexlify(nbits)[::-1]).decode('utf-8') + \
binascii.hexlify(binascii.unhexlify(nonce)[::-1]).decode('utf-8')

blockheader:000000200eba48f47bc0ab4bb35b230849868bf1d79aeb19006eed460000000000000000fb68998fe1dd4cc22fa1225c2592c27df5331d40d9b33ffce99b26c0aead8ef404d5bc590b3101181e3eef8c

Can someone please post an example of a recent BLOCK HEADER ... i think i m doing something wrong !

I am comparing this hash

hash = hashlib.sha256(hashlib.sha256(binascii.unhexlify(header)).digest()).digest()[::-1]

with the target "000000000001310b000000000000000000000000000000000000000000000000"

2 Answers 2

5

Let's take a look at this block, because it only has one transaction (the coinbase): 000000000000000000eb2d0ed97a7b2cff7f1408417dca83908004beb6fd9b95

Let's grab the raw hex data:

0000002053f7ebc735f54ac8d4ebcc1eeb3d3bdea89603bdd27431000000000000000000ff964ec70ea5a5356a04fdb044ca6c59ffda2dd6be02c63d9ea58fb6d46e01991a22bb590b310118f6df26f801010000000001010000000000000000000000000000000000000000000000000000000000000000ffffffff3103b46707244d696e656420627920416e74506f6f6c6b2f4542312f4144362f4e59412f332059bb221ab8080000615c0300ffffffff02807c814a000000001976a914660371326d3a2e064c278b20107a65dad847e8a988ac0000000000000000266a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf90120000000000000000000000000000000000000000000000000000000000000000000000000

There are 6 things which make up the block header:

  1. First we have the version (4 bytes), 00000020, which is reversed bytes of 0x20000000.
  2. Then we have the previous block hash (32 bytes), also in reversed byte order, 53f7ebc735f54ac8d4ebcc1eeb3d3bdea89603bdd27431000000000000000000 which is this block.
  3. Next we have the merkle root of the transactions (32 bytes), in reverse byte order, ff964ec70ea5a5356a04fdb044ca6c59ffda2dd6be02c63d9ea58fb6d46e0199. Because there is only one transaction, the coinbase transaction, the merkle root is just equal to that transactions txid.
  4. Then we have the time stamp (4 bytes), in reverse byte order, 1a22bb59 which is 0x59bb221a in hex or 1505436186 in decimal.
  5. Then we have the reversed bytes of the 'bits' field (4 bytes), 0b310118 which is 0x1801310b
  6. And then the nonce (4 bytes), f6df26f8 which is 0xf826dff6

Then after the header, we have the byte 01 which means there is one transaction, which follows. That one transaction is just the encoded coinbase transaction, as you can see here.

Of course, only the block header needs to be hashed, not the transactions too.

4
  • This is Awesome !! Thanks... All is very much clear now.. except the merkle root. binascii.hexlify(hashlib.sha256(hashlib.sha256(binascii.unhexlify('0100000000010100000000..........4e8cf90120000000000000000000000000000000000000000000000000000000000000000000000000')).digest()).digest()) This gives me "cb7ac0ad06517d90cd5a936aebb8b32205fefec268ac6970c04b528313f3ce26" and not "99016ed4b68fa59e3dc602bed62ddaff596cca44b0fd046a35a5a50ec74e96ff"
    – Santan
    Sep 16, 2017 at 18:14
  • That's a witness transaction, so you have to modify it a bit before hashing it to find the txid (you currently found the wtxid). But that's a separate question , remember to accept my answer to your question if it satisfies you btw :) Sep 16, 2017 at 20:49
  • 1
    Yes your answer is perfect, what i was looking for.
    – Santan
    Sep 17, 2017 at 4:19
  • So everything in the submitted header of the block is byte-reversed of actual value? In other words if you want to know the version from the block header, byte-reverse it, and you get the value. This design is so unnecessarily confusing.
    – andrewz
    Jul 26, 2022 at 13:31
0

MeshCollider's answer is awesome. I want to add a note. Why do the previous block header hash and merkle root hash need to reverse bytes? And why other fields doesn't need it?

Actually, if you want any transaction's hash, you need to double-sha256 hash. So, the point is: if some data need sha256 digest, and digest stores in the computer is little-endian, some case need to transform to big int in big-endian, so reverse the bytes. And you get the double-sha256 hash in bitcoin. In some case, the big-endian is necessary. e.g. big-int(block-header-hash) < Mining Target?

If you need transaction hash, the result is big-endian too.

For example, a raw transaction hex is:

0100000001c997a5e56e104102fa209c6a852dd90660a20b2d9c352423edce25857fcd3704000000004847304402204e45e16932b8af514961a1d3a1a25fdf3f4f7732e9d624c6c61548ab5fb8cd410220181522ec8eca07de4860a4acdd12909d831cc56cbbac4622082221a8768d1d0901ffffffff0200ca9a3b00000000434104ae1a62fe09c5f51b13905f07f06b99a2f7159b2225f374cd378d71302fa28414e7aab37397f554a7df5f142c21c1b7303b8a0626f1baded5c72a704f7e6cd84cac00286bee0000000043410411db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5cb2e0eaddfb84ccf9744464f82e160bfa9b8b64f9d4c03f999b8643f656b412a3ac00000000

then, you need transaction hash:

>>> import hashlib
>>> d = '0100000001c997a5e56e104102fa209c6a852dd90660a20b2d9c352423edce25857fcd3704000000004847304402204e45e16932b8af514961a1d3a1a25fdf3f4f7732e9d624c6c61548ab5fb8cd410220181522ec8eca07de4860a4acdd12909d831cc56cbbac4622082221a8768d1d0901ffffffff0200ca9a3b00000000434104ae1a62fe09c5f51b13905f07f06b99a2f7159b2225f374cd378d71302fa28414e7aab37397f554a7df5f142c21c1b7303b8a0626f1baded5c72a704f7e6cd84cac00286bee0000000043410411db93e1dcdb8a016b49840f8c53bc1eb68a382e97b1482ecad7b148a6909a5cb2e0eaddfb84ccf9744464f82e160bfa9b8b64f9d4c03f999b8643f656b412a3ac00000000'
>>> hashlib.sha256(bytes.fromhex(d)).hexdigest()
'240cf324ec3cf59609733e2a45e1408673306be8dcd4caf3067aa9355a0269e3'
>>> hashlib.sha256(bytes.fromhex('240cf324ec3cf59609733e2a45e1408673306be8dcd4caf3067aa9355a0269e3')).hexdigest()
'169e1e83e930853391bc6f35f605c6754cfead57cf8387639d3b4096c54f18f4'

169e1e83e930853391bc6f35f605c6754cfead57cf8387639d3b4096c54f18f4 is little-endian hash, in the bitcoin, all the double-sha256 hash is big-endian.

In fact, this transaction's identifier is big-endian hash: f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16.

When you need to find some info about this transaction, you need input the big-endian hash: https://www.blockchain.com/btc/tx/f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16

and little-endian hash you can find nothing.

Did you get it?

  • previous block header hash need to reverse bytes, because bitcoin need big-endian.
  • current block header hash need to reverse bytes, because bitcoin need big-endian.
  • transaction hash need to reverse bytes, because bitcoin need big-endian.
  • merkle root hash need to reverse bytes, because bitcoin need big-endian.

OMG! why? I found bitcoin core developer didn't know the reason.

Due to historical accident, the tx and block hashes that bitcoin core uses are byte-reversed. I'm not entirely sure why. May be something like using openssl bignum to store hashes or something like that, then printing them as a number.

ref: http://learnmeabitcoin.com/glossary/txid

1
  • There would be no need to check big/little endianes and do byte reversing if all data was stored as bytes instead of some as uint32. For example getblocktemplate provides the bits field as a hexadecimal string, they could have done the same for version, curtime/ntime. This design lacks simplicity and is unnecessarily twisted. It's stupid. Can't believe bitcoin is worth anything.
    – andrewz
    Jul 26, 2022 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.