57

Can anyone explain me in the plain English how difficulty is calculated. I have a very approximate understanding that it is calculated based on the amount of hash power in all the bitcoin community over a specific period of time. But this is very vague.

Also I understand it can change very rapidly. Can it only increase? Is there any formula how to calculate it or predict it?

Thanks for a detailed answer, Meni Rosenfeld. Just to make sure I got everything right. I am summing up all the time, it took to generate the last 2016 blocks. And then apply the formula.

  • 2
    I think follow up questions are better as comments to the answer. Basically yes, but no summing is actually needed - you can just take the timestamps of the last block and of the one 2016 blocks before, and subtract. – Meni Rosenfeld Dec 19 '12 at 14:18
73

The Bitcoin difficulty started at 1 (and can never go below that). Then for every 2016 blocks that are found, the timestamps of the blocks are compared to find out how much time it took to find 2016 blocks, call it T. We want 2016 blocks to take 2 weeks, so if T is different, we multiply the difficulty by (2 weeks / T) - this way, if the hashrate continues the way it was, it will now take 2 weeks to find 2016 blocks.

For example, if it took only 10 days it means difficulty is too low and thus will be increased by 40%.

The difficulty can increase or decrease depending on whether it took less or more than 2 weeks to find 2016 blocks. Generally, the difficulty will decrease after the network hashrate drops.

If the correction factor is greater than 4 (or less than 1/4), then 4 or 1/4 are used instead, to prevent the change to be too abrupt.

There is a bug in the implementation, due to which the calculation is based on the time to find the last 2015 blocks rather than 2016. Fixing it would require a hard fork and is thus deferred for now.

It is possible to give a rough estimate for the next difficulty change, based on the time to find the recent blocks. Nobody can make longer-term predictions for the future difficulty reliably, but anyone is free to speculate based on exchange rate trends, Moore's law and other hardware advances.

  • 1
    @StevenRoose: AFAIK it is, but I will leave it to people who are more involved with the core code to comment... This is adequate for a separate SE question. – Meni Rosenfeld Jun 12 '13 at 15:19
  • 2
    Good answer, but one small, but capital point is eluded: how do the nodes in the network agree on what is the difficulty ? – deadalnix Mar 14 '14 at 18:35
  • 5
    @deadalnix: The difficulty of a block is a deterministic calculation based on the data of the previous blocks. All nodes independently do the same calculation and get the same result. – Meni Rosenfeld Mar 15 '14 at 21:36
  • 2
    @deadalnix: The timestamp is a part of the block, which means that whoever found the block decides what to put in it. The timestamp must be no sooner than the median of the past 11 blocks. Also, if a node receives a block with a timestamp more than 2 hours in the future it will reject it and not propagate it. – Meni Rosenfeld Mar 16 '14 at 16:10
  • 2
    @tobi: Oh ok. Previous comments were about the bug, and "errors" hinted that it's the bug, so I assumed we're talking about that. So, yes. If we assume that around 2140 the hashrate will be around *1B what it is now, the schedule will be ahead by 96 weeks, or almost two years. But there is yet another effect - a delay caused by the fact that in the beginning, the difficulty was 1 even though the hashrate wasn't enough to justify it. – Meni Rosenfeld Jun 5 '18 at 13:15
13

Meni's answer is good. I just want to give some practical detail method about difficulty calculation, perhaps helpful for future views of this question's answer.

Let's take a look at Satoshi's genesis block header (part of related info):

$ bitcoin-cli getblockhash 0
000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f

$ bitcoin-cli getblockheader 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
{
  ...
  "height": 0,
  ...
  "bits": "1d00ffff",
  "difficulty": 1,
  ...
}

As we can see above, the genesis block has a '1' difficulty and '1d00ffff' bits. The bitcoin bits means the 'target' hash value, the new generated block must meet a condition: block header's double SHA-256 hash value must less than this 'target' value.

The '1d00ffff' bits value in genesis block means the 'target' value:

[0x00000000,0xffff,{0x00..0x00}]
                   {0x00..0x00} at above has 26 bytes 0x00.

Then, to find a new block, you must search that 32 bits nNonce value (and nTimes and the hashMerkleRoot also) until the block hash value has 4 bytes zero leading. By the way, the nNonce is one of the fields in block header structure:

 struct header_structure{            // BYTES   NAME
     uint32_t nVersion;              // 4       version
     uint8_t hashPrevBlock[32];      // 32      previous block header hash
     uint8_t hashMerkleRoot[32];     // 32      merkle root hash
     uint32_t nTime;                 // 4       time
     uint32_t nBits;                 // 4       target
     uint32_t nNonce;                // 4       nonce
 };

Because SHA-256 algorithm (as well as any cryptographically secure hash algorithm) produces output that will appear like an uniformly random sequence, the practical 'trial and error' method is the only way to find a new block to meet the condition. The probability to find a block with the 4 bytes zero leading hash value is 1/(2^32), that means the average 'trial and error" numbers are exactly 2^32 (i.e. 4G).

For human easy understanding about this 'target' hash value, We define the term 'difficulty', which means the average 'trial and error" numbers to find a block to meet the 'target' condition. And we define the 'difficulty' unit: 1 'difficulty' = 4G hashes

Then, till today, the bitcoin blockchain height reach 501509, let's take a look at its header:

$ bitcoin-cli getblockheader 0000000000000000006c5532f4fd9ee03e07f94df165c556b89c495e97680147
{
  ...
  "height": 501509,
  ...
  "bits": "18009645",
  "difficulty": 1873105475221.611,
  ...
}

The block 501509's bits = 0x18009645, it's the compact format of 256 bits integer, its 256 bits format is:

[0x00000000,0x00000000,0x009645,{0x00..0x00}]
                                {0x00..0x00} at above has 21 bytes 0x00.
that is  0x009645 * (256 ^ 21) 
The genesis block's target is  ( 0x00ffff * 256 ^ 26 )which is the difficulty unit '1.0'.
So, the difficulty 
= (0x00ffff * 256 ^ 26)/ (0x009645 * 256 ^ 21)
= 65535/38469 * (256^5)
= 1.703579505575918 * 2^40
= 1873105475221.611

So far, you have all the detail about how to calculate the 'difficulty'. In some cases, we also use the simple format 1.7T to say the difficulty, in above example:

 (1.703579505575918 * 2^40) = 1.703579505575918T
 1T = 2^40 = 1024^4
  • 1d is 29 in Dec(not 26). SHS is SHA – Boris Ivanov Aug 6 '18 at 18:31
  • thanks @BorisIvanov, the typo error SHS has been fixed. But 1d indeed means 26 bytes zero tail instead of 29, please read the example detail showed above. – gary Aug 7 '18 at 8:13
  • ah yeah. Significand – Boris Ivanov Aug 7 '18 at 15:29
1

I would like to give my 2 cents here, by expliciting the relationship between the probability of mining a block given the current target t and the corresponding difficulty d as it is calculated in bitcoin core.

So cryptographic hash functions are idealized by the random oracle abstraction [https://en.wikipedia.org/wiki/Random_oracle]. We can therefore model the output of the doubleSHA256 hash function used in PoW as a uniform variable in the space {0,1}^256, i.e. arrays of 256 bits. Thus the probability of a single hash h being a valid hash is:

p = P(h < t) = t /( 2^{256} - 1 )

On the other hand d is calculated as follows, just as @gary explained before only transformed into decimals:

d = ( (2^{16} - 1) * 2^{8*26} ) / t = ( (2^{16} -1) * 2^{208} ) / t

The implementation is in [https://github.com/bitcoin/bitcoin/blob/master/src/rpc/blockchain.cpp], line 60, function GetDifficulty. Actually if someone can explain how exactly the code maps to the formula above, that would be helpful. Combining those two formulas we obtain:

d = ( (2^{16} -1) * 2^{208} ) / ( p * (2^{256} - 1) ) ~ 2^{-32} / p

Analyzing this last expression the difficulty is the ratio between the probability of obtaining a hash lower than 2^{224} (which is the lowest decimal number that has a binary representation using 256 bits starting with 32 zero bits) and the probability of obtaining a valid hash based on the current target t. This is a direct implication of defining, in the genesis block, as difficulty 1 the one associated to the hexadecimal target 0x1d00ffff, expressed in what I think is called the 32-bit compact form for 256-bit numbers.

A nice question I believe is why this specific compact form was chosen for representing the target.

  • Upvoted! The compact form provides 3 most significant bytes for the target, in the min difficulty the 3 most significant bytes are 00ffff. – James C. Jan 27 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.