How is difficulty calculated?

Question

Can anyone explain me in the plain English how difficulty is calculated. I have a very approximate understanding that it is calculated based on the amount of hash power in all the bitcoin community over a specific period of time. But this is very vague.

Also I understand it can change very rapidly. Can it only increase? Is there any formula how to calculate it or predict it?

Thanks for a detailed answer, Meni Rosenfeld. Just to make sure I got everything right. I am summing up all the time, it took to generate the last 2016 blocks. And then apply the formula.

Answer 1

The Bitcoin difficulty started at 1 (and can never go below that). Then for every 2016 blocks that are found, the timestamps of the blocks are compared to find out how much time it took to find 2016 blocks, call it T. We want 2016 blocks to take 2 weeks, so if T is different, we multiply the difficulty by (2 weeks / T) - this way, if the hashrate continues the way it was, it will now take 2 weeks to find 2016 blocks.

For example, if it took only 10 days it means difficulty is too low and thus will be increased by 40%.

The difficulty can increase or decrease depending on whether it took less or more than 2 weeks to find 2016 blocks. Generally, the difficulty will decrease after the network hashrate drops.

If the correction factor is greater than 4 (or less than 1/4), then 4 or 1/4 are used instead, to prevent the change to be too abrupt.

There is a bug in the implementation, due to which the calculation is based on the time to find the last 2015 blocks rather than 2016. Fixing it would require a hard fork and is thus deferred for now.

It is possible to give a rough estimate for the next difficulty change, based on the time to find the recent blocks. Nobody can make longer-term predictions for the future difficulty reliably, but anyone is free to speculate based on exchange rate trends, Moore's law and other hardware advances.

Answer 2

Meni's answer is good. I just want to give some practical detail method about difficulty calculation, perhaps helpful for future views of this question's answer.

Let's take a look at Satoshi's genesis block header (part of related info):

$ bitcoin-cli getblockhash 0
000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f

$ bitcoin-cli getblockheader 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
{
  ...
  "height": 0,
  ...
  "bits": "1d00ffff",
  "difficulty": 1,
  ...
}

As we can see above, the genesis block has a '1' difficulty and '1d00ffff' bits. The bitcoin bits means the 'target' hash value, the new generated block must meet a condition: block header's double SHA-256 hash value must less than this 'target' value.

The '1d00ffff' bits value in genesis block means the 'target' value:

[0x00000000,0xffff,{0x00..0x00}]
                   {0x00..0x00} at above has 26 bytes 0x00.

Then, to find a new block, you must search that 32 bits nNonce value (and nTimes and the hashMerkleRoot also) until the block hash value has 4 bytes zero leading. By the way, the nNonce is one of the fields in block header structure:

struct header_structure{            // BYTES   NAME
    uint32_t nVersion;              // 4       version
    uint8_t hashPrevBlock[32];      // 32      previous block header hash
    uint8_t hashMerkleRoot[32];     // 32      merkle root hash
    uint32_t nTime;                 // 4       time
    uint32_t nBits;                 // 4       target
    uint32_t nNonce;                // 4       nonce
};

Because SHA-256 algorithm (as well as any cryptographically secure hash algorithm) produces output that will appear like an uniformly random sequence, the practical 'trial and error' method is the only way to find a new block to meet the condition. The probability to find a block with the 4 bytes zero leading hash value is 1/(2^32), that means the average 'trial and error" numbers are exactly 2^32 (i.e. 4G).

For human easy understanding about this 'target' hash value, We define the term 'difficulty', which means the average 'trial and error" numbers to find a block to meet the 'target' condition. And we define the 'difficulty' unit: 1 'difficulty' = 4G hashes

Then, till today, the bitcoin blockchain height reach 501509, let's take a look at its header:

$ bitcoin-cli getblockheader 0000000000000000006c5532f4fd9ee03e07f94df165c556b89c495e97680147
{
  ...
  "height": 501509,
  ...
  "bits": "18009645",
  "difficulty": 1873105475221.611,
  ...
}

The block 501509's bits = 0x18009645, it's the compact format of 256 bits integer, its 256 bits format is:

[0x00000000,0x00000000,0x009645,{0x00..0x00}]
                                {0x00..0x00} at above has 21 bytes 0x00.
that is  0x009645 * (256 ^ 21） 
The genesis block's target is  ( 0x00ffff * 256 ^ 26 ）which is the difficulty unit '1.0'.
So, the difficulty 
= (0x00ffff * 256 ^ 26）/ (0x009645 * 256 ^ 21)
= 65535/38469 * (256^5)
= 1.703579505575918 * 2^40
= 1873105475221.611

So far, you have all the detail about how to calculate the 'difficulty'. In some cases, we also use the simple format 1.7T to say the difficulty, in above example：

(1.703579505575918 * 2^40) = 1.703579505575918T
1T = 2^40 = 1024^4

Answer 3

I would like to give my 2 cents here, by expliciting the relationship between the probability of mining a block given the current target t and the corresponding difficulty d as it is calculated in bitcoin core.

So cryptographic hash functions are idealized by the random oracle abstraction [https://en.wikipedia.org/wiki/Random_oracle]. We can therefore model the output of the doubleSHA256 hash function used in PoW as a uniform variable in the space {0,1}^256, i.e. arrays of 256 bits. Thus the probability of a single hash h being a valid hash is:

p = P(h < t) = t /( 2^{256} - 1 )

On the other hand d is calculated as follows, just as @gary explained before only transformed into decimals:

d = ( (2^{16} - 1) * 2^{8*26} ) / t = ( (2^{16} -1) * 2^{208} ) / t

The implementation is in [https://github.com/bitcoin/bitcoin/blob/master/src/rpc/blockchain.cpp], line 60, function GetDifficulty. Actually if someone can explain how exactly the code maps to the formula above, that would be helpful. Combining those two formulas we obtain:

d = ( (2^{16} -1) * 2^{208} ) / ( p * (2^{256} - 1) ) ~ 2^{-32} / p

Analyzing this last expression the difficulty is the ratio between the probability of obtaining a hash lower than 2^{224} (which is the lowest decimal number that has a binary representation using 256 bits starting with 32 zero bits) and the probability of obtaining a valid hash based on the current target t. This is a direct implication of defining, in the genesis block, as difficulty 1 the one associated to the hexadecimal target 0x1d00ffff, expressed in what I think is called the 32-bit compact form for 256-bit numbers.

A nice question I believe is why this specific compact form was chosen for representing the target.