Would it be possible to retarget difficulty in real time?

Question

Could the mining difficulty adjust in real time, not only when blocks are found?

(NOTE: Not asking about retargetting after every block but continuously as a function of time.)

I'm worried about how Bitcoin (and many other cryptos) change the difficulty. The basic idea is that the difficulty is adjusted every N blocks (N=2016 for Bitcoin and N=1 for Monero). That is, a difficulty can change only when a block is found. The problem is that if hashrate suddenly drops significantly, say 99%, then it would take 100x more time than normally to reach new lower difficulty. The drop could happen if the price of the coin drops and thus mining isn't anymore profitable. Network slows down and the price goes even more down and thus even more miners stop mining, and thus the coin might die. This seems like a very unstable system to me. I don't see much incentive for anyone to mine bitcoin if the price drops dramatically. The only incentive for some people to mine would be that they have lots of coins and they don't want the system to die. Not sure if this is enough.

To avoid this problem, could the difficulty adjust in real time, not only when blocks are found? That is, the difficulty requirement for a valid block would depend on the time it was found. The difficulty would decrease as a function of time, thus eventually a block will be found very easily. For instance, if 2 hours has passed since the last block, the difficulty could have already dropped to 1% of what it has been for the previous blocks.

The difficulty requirement of block N at time T would be a decreasing function of T and depends on the timestamps and difficulties of past blocks somehow. For instance, the difficulty could be a exponentially decaying function of time since the 10th last block and the mean difficulty of those 10 blocks. The difficulty of a mined block must be at least as great as given by this function in order for it to be valid. So the timestamp of a block becomes an important part of the block data.

Would this idea make sense? Is it implemented in any coin? Or is there some reason why it is not a problem that the difficulty adjusts when a block is found?

Answer 1

It's a security property that the difficulty is not retargeted quickly, not a technical or design limitation. It prevents a case where you can isolate a node, or a portion of the network, wait a brief period, and then mine at a vastly lower difficulty in order to defraud those users. If the mining rate drops, the fees rise, and more miners are incentivized to join the network- it's a self balancing system.

Answer 2

I believe the answer is an emphatic YES, but just after a block has been found I would not immediately start applying the function-of-time difficulty reduction. I believe that would not be desirable, because mining is a random process. One should allow for the natural variability of the process before judging it to be "out of balance" and needing extra-ordinary adjusment. (Think of mining as tossing a dice with 2^256 faces, but only a subset of those faces are winners for a given toss.) It obeys these laws of probability and statistics:

• The number of blocks in an interval follows the Poisson distribution
• The time between consecutive blocks follows the Exponential distribution Probability of a block being found in time less than or equal to "t" is: P(X≤t) = 1-e^(-λt) where λ = 1/β , and β = target interval time

Applying the exponential probability function in the case of bitcoin, the likelihood of a block being found in 10 minutes or less is:

1-e^(-(1/600 seconds) * 600 seconds) = 1-e^(-1) = 63.2120558829%

The likelihood of a block being found in 60 minutes or less is: 99.7521247823% .... and If we go out to 15,000 seconds (about 4.17 hours), the likelihood of a block having been found is 99.9999999986

Another way to state this is that, in 99.9999999986% of all cases, a block SHOULD HAVE BEEN FOUND by 4.17 hours; it is therefore a very sure bet, if the block has not been found, that there is, as you notice could happen, a serious mismatch between the true network hash rate and the difficulty factor of mining.

An arbitrary threshold time can be chose, beyond the which, if a block has not been found, the mining difficulty factor is to be lowered. Choosing this threshold is equivalent to asking 'How sure do you want to be that the system is unbalanced ... before "messing" with it ? '

Only if we're fairly certain that the system is really unbalanced (meaning difficulty is not matched to the current real network hashrate) should we start executing a protocol or algorithm, of 'extraordinary' difficulty reduction. It is at this point that I would consider the idea of an exponentially decaying-with-time difficulty function.

All this is predicated on accurate, shared time consensus. Establishing what the correct time is among a number of machines on the internet is not a trivial problem, but fortunately, has been solved since the mid 80's by the Network Time Protocol (NTP), and since the 1990's complemented by the inexpensive availability of very precise and globally availabe GPS satellite time signals.

(NTP was developed by Dr. Dave Mills at the University of Delaware and a new, streamlined and security-hardened version of NTPSec is by Eric S. Raymond and his group). The correct time is not a matter of opinion, but of astronomical fact. Building a strong consensus of the best expression of current real time is NTP's job, and it does it quite well.

So, assuming that honest nodes, the mining "true chimers", have and use the correct real time, they will all know when a block is overdue, and according to the coin's protocol, can then agree that it is "too long overdue": current wait for the next block has exceeded the pre-established trigger wait threshold. Extraordinary measures must be taken. The question now becomes how much to reduce the difficulty factor.

The difficulty factor must be set to match network hashrate, BUT since no blocks are being produced, curent network hashrate can not be calculated : what must follow is a guessing stage, where a first difficulty reduction is made, then the network waits a reasonable time for a block to be produced. But, if again, no blocks are produced, then another difficulty reduction has to be made, since clearly the initial reduction was insufficient.

This protocol for difficulty reduction, (let's call it extraordinary difficulty reduction mode or 'EDRed mode'), may have many variations. I would suggest the first reduction be maybe 1/2 to 2/3, and that he waiting period for the next block after 1st reduction be a shorter wait than the first trigger threshold. Eventually, after 1, 2 or maybe even 3 or 4 reductions, blocks will start to be produced again, because the network's mining difficulty factor will be matched to the real network hash rate. It is even likely that the difficulty will be a bit too "easy" since substantial, "guessing" reductions were done - but the regular difficulty adjusting algos (Kimoto's Gravity Well or Dark Gravity Wave for example) used in alt-coins are quite good at quickly raising difficulty to more precisely match hashrate.

Once blocks are coming out again (that is, 2 or 3 blocks have been produced without another EDRed mode reduction being triggered), then the usual KGW or DGW algos can be allowed to govern difficulty again and EDRed mode is canceled.

(Bitcoin may have to adopt a more sophisticated difficulty-adjusting algo for its regular operation to implement a real-time-aware difficulty reduction protocol like this. I believe the recent Bitcoin Cash fork did adopt a more responsive algo than the simple adjustment every 2016 blocks of Bitcoin Core)

Answer 3

It is not trivial to retarget difficulty in real time, because it's non-trivial to achieve consensus on the current time of the network. It's sort of a catch-22. In order to introduce a real-time adaption into our consensus system, we would need to achieve consensus on something else before.

---

I realized after writing my answer that you explicitly exclude retargetting on every block, but since it's related, I'll leave it here anyway:

What could be done instead, would be to adjust the difficulty on every block. The challenges hereby are to find parameters that are clearly better: there is e.g. a trade-off between making the adjustment fast enough to make it useful, and so fast that it facilitates eclipse attacks as mentioned by Anonymous's answer.

Another subtle issue is that a readjustment on every block with the same best-chain rules would simplify reorganization attacks.

The attacker would go about this in the following fashion: To reorganize a block 1A, they would search for a competing block 1B that has a slightly earlier date than 1A. However minuscule the adjustment on every block, this would mean that a follow-up block 2B would have a slightly higher difficulty score than the honest block 2A.

honest chain: 0 ← 1A ← 2A
attacker:       ↖ 1B ← 2B

Thus, when the attacker finds a second block, their chain would be the heaviest and the network would reorganize to that chain-tip even when 2A was found first. Previously, due to matching difficulty across the whole difficulty period, this attack would have been harder, as the attacker would have to find 2B before the honest miners author 2A.