bitcoind resumes from low block after restart

Question

I have bitcoind full node running, that is usually fully in sync with the chain. However, if it is restarted, e.g. due to machine reboot, it does not start from the top of the chain, but from somewhat behind:

Switching active chainstate to Chainstate [ibd] @ height -1 (null)
Opening LevelDB in /home/bitcoin/.bitcoin/blocks/index
Opened LevelDB successfully
Using obfuscation key for /home/bitcoin/.bitcoin/blocks/index: 0000000000000000
LoadBlockIndexDB: last block file = 2764
LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=34, size=46998614, heights=705023...705056, time=2021-10-14...2021-10-15)
Checking all blk files are present...
Opening LevelDB in /home/bitcoin/.bitcoin/chainstate
Opened LevelDB successfully
Using obfuscation key for /home/bitcoin/.bitcoin/chainstate: aab110aad4c72780
Loaded best chain: hashBestChain=0000000000000000000afcc2f0c30a3b66ac9e1ae1d98f86ff83c1586ed0f35a height=671403 date=2021-02-20T10:30:30Z progress=0.895949
init message: Rewinding blocks...
FlushStateToDisk: write coins cache to disk (0 coins, 0kB) started
FlushStateToDisk: write coins cache to disk (0 coins, 0kB) completed (0.00s)
init message: Verifying blocks...
Verifying last 6 blocks at level 3
[0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
No coin database inconsistencies in last 6 blocks (14014 transactions)
 block index           20504ms
Opening LevelDB in /home/bitcoin/.bitcoin/indexes/txindex
Opened LevelDB successfully
Using obfuscation key for /home/bitcoin/.bitcoin/indexes/txindex: 0000000000000000
txindex thread start
txindex is enabled at height 671403
txindex thread exit
block tree size = 705057
nBestHeight = 671403
loadblk thread start
torcontrol thread start

So, the chain is at 705057 but bitcoind restarts from 671403. Then it catches up back to the top eventually and proceeds normally.

Interestingly enough, this happens on every restart - i.e. if I restart it again it goes back to 671403 again. I am not sure what is magical about this number - the configuration is pretty standard and we aren't using any special options - or why this happens to the node. Any ideas about why it happens and how to make it not reset itself?

The version is:

Bitcoin Core version v0.21.0 (release build)

Answer 1

Bitcoin Core aggressively caches updates to the UTXO database ("chainstate") during its operation, and only occasionally flushes the resulting changes in batch to disk. This happens (a) at shutdown, (b) once per day, or (c) whenever Bitcoin Core runs out of memory (configurable with dbcache= setting).

However, when Bitcoin Core is shutdown uncleanly (crash, power failure, ...), it is possible that the database wasn't flushed, and thus the on-disk version still corresponds to an older state. This isn't a problem, as it gets detected, and the blocks which were downloaded after the last flush get re-validated at startup.

To investigate, it's useful to know what's in your debug.log before you restart (i.e., the last lines added in the previous run).