Issues with stopping bitcoind after 22.0 update

Question

My setup was originally Bitcoin core 0.21.0, with IBD done long time back.

Now for some experimental purposes, I turned on txindex in bitcoin.conf and reindexing was happening for hours.

In mid of it, I stopped the process, upgraded to 22.0 and started 'bitcoind' again. I picked up re-indexing from where it was terminated before. But then when I did 'bitcoin-cli stop', it got stuck with the following logs:

2021-11-25T13:17:05Z UpdateTip: new best=0000000000000000000883a03094dde1221683e17fb3982ca36d84cb114b3eb8 height=605407 version=0x20400000 log2_work=91.374385 tx=478072907 date='2019-11-26T01:00:35Z' progress=0.699403 cache=303.2MiB(2246070txo)
2021-11-25T13:17:05Z UpdateTip: new best=00000000000000000014e207348c17b888af84a47b909006d92f4e764368a8e9 height=605408 version=0x20000000 log2_work=91.374410 tx=478076043 date='2019-11-26T01:10:59Z' progress=0.699408 cache=303.4MiB(2248147txo)
2021-11-25T13:18:45Z tor: Thread interrupt
2021-11-25T13:18:45Z torcontrol thread exit
2021-11-25T13:18:45Z addcon thread exit
2021-11-25T13:18:45Z Shutdown: In progress...
2021-11-25T13:18:45Z opencon thread exit
2021-11-25T13:18:46Z msghand thread exit
2021-11-25T13:18:46Z net thread exit
2021-11-25T13:18:47Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat started
2021-11-25T13:18:47Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat completed (0.29s)
2021-11-25T13:19:54Z scheduler thread exit

I waited for 1-2 hour for it to properly quit. It didn't. So I closed my terminal, wishing it shouldn't be screwing up / corrupt the files!

Gladly it didn't. I then increase dbcache to 4096 (with the hope of increasing the speed of reindexing) and started bitcoind again. Now I again tried terminating with 'bitcoin-cli stop' and again got stop with these logs:

2021-11-25T17:33:01Z UpdateTip: new best=0000000000000000000c7b7267385a7a8b0136a6723380872b6846dfa1b5a802 height=627203 version=0x20000000 log2_work=91.883753 tx=523176359 date='2020-04-22T21:25:52Z' progress=0.765346 cache=2827.6MiB(21174875txo)
2021-11-25T17:33:04Z UpdateTip: new best=0000000000000000000f0b3f604df25db94b1248e69097548baafa7125b1bc65 height=627204 version=0x3fffe000 log2_work=91.883775 tx=523178937 date='2020-04-22T21:32:05Z' progress=0.765350 cache=2827.6MiB(21175427txo)
2021-11-25T17:33:05Z UpdateTip: new best=000000000000000000103cbcbe3cbb20759da63f5c9574d33400ae81e873b4bb height=627205 version=0x20000000 log2_work=91.883796 tx=523182006 date='2020-04-22T21:34:48Z' progress=0.765354 cache=2827.8MiB(21176750txo)
2021-11-25T17:33:06Z tor: Thread interrupt
2021-11-25T17:33:06Z addcon thread exit
2021-11-25T17:33:06Z opencon thread exit
2021-11-25T17:33:06Z torcontrol thread exit
2021-11-25T17:33:06Z Shutdown: In progress...
2021-11-25T17:33:06Z net thread exit
2021-11-25T17:33:09Z UpdateTip: new best=0000000000000000000ff51c79d6d89c3c9368b2c44cc763e104e911137a833d height=627206 version=0x20000000 log2_work=91.883818 tx=523185004 date='2020-04-22T21:57:24Z' progress=0.765359 cache=2828.2MiB(21179795txo)
2021-11-25T17:33:09Z msghand thread exit
2021-11-25T17:33:19Z loadblk thread exit
2021-11-25T17:33:19Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat started
2021-11-25T17:33:19Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat completed (0.27s)
2021-11-25T17:33:19Z scheduler thread exit
2021-11-25T17:33:20Z FlushStateToDisk: write coins cache to disk (21179795 coins, 2965556kB) started

I am not afraid to kill the terminal, since the last line of the log says 'Writing something to disk'. I waited for almost an hour now.

My system monitor doesn't show bitcoind, it shows this process: b-shutoff, which is not doing any active Disk read / writes!

Is it safe to close the Terminal? I don't want to do IBD again! Any measures I can take to avoid this (bitcoind not stopping) in future?

Thanks.

Edit: It ended properly exactly after an hour, once I made this post:

2021-11-25T18:33:41Z FlushStateToDisk: write coins cache to disk (21179795 coins, 2965556kB) completed (3620.90s)
2021-11-25T18:33:43Z FlushStateToDisk: write coins cache to disk (0 coins, 249287kB) started
2021-11-25T18:33:43Z FlushStateToDisk: write coins cache to disk (0 coins, 249287kB) completed (0.17s)
2021-11-25T18:33:48Z [default wallet] Releasing wallet
2021-11-25T18:34:02Z Shutdown: done

I am glad. But any hints to improve this situation?

Answer 1

The message "FlushStateToDisk: write coins cache to disk (21179795 coins, 2965556kB) started" got me here, I am still in the IBD process setting up a new node.

Reason: this flush to disk took a considerate amount of time.. the impatient nature of mind got me looking into a potential problem. Before even writing this post, the IBD already continued.

I was not able to capture the amount of data being written... I know it is huge, as it's the current validation cache flushing to disk triggered by running out of cache space. More to read: What should I do about these messages in Bitcoin-core's debug-log?

To answer your question on preventing this, besides not stopping your bitcoin daemon.. You can either let this process finish itself or the validation will be forced to start from an earlier "checkpoint" (potentially means a re-run of the full chain)

Increasing the "dbcache" I don't consider a viable option. When you increase the cache value, the amount of data to be written when the flush triggers will increase to be a.. big dump :)