My bitcoind daemon sometimes takes an hour to go past Opening LevelDB in .../.bitcoin/chainstate
After googling a bit, i thought it was simply an unclean daemon stop causing a leveldb corruption. But then this happened again this week despite having been stopped correctly (as indicated by Shutdown: done log entries). I was using systemctl stop, not bitcoin-cli stop if that makes any difference.
$ journalctl -u snap.bitcoind.daemon --since "1 week ago" | grep -B1 -E "Opened LevelDB successfully|Shutdown: done"
...
Nov 12 02:09:34 localhost bitcoind.daemon[13878]: 2019-11-12T07:09:34Z Opening LevelDB in /root/snap/bitcoind/common/.bitcoin/blocks/index
Nov 12 02:09:34 localhost bitcoind.daemon[13878]: 2019-11-12T07:09:34Z Opened LevelDB successfully
--
Nov 12 02:09:39 localhost bitcoind.daemon[13878]: 2019-11-12T07:09:39Z Opening LevelDB in /root/snap/bitcoind/common/.bitcoin/chainstate
Nov 12 02:09:39 localhost bitcoind.daemon[13878]: 2019-11-12T07:09:39Z Opened LevelDB successfully
--
Nov 12 02:28:16 localhost bitcoind.daemon[13878]: 2019-11-12T07:28:16Z scheduler thread interrupt
Nov 12 02:28:17 localhost bitcoind.daemon[13878]: 2019-11-12T07:28:17Z Shutdown: done
--
Nov 12 02:28:18 localhost bitcoind.daemon[14353]: 2019-11-12T07:28:18Z Opening LevelDB in /root/snap/bitcoind/common/.bitcoin/blocks/index
Nov 12 02:28:18 localhost bitcoind.daemon[14353]: 2019-11-12T07:28:18Z Opened LevelDB successfully
--
Nov 12 02:28:25 localhost bitcoind.daemon[14353]: 2019-11-12T07:28:25Z Opening LevelDB in /root/snap/bitcoind/common/.bitcoin/chainstate
Nov 12 02:28:25 localhost bitcoind.daemon[14353]: 2019-11-12T07:28:25Z Opened LevelDB successfully
--
Nov 12 06:57:36 localhost bitcoind.daemon[14353]: 2019-11-12T11:57:36Z scheduler thread interrupt
Nov 12 06:57:37 localhost bitcoind.daemon[14353]: 2019-11-12T11:57:37Z Shutdown: done
--
Nov 12 06:57:39 localhost bitcoind.daemon[18691]: 2019-11-12T11:57:39Z Opening LevelDB in /root/snap/bitcoind/common/.bitcoin/blocks/index
Nov 12 06:57:39 localhost bitcoind.daemon[18691]: 2019-11-12T11:57:39Z Opened LevelDB successfully
--
Nov 12 06:57:51 localhost bitcoind.daemon[18691]: 2019-11-12T11:57:51Z Opening LevelDB in /root/snap/bitcoind/common/.bitcoin/chainstate
Nov 12 06:57:52 localhost bitcoind.daemon[18691]: 2019-11-12T11:57:52Z Opened LevelDB successfully
--
Nov 17 13:27:36 localhost bitcoind.daemon[18691]: 2019-11-17T18:27:36Z scheduler thread interrupt
Nov 17 13:27:38 localhost bitcoind.daemon[18691]: 2019-11-17T18:27:38Z Shutdown: done
--
Nov 17 13:31:00 localhost bitcoind.daemon[18236]: 2019-11-17T18:31:00Z Opening LevelDB in /root/snap/bitcoind/common/.bitcoin/blocks/index
Nov 17 13:31:00 localhost bitcoind.daemon[18236]: 2019-11-17T18:31:00Z Opened LevelDB successfully
--
Nov 17 13:31:11 localhost bitcoind.daemon[18236]: 2019-11-17T18:31:11Z Opening LevelDB in /root/snap/bitcoind/common/.bitcoin/chainstate
Nov 17 14:32:53 localhost bitcoind.daemon[18236]: 2019-11-17T19:32:53Z Opened LevelDB successfully
This delay coincides with a spike in memory usage that i only experience in this situation. Otherwise, the daemon runs happily with 512M RAM and no swapping, albeit slowly of course.
# swapon --show=NAME,USED,PRIO
NAME USED PRIO
/dev/vda2 511.2M 1
/dev/dm-2 441.4M -2
It is not a tragic situation since the daemon eventually starts. But I'd like to understand whether this is happening because of something I'm doing wrong. I'm happy to enable debug logs to provide more info if needed.
