bitcoin daemon crashing on initial block download

Question

I'm trying to do the initial block download so I can run a full node. Didn't experience any issues until the download got to about 87% then bitcoind crashed without logging any errors. I've tried restarting it several times, however each time when I come back to my computer a few hours later to check on it I find that it has crashed and any download progress has been lost.

I am doing this on my personal laptop, using a 1 terabyte external hard drive (not SSD) and a wired Ethernet connection. I have disabled sleep on my computer and have been leaving it plugged into power.

Details:

• OS: Ubuntu 22.04.5 LTS
• CPU architecture: x86_64
• CPU: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
• RAM: 16gb
• Internet speed: 90 Mbit/s download
• Bitcoin Core version: v27.0.0

This is the command I'm using to run the daemon:

bitcoind -daemon -debug -datadir=/media/me/TOSHIBA\ EXT/bitcoin/ -dbcache=10000

This is the last 25 lines from the debug.log file before it crashed:

2024-09-14T03:44:04Z [validation] Enqueuing MempoolTransactionsRemovedForBlock: block height=830276 txs removed=0
2024-09-14T03:44:04Z UpdateTip: new best=00000000000000000001914a6db5b53b3f8d673fbd9ebc1a569f37ccbbf97861 height=830276 version=0x20c98000 log2_work=94.729419 tx=965218034 date='2024-02-13T10:37:40Z' progress=0.888359 cache=2550.2MiB(20688484txo)
2024-09-14T03:44:04Z [bench]   - Connect postprocess: 0.40ms [3.08s (0.70ms/blk)]
2024-09-14T03:44:04Z [bench] - Connect block: 2522.60ms [6781.75s (1545.87ms/blk)]
2024-09-14T03:44:04Z [validation] Enqueuing BlockConnected: block hash=00000000000000000001914a6db5b53b3f8d673fbd9ebc1a569f37ccbbf97861 block height=830276
2024-09-14T03:44:04Z [validation] Enqueuing UpdatedBlockTip: new block hash=00000000000000000001914a6db5b53b3f8d673fbd9ebc1a569f37ccbbf97861 fork block hash=000000000000000000018e3f91a815472d0d994b832624751bbfbac25aa739cf (in IBD=true)
2024-09-14T03:44:04Z [validation] MempoolTransactionsRemovedForBlock: block height=830273 txs removed=0
2024-09-14T03:44:04Z [estimatefee] Blockpolicy estimates updated by 0 of 0 block txs, since last block 0 of 0 tracked, mempool map size 0, max target 0 from current
2024-09-14T03:44:04Z [validation] BlockConnected: block hash=000000000000000000031be2ac7b85a7db0493498a39f4580be01431134e4c13 block height=830273
2024-09-14T03:44:04Z [validation] UpdatedBlockTip: new block hash=000000000000000000031be2ac7b85a7db0493498a39f4580be01431134e4c13 fork block hash=00000000000000000003238c6e2e4551b0b5375ae52499bf03bf5de12a29e2bf (in IBD=true)
2024-09-14T03:44:04Z [validation] MempoolTransactionsRemovedForBlock: block height=830274 txs removed=0
2024-09-14T03:44:04Z [estimatefee] Blockpolicy estimates updated by 0 of 0 block txs, since last block 0 of 0 tracked, mempool map size 0, max target 0 from current
2024-09-14T03:44:04Z [validation] BlockConnected: block hash=00000000000000000000f3e8fcccf8ee483ebed02694c5773780a61a17a95037 block height=830274
2024-09-14T03:44:04Z [validation] UpdatedBlockTip: new block hash=00000000000000000000f3e8fcccf8ee483ebed02694c5773780a61a17a95037 fork block hash=000000000000000000031be2ac7b85a7db0493498a39f4580be01431134e4c13 (in IBD=true)
2024-09-14T03:44:04Z [validation] MempoolTransactionsRemovedForBlock: block height=830275 txs removed=0
2024-09-14T03:44:04Z [estimatefee] Blockpolicy estimates updated by 0 of 0 block txs, since last block 0 of 0 tracked, mempool map size 0, max target 0 from current
2024-09-14T03:44:04Z [validation] BlockConnected: block hash=000000000000000000018e3f91a815472d0d994b832624751bbfbac25aa739cf block height=830275
2024-09-14T03:44:04Z [validation] UpdatedBlockTip: new block hash=000000000000000000018e3f91a815472d0d994b832624751bbfbac25aa739cf fork block hash=00000000000000000000f3e8fcccf8ee483ebed02694c5773780a61a17a95037 (in IBD=true)
2024-09-14T03:44:04Z [validation] MempoolTransactionsRemovedForBlock: block height=830276 txs removed=0
2024-09-14T03:44:04Z [estimatefee] Blockpolicy estimates updated by 0 of 0 block txs, since last block 0 of 0 tracked, mempool map size 0, max target 0 from current
2024-09-14T03:44:04Z [validation] BlockConnected: block hash=00000000000000000001914a6db5b53b3f8d673fbd9ebc1a569f37ccbbf97861 block height=830276
2024-09-14T03:44:04Z [validation] UpdatedBlockTip: new block hash=00000000000000000001914a6db5b53b3f8d673fbd9ebc1a569f37ccbbf97861 fork block hash=000000000000000000018e3f91a815472d0d994b832624751bbfbac25aa739cf (in IBD=true)
2024-09-14T03:44:04Z [bench]   - Load block from disk: 194.05ms
2024-09-14T03:44:04Z [bench]     - Sanity checks: 4.44ms [12.23s (2.79ms/blk)]
2024-09-14T03:44:04Z [bench]     - Fork checks: 0.04ms [0.14s (0.03ms/blk)]

Things I've already tried:

• run the daemon with debugging enabled and check the debug.log file for obvious problems
• check if the external hard drive is corrupted: I've run some tests using smartctl and badblocks and they haven't reported any issues
• experiment with different values for dbcache
• look up the last block hash listed in the debug.log file in a public block explorer to verify it exists
• look at the bitcoin source code where the last log comes from (validation.cpp line 2592). If I had to guess I'd say maybe something between that and the next log on line 2683 is causing bitcoind to crash without logging anything.

I would really appreciate any help with this. I'm finding it really hard to debug because I have to leave my computer running overnight to see if it crashes every time I try something new.

Answer 1

Id try running a 'pruned' node to see whether its a problem with hard drive space.

bitcoind -prune=100

This will download the latest blocks whilst keeping total blockchain size < 100Mb.

If you get this working you know its a size issue. Should be faster for debugging too.

Id also invest in a SSD for better performance if your running it longer term.

Answer 2

i've had this same heisenbug for a longtime with a similar setup and think have found a workaround:

i've been accumulating datfiles on and off for years on one particular laptop and have been having the same problem across different bitcoind versions for a while using precompiled bitcoind binaries, always kept up to date. if there are a lot of blocks to download and verify, it will most times, not always, segfault well into syncing. i could run the daemon, let it accumulate about a months' worth of blocks, then stop it normally and restart it, and bootstrap my way to the current tip like that avoiding segfaults, but that's not ideal.

the laptop has two (internal) ssd's, the executable and conffile are on one, and the datadir on the other, hence i specify the datadir in the conf file.

i suspected hardware failure or corrupt datfile somewhere. after running memtest86 (32gb ram) and ruling out ram, i deleted the datadir for a fresh run with v28.0, but got the same segfaults (never at the same point in the download and verification, usually ~300k, 400k or 500k blocks in, and i'm not going to bootstrap on and off from there!).

i tried gcc compiling source tagged v28.0 with --disable-wallet to see if that made a difference. i tried compiling bleeding edge master also with and without wallet. each executable segfaulted at some point well into the initial download (eg 9hrs). never got anything from a strace. on one occasion i ran it under gdb and it segfaulted at /usr/include/c++/14.2.1/bits/hashtable.h:2071

however i really doubt it was an issue in the c++ standard library.

anyway, i then compiled v28.0 using clang instead of gcc, and haven't experienced the problem again.

the precompiled binaries are compiled using gcc.