bitcoind 23.0 crashed mysteriously after 2 hours, what do I do next?

Question

My new bitcoin node ran for about two hours and then crashed for no apparent reason. What do I do next to debug or fix the problem?

This is my first time running a bitcoin node. I have lots of experience running Linux services in general. I started by doing a fresh install of Ubuntu 20.04.1 on an laptop computer with 8 G of RAM and a 1TB hard drive. I downloaded the bitcoin core 23.0 .tar.gz file and extracted it into /opt . I created a separate "bitcoin" user without sudo privileges in order to run bitcoind. I forwarded port 8333 from my router to the laptop running the bitcoin node. I launched bitcoind as user "bitcoin" by running bitcoind -daemon with no custom configuration.

I watched as bitcoind started the intial block download. I could see it was using a bunch of bandwidth as expected, and waited until it had downloaded a few gigabytes of data onto disk. Then I went to bed.

The next morning I expected to see bandwidth still being used and a lot more blocks stored on disk. Instead, I saw no bandwith being used, and bitcoind was no longer running. The .bitcoind/bitcoind.pid file was still there but the process with the ID in that file was no longer running. The laptop computer is not configured to "go to sleep", and my SSH session from another computer was still connected and had not been interrupted. About 34 G of disk space had been used by the downloaded blocks before bitcoind crashed. I saw a .bitcoin/debug.log file with size 79M and last modification time about two hours after I had started bitcoind. I tailed that file, but saw no error message, only what looked like normal log messages. This is the last line in debug.log:

2022-08-28T05:11:25Z UpdateTip: new best=000000000000000012ef16fc2f2c7f6752a6affa988d98098f837260de9a8989 height=342602 version=0x00000002 log2_work=82.191226 tx=59049857 date='2015-02-08T21:31:10Z' progress=0.077785 cache=716.8MiB(5441897txo)

My guess is that bitcoind hit a bug and exited abnormally. These kinds of problems are a pain to debug and my sympathies go to the developers.

I am willing to take steps to help debug this problem if someone is willing/able to give me some guidance. For example if someone could point to an (official!) debug build that would render a stacktrace (I notice that the bitcoin executables are all stripped). And/or instructions to run a debug build using gdb to capture the crash. Or maybe some extra debug parameters I could run with to log more info - if someone is interested in looking at the results.

Otherwise, if no debug guidance is available, would y'all recommend I downgrade to a non-crashing version? or wait till the next release?

Update 1: I mentioned above that no, my laptop computer did not go to sleep.

Update 2: As I mentioned in a comment below, I'm pretty sure I did not run out of memory. The machine has 8G of RAM, nothing else was running, and when I checked the log for any reported OOM kills by the operating system, there were none. I did see in the logs that at boot time the OOM killer was started.

Update 3: As suggested in a comment below, I tried running in foreground mode with debug logging turned on: bitcoind -nodaemon -debug . I did this on a fresh bitcoin data directory, to restart the initial block download from scratch. This time it ran only for an hour an a half before crashing, with this message:

2022-08-30T20:56:45Z UpdatedBlockTip: new block hash=0000000000000000175471627926fa912a59576262795b54ebd961c3167580c3 fork block hash=0000000000000000203dada33f89ef12aaa3995c390e0f575cf5d80901b79b49 (in IBD=true)
Bus error (core dumped)

Answer 1

I think I found a likely cause of the crashes. I saw some suspicious log messages:

[ 6067.607405] blk_update_request: I/O error, dev sda, sector 155782528 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

This made me start to suspect disk hardware errors. So I ran this check:

$ sudo apt-get install smartmontools
$ sudo smartctl -H /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-46-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
Failed Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   001   001   050    Pre-fail  Always   FAILING_NOW 16376

... and there we have it. Disk failure. All bets are off once the hardware starts failing.