Where can I find a usefull log if bitcoind stopped for no apparent reason?

Question

I am running bitcoind (Bitcoin Core) on a Raspberry 3B and the process occationally "disappears" from the list in top. The log debug.log shows no etries, last entry is usually something like

2019-11-08T20:21:33Z UpdateTip: new best=0000000000000000001dd32ef554a1bb9e204d6f46ab345302019412ac2f6af9 height=506280 version=0x20000000 log2_work=87.972007 tx=295647959 date='2018-01-27T01:16:11Z' progress=0.631698 cache=13.3MiB(59241txo)

What could be a reason and where could I search?

My config for my low-memory-system is:

 $ cat bitcoin.conf
daemon=1
#printtoconsole=1

# is required by Fail2Ban described below
logips=1
# magic RBP optimisations
maxconnections=40
maxuploadtarget=5000

# running without swap
dbcache=100
maxorphantx=10
maxmempool=50

txindex=1

rpcuser=...
rpcpassword=...
rpcclienttimeout=300
server=1
#datadir=/home/bitcoin

Maybe I have a problem with my external HDD:

/var/log/kern.log shows

Nov 17 21:23:22 RPi3 kernel: [642589.539592] EXT4-fs error (device sdb1): ext4_find_entry:1455: inode #2: comm smbd: reading directory lblock 0
Nov 17 21:23:22 RPi3 kernel: [642589.539653] EXT4-fs error (device sdb1): ext4_find_entry:1455: inode #2: comm smbd: reading directory lblock 0
Nov 17 21:23:22 RPi3 kernel: [642589.539798] EXT4-fs warning (device sdb1): htree_dirblock_to_tree:995: inode #2: lblock 0: comm smbd: error -5 reading directory block
Nov 17 21:23:22 RPi3 kernel: [642589.539895] EXT4-fs error (device sdb1): __ext4_get_inode_loc:4674: inode #2: block 2081: comm smbd: unable to read itable block
Nov 17 21:23:22 RPi3 kernel: [642589.539913] EXT4-fs error (device sdb1) in ext4_reserve_inode_write:5883: IO failure
Nov 17 21:23:22 RPi3 kernel: [642589.539932] ------------[ cut here ]------------
Nov 17 21:23:22 RPi3 kernel: [642589.539953] WARNING: CPU: 0 PID: 29585 at fs/fs-writeback.c:2238 __mark_inode_dirty+0x2f4/0x408
Nov 17 21:23:22 RPi3 kernel: [642589.539959] bdi-block not registered
Nov 17 21:23:22 RPi3 kernel: [642589.539967] Modules linked in: fuse rfcomm bnep hci_uart btbcm serdev bluetooth ecdh_generic 8021q garp stp llc sg brcmfmac brcmutil sha256_generic cfg80211 $
Nov 17 21:23:22 RPi3 kernel: [642589.540331]  ip_tables x_tables ipv6
Nov 17 21:23:22 RPi3 kernel: [642589.540359] CPU: 0 PID: 29585 Comm: smbd Tainted: G         C        4.19.66-v7+ #1253
Nov 17 21:23:22 RPi3 kernel: [642589.540364] Hardware name: BCM2835
Nov 17 21:23:22 RPi3 kernel: [642589.540391] [<80111f38>] (unwind_backtrace) from [<8010d4b0>] (show_stack+0x20/0x24)
Nov 17 21:23:22 RPi3 kernel: [642589.540410] [<8010d4b0>] (show_stack) from [<808191e0>] (dump_stack+0xd4/0x118)
Nov 17 21:23:22 RPi3 kernel: [642589.540428] [<808191e0>] (dump_stack) from [<801209c8>] (__warn+0x104/0x11c)
Nov 17 21:23:22 RPi3 kernel: [642589.540444] [<801209c8>] (__warn) from [<80120a38>] (warn_slowpath_fmt+0x58/0x74)
Nov 17 21:23:22 RPi3 kernel: [642589.540460] [<80120a38>] (warn_slowpath_fmt) from [<802dd4f8>] (__mark_inode_dirty+0x2f4/0x408)
Nov 17 21:23:22 RPi3 kernel: [642589.540477] [<802dd4f8>] (__mark_inode_dirty) from [<802c8434>] (generic_update_time+0x8c/0x100)
Nov 17 21:23:22 RPi3 kernel: [642589.540493] [<802c8434>] (generic_update_time) from [<802ca614>] (touch_atime+0xc8/0xe4)
Nov 17 21:23:22 RPi3 kernel: [642589.540510] [<802ca614>] (touch_atime) from [<802c0138>] (iterate_dir+0xf8/0x160)
Nov 17 21:23:22 RPi3 kernel: [642589.540528] [<802c0138>] (iterate_dir) from [<802c0728>] (ksys_getdents64+0x98/0x140)
Nov 17 21:23:22 RPi3 kernel: [642589.540544] [<802c0728>] (ksys_getdents64) from [<802c07e8>] (sys_getdents64+0x18/0x1c)
Nov 17 21:23:22 RPi3 kernel: [642589.540560] [<802c07e8>] (sys_getdents64) from [<80101000>] (ret_fast_syscall+0x0/0x28)
Nov 17 21:23:22 RPi3 kernel: [642589.540568] Exception stack(0x81adffa8 to 0x81adfff0)
Nov 17 21:23:22 RPi3 kernel: [642589.540579] ffa0:                   017bc2b8 017bc2d8 00000021 017bc2d8 00008000 00000000
Nov 17 21:23:22 RPi3 kernel: [642589.540592] ffc0: 017bc2b8 017bc2d8 017bc2bc 000000d9 017bc2b8 7eae92e0 017b5db0 76d6c534
Nov 17 21:23:22 RPi3 kernel: [642589.540601] ffe0: 76dfaf58 7eae91b0 76237474 762373a8
Nov 17 21:23:22 RPi3 kernel: [642589.540609] ---[ end trace 0aca54ceb067b5f8 ]---
Nov 17 21:23:22 RPi3 kernel: [642589.541693] EXT4-fs error (device sdb1): ext4_find_entry:1455: inode #2: comm smbd: reading directory lblock 0
Nov 17 21:23:22 RPi3 kernel: [642589.541744] EXT4-fs error (device sdb1): ext4_find_entry:1455: inode #2: comm smbd: reading directory lblock 0
Nov 17 21:23:22 RPi3 kernel: [642589.541881] EXT4-fs warning (device sdb1): htree_dirblock_to_tree:995: inode #2: lblock 0: comm smbd: error -5 reading directory block
Nov 17 21:23:27 RPi3 kernel: [642594.368519] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:f7:ef:00:04:20:1e:dd:54:08:00:45:00:00:28:31:d9:00:00:40:06:fe:7c SRC=192.168.100.128 DST=192.168.100.16$
Nov 17 21:23:27 RPi3 kernel: [642595.072268] jbd2_journal_bmap: journal block not found at offset 13813 on sdb1-8
Nov 17 21:23:27 RPi3 kernel: [642595.072290] Aborting journal on device sdb1-8.

also there is an issue with a timeout waiting for hardware interrupt but I don't know anything about it, except there seems to be a related Github issue:

Nov 18 09:51:04 RPi3 kernel: [687451.651659] mmc0: timeout waiting for hardware interrupt.
...
Nov 18 10:22:42 RPi3 kernel: [689349.901550] [c03fe701] PRD3 b5880284 0
Nov 18 10:22:42 RPi3 kernel: [689349.901557] [c03fe702] PDM> b9738fa0 0
Nov 18 10:22:42 RPi3 kernel: [689349.901565] [c03fe703] REQ< b9738ec0 10801
Nov 18 10:22:42 RPi3 kernel: [689349.901572] [c03fe704] CMD< 17 8
Nov 18 10:22:42 RPi3 kernel: [689349.901580] [c03fe708] FCM< b9738ec0 b9738f04
Nov 18 10:22:42 RPi3 kernel: [689349.901587] [c03fe70b] RSP  900 0
Nov 18 10:22:42 RPi3 kernel: [689349.901595] [c03fe70b] CMD< 12 929e08 
Nov 18 10:22:42 RPi3 kernel: [689349.901603] [c03fe70d] CMDD 8 200
Nov 18 10:22:42 RPi3 kernel: [689349.901610] [c03fe70e] SDMA b9738fa0 b9ebe840
Nov 18 10:22:42 RPi3 kernel: [689349.901618] [c03fe710] FCM< b9738ec0 b9738f38
Nov 18 10:22:42 RPi3 kernel: [689349.901625] [c03fe710] RSP  900 0
Nov 18 10:22:42 RPi3 kernel: [689349.901632] [c03fe711] FCM> b9738ec0 0
Nov 18 10:22:42 RPi3 kernel: [689349.901640] [c03fe711] FCM> b9738ec0 0
Nov 18 10:22:42 RPi3 kernel: [689349.901647] [c03fe711] CMD  12 200
Nov 18 10:22:42 RPi3 kernel: [689349.901655] [c03fe712] REQ> b9738ec0 0
Nov 18 10:22:42 RPi3 kernel: [689349.901662] [c0db9160] TIM< 0 0
Nov 18 10:22:42 RPi3 kernel: [689349.901676] mmc0: sbc op 23 arg 0x8 flags 0x15 - resp 00000900 00000000 00000000 00000000, err 0
Nov 18 10:22:42 RPi3 kernel: [689349.901688] mmc0: cmd op 18 arg 0x929e08 flags 0xb5 - resp 00000900 00000000 00000000 00000000, err 0
Nov 18 10:22:42 RPi3 kernel: [689349.901695] mmc0: data blocks 8 blksz 200 - err 0
Nov 18 10:22:42 RPi3 kernel: [689349.901706] mmc0: stop op 12 arg 0x0 flags 0x95 - resp 00000000 00000000 00000000 00000000, err 0
Nov 18 10:22:42 RPi3 kernel: [689349.901711] mmc0: =========== REGISTER DUMP ===========
Nov 18 10:22:42 RPi3 kernel: [689349.901716] mmc0: SDCMD  0x00004052   
Nov 18 10:22:42 RPi3 kernel: [689349.901722] mmc0: SDARG  0x00929e08   
Nov 18 10:22:42 RPi3 kernel: [689349.901727] mmc0: SDTOUT 0x017d7840
Nov 18 10:22:42 RPi3 kernel: [689349.901732] mmc0: SDCDIV 0x00000003   
Nov 18 10:22:42 RPi3 kernel: [689349.901737] mmc0: SDRSP0 0x00000900   
Nov 18 10:22:42 RPi3 kernel: [689349.901742] mmc0: SDRSP1 0x00001269   
Nov 18 10:22:42 RPi3 kernel: [689349.901747] mmc0: SDRSP2 0x7fffffff   
Nov 18 10:22:42 RPi3 kernel: [689349.901753] mmc0: SDRSP3 0x00024007   
Nov 18 10:22:42 RPi3 kernel: [689349.901758] mmc0: SDHSTS 0x00000080
Nov 18 10:22:42 RPi3 kernel: [689349.901763] mmc0: SDVDD  0x00000001
Nov 18 10:22:42 RPi3 kernel: [689349.901768] mmc0: SDEDM  0x00010801   
Nov 18 10:22:42 RPi3 kernel: [689349.901773] mmc0: SDHCFG 0x0000040e   
Nov 18 10:22:42 RPi3 kernel: [689349.901778] mmc0: SDHBCT 0x00000200
Nov 18 10:22:42 RPi3 kernel: [689349.901783] mmc0: SDHBLC 0x00000008   
Nov 18 10:22:42 RPi3 kernel: [689349.901788] mmc0: ===========================================

Pieter Wuille · Accepted Answer · 2019-11-09 18:25:12Z

2

That sounds like it's going out of memory and is killed by the system. Especially on a low-memory device like an RPi3 this seems very likely.

For information on how to reduce bitcoind memory usage, see https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md

answered Nov 9, 2019 at 18:25

Pieter Wuille

108k9 gold badges200 silver badges313 bronze badges

I will check it. Especially the last comment regarding the Linux-specific thread thing seems interesting. I also added my config to my question since this already was a setup for a low-memory environment.
– bomben
Commented Nov 9, 2019 at 18:56
Read your kernel log.
– Claris
Commented Nov 10, 2019 at 6:18
@Anonymous You mean with journalctl -k, right? Did not find anything relevant around the time the last entry was in debug.log from bitcoind.
– bomben
Commented Nov 12, 2019 at 14:31

Add a comment |

Stack Exchange Network

Where can I find a usefull log if bitcoind stopped for no apparent reason?

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged
bitcoind
raspberry-pi
logging
or ask your own question.

Hot Network Questions

Where can I find a usefull log if bitcoind stopped for no apparent reason?

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged bitcoindraspberry-pilogging or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
bitcoind
raspberry-pi
logging
or ask your own question.