Node Stability And Troubleshooting

Question

Background: Not a lot of linux experience but technically able

What's the best way to troubleshoot the stability of bitcoind? I've had it just stop for no discernable reason.

I'm doing a full sync with txindex=1 about 615000 some blocks synced so far.

Running on a raspberry pi 3. The debug.log doesn't show anything particularly interesting. Simply starting it back up again works with no errors.

If i want to make sure my node is up... what's the best way to do that? A cron script that tries to start it if it's down? What about if that's failing?

Best solution seems to be to update some very stable external system (azure or something) every time it learns about a new block and alerting on that system if it doesn't see one for a couple hours or something.

Either using zeromq or a cron script checking for current block height to do the update.

Open to any best practices or already built solutions for this though.

Answer 1

What's the best way to troubleshoot the stability of bitcoind?

I would say logging.

You could also use Linux tools to monitor usage of memory and other resources. Maybe vmstat or top would show something interesting. Maybe bitcoind is filling a filesystem you didn't expect - df -Pk.

The debug.log doesn't show anything particularly interesting.

If you are not already running bitcoind with option -debug=1, I suggest you try it.

-debug=<category>

Output debugging information (default: -nodebug, supplying <category> is 
optional). If <category> is not supplied or if <category> = 1, output all 
debugging information. <category> can be: net, tor, mempool, http, bench, 
zmq, walletdb, rpc, estimatefee, addrman, selectcoins, reindex, cmpctblock, 
rand, prune, proxy, mempoolrej, libevent, coindb, qt, leveldb, validation.

Note that the default is -nodebug.

I've had it just stop for no discernable reason.

As well as turning up debug to the max, I would try to capture the return code ($?) of bitcoind.

If i want to make sure my node is up... what's the best way to do that?

The daemon process startup system on your Linux distro ought to be able to auto-restart a stopped daemon. You configure the startup system appropriately. Details probably depend in distro and version. E.g. in systemd: restart=always