Number of bitcoind peers drops precipitously every 30 days or so

Question

I run bitcoind headless on a linux box with maxconnections=256 and port 8333 open on the router. Using Bitcoin Node Manager I monitor node status occasionally. Peer count of course starts at 8 and slowly, over the course of three or four weeks, ramps up to 150 or so. Roughly every month or so the number of node drops precipitously to less than 20, then starts climbing again. I can verify via the logs that bitcoind did not restart and that my network did not go down. I have a second node that runs solely over TOR and while it does not have an many peer connections, its peer count doesn't fluctuate in this way.

1. Does anyone else notice this?

2. Any ideas what might cause this?

3. Any tips on how (or even if) I should troubleshoot this? (i.e. things to look for in the logs)

Answer 1

Being doing my own diagnosis on this issue. I have bitcoind and lnd running on the same raspberry pi, as per the excellent Raspibolt guide by Staticus. I wrote the following script to log both daemons' cpu usage, mem usage and total number of bitcoind peers:

#!/bin/bash
echo \
"$(date +%D,%T),"\
"$(ps -U satoshi -o comm= -o %cpu= | grep bitcoind | awk '{print $2}'),"\
"$(ps -U satoshi -o comm= -o %mem= | grep bitcoind | awk '{print $2}'),"\
"$(ps -U satoshi -o comm= -o %cpu= | grep lnd | awk '{print $2}'),"\
"$(ps -U satoshi -o comm= -o %mem= | grep lnd | awk '{print $2}'),"\
"$(/usr/local/bin/bitcoin-cli -conf=/etc/bitcoin.conf getconnectioncount)" >> /home/satoshi/bitcoin_connections

My issue may be simply allowing too many connections to my daemon. As peer count grows, mem usage does as well. When it gets to 130+ peers total system memory usage is >80%. My issue may simply be an issue with running out of memory. I'll edit this post when I have more results.

Edit:

It seems that as I get to around 120-130 peers the combined mem usage of bitcoind and lnd exceed 60% of the 1 gig on my RPi, pushing total mem usage over 90%. At that point peer count drops off. I presume the daemon starts to respond very slowly as memory runs out and other peers drop mine. I set maxconnections=125 (default) and the daemon has run for weeks without issues. Since 125 is the default, the same result could be achieved by commenting out the maxconnections line in bitcoin.conf completely.