Blockchain sync seems to be stuck in a loop?

Question

SOLVED! UPDATE: The issue was not bitcoin related. I had to increase the root partition of ubuntu, as it wasn't using the full SSD size I had, but only 100 GB and the sync therefore ran out of space and restarted from a certain point.

I connect to my node that I've set up after raspibolt.org (I have a NUC and not a raspberry though, so 32 Bit). I connect to it over SSH and sometimes check the status of the sync. I'm not sure if I read the logs wrong, but they get renewed permanently, so it's not stuck, but it seems like the sync got into some kind of loop where it always restarts. Another node of me synced within 30 hours from 0 to absolute completeness, and it's on the same connection.

I login over SSH and execute this command: tail -f /home/bitcoin/.bitcoin/debug.log

bitcoind is running as a service, not manual. I wonder if me disconnecting from SSH could somehow halt the blockchain sync?

OS: Ubuntu 24.04

Recent logs (it's always from height 400'000 and TXes from 2026):

Your help is highly appreciated.

I assume that progress=0.110276 is the percentage (in decimals) of the sync. It hovers from 0.11 to 0.13 since a day.

Update:

I have no idea if related, but I saw this error. Not sure if it's from the ssh client or pc:

2024-05-12T16:32:00Z UpdateTip: new best=0000000000000000050dc384f0c8f3ed586bf88                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             53f4f94a2da912cf2a6aff54d height=414977 version=0x00000004 log2_work=84.782407 t                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             x=133826461 date='2016-06-06T00:57:37Z' progress=0.131048 cache=1743.6MiB(156232                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             88txo)
2024-05-12T16:32:01Z UpdateTip: new best=000000000000000004b29c6c8e3ff428912c528                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             b3fc4124ccc1314c43e08247d height=414978 version=0x30000000 log2_work=84.782444 t                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             x=133827957 date='2016-06-06T01:11:03Z' progress=0.131049 cache=1744.6MiB(156340                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             76txo)
2024-05-12T16:32:01Z Cache size (1830114096) exceeds total space (1829784384)
2024-05-12T16:32:01Z *** Disk space is too low!
2024-05-12T16:32:01Z Error: Disk space is too low!
2024-05-12T16:32:01Z tor: Thread interrupt
2024-05-12T16:32:01Z Shutdown: In progress...
2024-05-12T16:32:01Z addcon thread exit
2024-05-12T16:32:01Z RemoveLocal(gjzvhraxkhrxjgq7cswsl2xw5zapifxg5fmh5gwhj3ljgvmbqlo7w7yd.onion:8333)
2024-05-12T16:32:01Z *** Failed to connect best block (Disk space is too low!)
2024-05-12T16:32:01Z Error: A fatal internal error occurred, see debug.log for details
2024-05-12T16:32:01Z torcontrol thread exit
2024-05-12T16:32:01Z msghand thread exit
2024-05-12T16:32:01Z txindex thread start
2024-05-12T16:32:01Z txindex: m_interrupt set; exiting ThreadSync
2024-05-12T16:32:01Z txindex thread exit
2024-05-12T16:32:01Z Loading 0 mempool transactions from disk...
2024-05-12T16:32:01Z Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
2024-05-12T16:32:01Z initload thread exit
2024-05-12T16:32:01Z net thread exit
2024-05-12T16:32:01Z ERROR: Error while reading proxy response
2024-05-12T16:32:01Z opencon thread exit
2024-05-12T16:32:01Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat started
2024-05-12T16:32:01Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat completed (0.00s)
2024-05-12T16:32:01Z scheduler thread exit
2024-05-12T16:32:01Z Cache size (1830114096) exceeds total space (1829784384)
2024-05-12T16:32:01Z *** Disk space is too low!
2024-05-12T16:32:01Z Error: Disk space is too low!
2024-05-12T16:32:01Z ForceFlushStateToDisk: failed to flush state (Disk space is too low!)
2024-05-12T16:32:01Z Cache size (1830114096) exceeds total space (1829784384)
2024-05-12T16:32:01Z *** Disk space is too low!
2024-05-12T16:32:01Z Error: Disk space is too low!
2024-05-12T16:32:01Z ForceFlushStateToDisk: failed to flush state (Disk space is too low!)
2024-05-12T16:32:03Z Shutdown: done
tail: /home/bitcoin/.bitcoin/debug.log: file truncated

Output of df -h is

tmpfs 1.6G 1.7M 1.6G 1% /run 
efivarfs 192K 115K 73K 62% /sys/firmware/efi/efivars  
/dev/mapper/ubuntu--vg-ubuntu--lv 98G 92G 1.5G 99% / 
tmpfs 7.6G 0 7.6G 0% /dev/shm 
tmpfs 5.0M 0 5.0M 0% /run/lock 
/dev/nvme0n1p2 2.0G 95M 1.7G 6% /boot 
/dev/nvme0n1p1 1.1G 6.2M 1.1G 1% /boot/efi 
tmpfs 1.6G 12K 1.6G 1% /run/user/1000

Answer 1

In my case, it was the disk running out of space (running Ubuntu Server 24.04). I didn't notice, that the root partition only has 100Gb assigned.

If you have the same issue and you run Ubuntu, this was my solution. ChatCpt helped me here.

Here's how you can increase the size of the LVM partition /dev/mapper/ubuntu--vg-ubuntu--lv using the lvresize command:

Make sure you have a backup of your important data before proceeding.

Boot your Ubuntu system and open a terminal.

Verify the current size of the LV partition using the lvdisplay command:

sudo lvdisplay

Look for the LV that corresponds to /dev/mapper/ubuntu--vg-ubuntu--lv to check its current size.

To increase the size of the LV, use the lvresize command with the desired new size. For example, to increase it by 5GB:

sudo lvresize -L +5G /dev/mapper/ubuntu--vg-ubuntu--lv

Adjust the size (+5G in this case) according to your needs.

Now, if the filesystem on the LV is ext4, you can expand it to utilize the additional space using the resize2fs command:

sudo resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv If the filesystem is different, you'll need to use the respective filesystem-specific command to resize it.

With these steps, you should be able to increase the size of the LVM partition (/dev/mapper/ubuntu--vg-ubuntu--lv) correctly. However, always ensure you have backups and exercise caution when modifying partitions and filesystems to prevent any potential data loss.