Very hard to sync Core Client from scratch

Question

Recently I started running a Core 15.1 node with the blockchain on an external hard drive, and found it very hard to fully sync the blockchain. The problem I keep running into is that (in about 90% of my attempts) the sync does not complete. A DB corruption occurs at some point (usually pretty late in the game, maybe 36 hours into it).

I did some digging in forums and saw that I am not the first user to notice this problem. Peter Wuille mentioned that he does a full sync on like a weekly basis and it always works, and he thus suspects that these type of issues are due to sub optimal hardware.

That seemed very reasonable at first because my initial attempt was done using a pretty old 400gig pocket hard drive. So I switched hard drives. By now, about 2 months later, I have tried to sync the blockchain from scratch about 20 or 30 times and only twice managed to complete the process. I did that on 4 different types of hard drives, including a brand new 1T Western Digital.

One may say that all 4 hard drives I used are "suboptimal", but I would object to that because

• After the sync does complete, the node runs just fine.
• I think it is desirable to be able to run a full node on a raspberry pi with a pocket size hard drive (note that I run the node on a raspberry pi but use my linux desktop to do the initial sync)

Now the issue I have here is not so much the crash during sync, but the fact that when I restart the client after the crash the only option is to basically start from scratch (If I understand correctly, the client is not necessarily downloading everything from scratch but that is not my point) and sync again from Genesis. Ideally I would like the client to keep synching from the point it successfully got to and not re sync from scratch on such errors.

I am wondering if this resync from scratch on error is necessary. I assume that corruption occurs in one (or more) of the files in the blocks or chainstate directories. If that is so then why sync from Genesis? Is it not possible to, say, delete all files post the corrupted file and sync from there?

If there is no fundamental reason for this behaviour, I would also appreciate a pointer to the relevant code in the client.

Thanks!!

Answer 1

Database corruption means the state of your node is corrupted, not just data it downloaded. Resuming after state corruption would be like resuming your life after having a lobotomy.

During synchronization your node operates at hundreds of times its normal load: Consider, Bitcoin has existed for over 3000 days yet sync will complete in a day. This tends to expose marginal hardware. "suboptimal" would not be a very accurate way to describe it, "totally broken" would be better.

USB drive in general are notorious for causing corruption. Not only are the drives themselves often unreliable, but many people often suffer issues from their USB controllers and even cables.

At the expense of several more gigabytes of space usage Bitcoin nodes could periodically back up their state so that they could recover more gracefully from corruption... but the cost of doing that is great enough that the gains aren't completely clear. Especially since hardware that is messing up is also probably likely to corrupt the backups.

Answer 2

I have two and a half comments to make, from similar experience: a RasPi is a nice piece of hardware, but may not be the optimal device for the full node. Besides the many links in the forum here and bitcointalk.org, I think it boils down to these two issues:

The process of loading the blockchain into your computer is I/O heavy to the disk. And as the RasPi has only a USB port to the disk, it is times 10 slower than a standard interface to a disk drive. Whereas my std IDE interface is running at +20 megabytes per second, I had a hard time to achieve more than 2 megabytes per second via a USB device. And of course the SSD runs even faster...

But this is only the part of writing data to the disk. Then you also have to read previous sectors from the disk, which contain several blockchain data with previous transaction data. Here the amount of data is less relevant, but the disk I/O cycles come into the game. This is like randomly seeking through the disk drive, and might also slow down the process. Measuring CPU load showed +4.x, so there where more processes asking to do I/O, than the system could handle. I also played with the memory parameters, but to no avail. The I/O was the limiting factor.

And then there is the capacity of the RasPi CPU. I have learned, that my RasPi loaded the blockchain fairly well and in steady process up to July/August 2017, and then started to choke every now and then. This is exactly the time when blocks got more and more full. And I understand, that loading a full block means many, many data to be verified, adding load to the RasPi. For the operating system, I had more success loading the blockchain with Linux. Windows was simply not reliable enough. But still it took me more than 3 weeks on a RasPi2 to get the blockchain loaded.

What I then did: I have three different Macintosh’s, two equipped with SSD and Intel Core Duo and 4 Gig RAM (so fairly old, like +6 years). These SSD systems didn’t have any problem to download the blockchain, it still took them between 2 and three days.

Btw: I observed exactly the same issues with Ethereum and RasPi. So for me it looks like the RasPi has it’s limitations, when it comes to high loads. I haven’t debugged anything, but it looks like the RasPi is loosing some interrupts at high load, which leads to the errors in the writing of data to the disk. Also I hadn’t used a cooling device, so CPU got certainly to it’s max temp, when calculating all the sha256, base58 and sigs.

To circumvent this, I copied the downloaded blockchain from the Mac with the SSD to the RasPi, and then he would work again like a charm. Hint: I did these tests with a RasPi2 up until Nov 2017, the type3 model should be slightly faster, but not magnitudes.

So when you say, you load the initial blockchain from your PC, is HDD still connected to the USB drive? Have you tried I’ver local disk, and then do an sync to the RasPi? Maybe worthwhile to check. I found, that the bandwidth of the internet link is not the limiting issue, while you are downloading the blockchain...