Recently I started running a Core 15.1 node with the blockchain on an external hard drive, and found it very hard to fully sync the blockchain. The problem I keep running into is that (in about 90% of my attempts) the sync does not complete. A DB corruption occurs at some point (usually pretty late in the game, maybe 36 hours into it).
I did some digging in forums and saw that I am not the first user to notice this problem. Peter Wuille mentioned that he does a full sync on like a weekly basis and it always works, and he thus suspects that these type of issues are due to sub optimal hardware.
That seemed very reasonable at first because my initial attempt was done using a pretty old 400gig pocket hard drive. So I switched hard drives. By now, about 2 months later, I have tried to sync the blockchain from scratch about 20 or 30 times and only twice managed to complete the process. I did that on 4 different types of hard drives, including a brand new 1T Western Digital.
One may say that all 4 hard drives I used are "suboptimal", but I would object to that because
- After the sync does complete, the node runs just fine.
- I think it is desirable to be able to run a full node on a raspberry pi with a pocket size hard drive (note that I run the node on a raspberry pi but use my linux desktop to do the initial sync)
Now the issue I have here is not so much the crash during sync, but the fact that when I restart the client after the crash the only option is to basically start from scratch (If I understand correctly, the client is not necessarily downloading everything from scratch but that is not my point) and sync again from Genesis. Ideally I would like the client to keep synching from the point it successfully got to and not re sync from scratch on such errors.
I am wondering if this resync from scratch on error is necessary. I assume that corruption occurs in one (or more) of the files in the blocks or chainstate directories. If that is so then why sync from Genesis? Is it not possible to, say, delete all files post the corrupted file and sync from there?
If there is no fundamental reason for this behaviour, I would also appreciate a pointer to the relevant code in the client.