6

I just reindexed a node that was already synced (in order to enable txindex); it took around 9 hours. It's on a pretty good system: 16G RAM, i7 with 4 physical cores. Storage is a weak point, with LUKS on software RAID0 on spinning disks. dbcache is at the default 100M, though I'm running 0.11 so I gather this ends up smaller than the previous default due to more accurate accounting. By extrapolation from testing a subset, a sequential read of the full ~38G of blk files should take ~4 minutes.

I understand reindex includes rebuilding the chainstate db, so I'm guessing that's the bottleneck. Questions: 1) reindex doesn't also re-validate blocks, does it? 2) Any experience with speeding this up -- perhaps putting chainstate on a tmpfs and copying to permanent storage once complete?

Followup: by increasing dbcache as suggested, and adding a recent custom checkpoint to the code, I got another reindex done in about 3 hours.

  • I like the "custom checkpoint" idea. It suggests that a feature could be added to allow any user to make one from the GUI, so, for example, if you want your client to keep running but you know you're doing something that might crash your machine, you could checkpoint right here and vastly speed up your recovery - or "replace last checkpoint with current state". Have you made such a suggestion anywhere? – Dave Scotese Jan 6 '17 at 20:30
7

reindex doesn't also re-validate blocks, does it?

It does, but signature verification (the slowest bit) is skipped on most of the early blockchain. The command attempts to sync with the network by reusing as many blocks as possible that are already written to the disk, which may very well include damaged or partially written ones.

Any experience with speeding this up

Increasing the dbcache will speed up reindexing, putting it at 2G (dbcache=2048) would not be an unreasonable choice given you have the memory. This is about all you can do, until it gets to the point where you are validating signatures a reindex is heavily single threaded.

  • How is it determined which signatures are verified -- after the last checkpoint? Is there a technical reason this is necessary, as opposed to, say, just verifying block hashes and PoW until the last couple dozen/hundred? – jwelsh Jul 5 '15 at 20:20
  • 1
    All signatures after the last checkpoint. Optimally all signatures would always be verified but the sync time would just be extreme, not doing it at all would risk decentralization. It's a crutch for the fact that ECDSA is slow and there's a lot of transactions in the Bitcoin blockchain to verify. – Anonymous Jul 5 '15 at 20:32
  • 4
    Put more simply: reindez does exactly the same as redownloading the chain from scratch, except it reuses the blocks on disk you already have. – Pieter Wuille Jul 5 '15 at 22:45
  • Ahh... and that explains why the first 295k blocks go so much faster, in both initial sync and reindex. – jwelsh Jul 6 '15 at 2:00
  • 1
    @fossilet Sure it does. Read the answer above. – Pieter Wuille Jun 5 '16 at 7:16
1

The behavior I noticed from a reindex on my node was that it ended up reloading the entire blockchain from the Internet. This is consistent with what the Bitcoin Meister Pieter Wuille indicated. Will have to remember the dbcache trick...

0

I know this is an older question, but I've found it now, so I think it can be good for others who will find this thread.

You should download the latest version of bitcoin QT (0.15.1 as of writing this) which has some performance tweaks and it reindex on disk about 10 times faster than the 0.12 which i had before.

saying that, you could also download blockchain from torrents. Here is the official link to torrent https://bitcoin.org/bin/block-chain/

After downloading it, you should call

bitcoin-qt.exe -datadir=<path to Bitcoin dir> -rescan

And the speed will come to you. ;)

  • 1
    Downloading through torrents is actually not faster than letting bitcoin-qt do the downloading. Used to be several years ago, but these days bitcoin itself does a better job than bittorrent and can then do validation in parallel to the downloading. The link you provided points to a 2014 torrent file. That shows how outdated this trick is. – Jannes Jan 13 '18 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.