10

I just reindexed a node that was already synced (in order to enable txindex); it took around 9 hours. It's on a pretty good system: 16G RAM, i7 with 4 physical cores. Storage is a weak point, with LUKS on software RAID0 on spinning disks. dbcache is at the default 100M, though I'm running 0.11 so I gather this ends up smaller than the previous default due to more accurate accounting. By extrapolation from testing a subset, a sequential read of the full ~38G of blk files should take ~4 minutes.

I understand reindex includes rebuilding the chainstate db, so I'm guessing that's the bottleneck. Questions: 1) reindex doesn't also re-validate blocks, does it? 2) Any experience with speeding this up -- perhaps putting chainstate on a tmpfs and copying to permanent storage once complete?

Followup: by increasing dbcache as suggested, and adding a recent custom checkpoint to the code, I got another reindex done in about 3 hours.

Improve this question
1
  • I like the "custom checkpoint" idea. It suggests that a feature could be added to allow any user to make one from the GUI, so, for example, if you want your client to keep running but you know you're doing something that might crash your machine, you could checkpoint right here and vastly speed up your recovery - or "replace last checkpoint with current state". Have you made such a suggestion anywhere?
    – Dave Scotese
    Commented Jan 6, 2017 at 20:30

4 Answers 4

Reset to default
11

reindex doesn't also re-validate blocks, does it?

It does, but signature verification (the slowest bit) is skipped on most of the early blockchain. The command attempts to sync with the network by reusing as many blocks as possible that are already written to the disk, which may very well include damaged or partially written ones.

Any experience with speeding this up

Increasing the dbcache will speed up reindexing, putting it at 2G (dbcache=2048) would not be an unreasonable choice given you have the memory. This is about all you can do, until it gets to the point where you are validating signatures a reindex is heavily single threaded.

Improve this answer
6
  • How is it determined which signatures are verified -- after the last checkpoint? Is there a technical reason this is necessary, as opposed to, say, just verifying block hashes and PoW until the last couple dozen/hundred?
    – jwelsh
    Commented Jul 5, 2015 at 20:20
  • 1
    All signatures after the last checkpoint. Optimally all signatures would always be verified but the sync time would just be extreme, not doing it at all would risk decentralization. It's a crutch for the fact that ECDSA is slow and there's a lot of transactions in the Bitcoin blockchain to verify.
    – Claris
    Commented Jul 5, 2015 at 20:32
  • 4
    Put more simply: reindez does exactly the same as redownloading the chain from scratch, except it reuses the blocks on disk you already have.
    – Pieter Wuille
    Commented Jul 5, 2015 at 22:45
  • Ahh... and that explains why the first 295k blocks go so much faster, in both initial sync and reindex.
    – jwelsh
    Commented Jul 6, 2015 at 2:00
  • 1
    @fossilet Sure it does. Read the answer above.
    – Pieter Wuille
    Commented Jun 5, 2016 at 7:16
1

The behavior I noticed from a reindex on my node was that it ended up reloading the entire blockchain from the Internet. This is consistent with what the Bitcoin Meister Pieter Wuille indicated. Will have to remember the dbcache trick...

Improve this answer
0

I know this is an older question, but I've found it now, so I think it can be good for others who will find this thread.

You should download the latest version of bitcoin QT (0.15.1 as of writing this) which has some performance tweaks and it reindex on disk about 10 times faster than the 0.12 which i had before.

saying that, you could also download blockchain from torrents. Here is the official link to torrent https://bitcoin.org/bin/block-chain/

After downloading it, you should call

bitcoin-qt.exe -datadir=<path to Bitcoin dir> -rescan

And the speed will come to you. ;)

Improve this answer
1
  • 2
    Downloading through torrents is actually not faster than letting bitcoin-qt do the downloading. Used to be several years ago, but these days bitcoin itself does a better job than bittorrent and can then do validation in parallel to the downloading. The link you provided points to a 2014 torrent file. That shows how outdated this trick is.
    – Jannes
    Commented Jan 13, 2018 at 13:44
0

Increasing dbcache is probably the most important thing you can do:

-dbcache=1000

Another thing to do is to disable all console output:

-printtoconsole=0

Apart from that, if your disk is the bottleneck, you can move parts of the blockchain data to a faster disk. The majority of the blockchain (the blocks themselves) can stay on a slow HDD, and you can move chainstate and index (inside blocks) folders to an SSD by creating symbolic links.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.