Why is downloading blocks taking longer and longer?

Question

First off, this really needs to get fixed. I'm downloading blocks for at least 8 hours now, and it's still not done. I know the problem of having to download the entire block chain is being worked on, but I feel it's already out of hand. Setting up an application should take something you would measure in minutes, not hours or days. (After spending the entire night, it's now 4:15am, I would estimate the setup time to be two days.)

Also downloading can be much optimized as it is. It's far from using my maximum download speed, and being connected to one peer is as fast as being connected to 8+ (yup, also tried port forwarding).

Now, ontopic.

Is it normal that downloading blocks goes slower, the further you progress? The first half (out of 194.000) went in the blink of an eye, after that it became less and less fast. Now at 20.000 blocks remaining, progress comes in bursts, but there can easily be 20 seconds of inactivity and when it bursts it's like 10 blocks.

Is this normal behavior? And also, why doesn't the number of connections make any difference for the speed at all? If it works like Bittorrent, which some describe it to work like, it should get faster with more people who can upload to you (more connections).

I've Googled, but all I've found is that it takes a few hours and that it may get stuck. It's not stuck and taking as long as people describe it to be, but I just wonder if it's normal that it's going slower and slower, and why it doesn't saturate my download capacity (like BitTorrent) no matter how many connections.

Thanks

Answer 1

Very little of the time is spent downloading the blockchain data compared to how much time your computer is spending verifying each transaction. Bitcoin does a ton of disk i/o for this.

As there are more transactions since April (thanks to SatoshiDICE, we now see 40K and more transactions a day) those blocks take longer to verify.

If you are on an encrypted filesystem, that will really be hurtful for perfomance. If so, consider moving the blockchain and index to a filesystem that isn't encrypted and then symlink to them there.

Also, the latest release of the Bitcoin.org client (v0.6.3) has peformance improvements in this area over previous releases.

End uses generally will not need to run the client and instead use SPV (Simplified Payment Verification). So individuals and organizations that do want to run the full client should recognize that to do so requires a beefy CPU and quick storage (SSD maybe even).

A future version (after 0.7) will include LevelDB which should make huge gains as far as performance.

In the meantime, you can always securely download a full set of blockchain data from Sourceforge that is as-of a fairly recent point in time.

Answer 2

Okay, combining answers and what more I've observed:

• The rate of blocks slows down because more recent blocks are larger. The number of blocks remaining is not actually a good measure for estimated time.
• You can make an estimate based on how much data is downloaded (in how much time) and how much data is left: At the moment of writing, my blk000x.dat files are totaling 8.3GB (10 Feb 2013).
• I can't say anything about how fast the data size will increase, perhaps someone can comment on this for future readers.
  Update: It seems to be growing rather fast, probably linearly with the number of transactions per block
• The reason it doesn't max out your download speed, is because it's actually doing a lot of other things to verify the transactions.
• Downloading a zip with nightly blk000x.dat files is a lot faster, but copying the files didn't work... In the end I just left it running a few more hours. Also see the comment about a bug this might introduce when an error is present in the files (intentionally or not).
• Encrypted filesystems make it take even longer because verifying blocks is an disk-intensive process.
• If you can, use an SSD, this speeds it up by a lot. You could be done in a few hours instead of taking all night.
• Usually this process just takes a while. 24 hours is not uncommon. Best is to leave your computer running during the night, and don't be disappointed when it's still not done in the morning.

Hope this helped! Credits to the other answers for some of the bullet points.

Answer 3

Yes, it's normal for the rate of blocks being received to decrease as the download progresses.

The reason it slows down is because there were very few transactions in each block in the early days of Bitcoin, so the blocks were smaller. Later blocks are bigger.

The Satoshi Bitcoin Client is still a Beta version, and even if it wasn't it may still need to download the entire blockchain which I believe is over 2GB.

Which platform are you running the client on? You should be able estimate how much data is left to download by looking at the blockchain files on disk in the application folder.

It appears that HDD speed may be a significant factor too, as verification of each block is performed as it's downloaded, and that can be complicated for a transaction with many inputs. Discussion here. dooglus provides evidence of that theory by testing with a RAM disk and heavily reducing the download time.

Answer 4

1. Moving blkindex.dat to a let's say ram drive dramatically improves bitcoin/bitcoind performance. Just notice that currently index is around 1.5Gb size.
2. Next version 0.8, as already been said, is using a new database instead of BDB and this works much faster and don't need to be moved to a ram drive or something.

Answer 5

I kinda figured out how to improve performance on ext4/linux:

http://tlfabian.blogspot.hu/2013/04/speeding-up-bitcoin-qt-on-linux.html

Answer 6

Bitcoin is P2P like BitTorrent. It is likely to be because your computer is slow validating the blockchain as it downloads. Most of the work is not actually the downloading. This answer may help you.