Creating my own bootstrap.dat?

Question

Since I have been tearing down and setting up bitcoin clients a lot for some experiments lately, I was wondering whether there is a simple way to speed up the synchronization with the network. One method that comes to mind is the bootstrap.dat file distributed by some people. The problem is that the last 40'000 blocks, which are the most expensive to get and verify, are not included.

Now I was wondering how to create such a bootstrap.dat myself, from a good copy I have on another machine. Any idea?

Answer 1

The block files are in the same format as the bootstrap file. Their format is really simple: just concatenate all blocks after prefixing them by the network number (to avoid mixing the testnets) and the block length.

[network number] [length] [block header] [block transactions]
[network number] [length] [block header] [block transactions]
[network number] [length] [block header] [block transactions]
... repeat for all blocks

To create a bootstrap file, all you need to do is to just grab the data from blk00001.dat up to your last block file (here that's blk000054.dat) and put all of it in the bootstrap.dat file, preferably sorted so that it can just import them all in one go without having to swap blocks.

I think you might even be able to just concatenate the block files themselves, as they are already in the right format, and the format supports concatenation.

Answer 2

Bitcoin Core contains a tool to do this properly (filtering out any orphan blocks, putting everything in the correct order — which will work a lot better than just concatenating the block files. It's documented in its own README, here: https://github.com/bitcoin/bitcoin/tree/master/contrib/linearize

In short:

1. Launch Bitcoin with an rpcuser and rpcpassword set in your config file, with the correct location as per https://en.bitcoin.it/wiki/Running_Bitcoin#Bitcoin.conf_Configuration_File
2. Checkout the Bitcoin source from Github and cd to contrib/linearize.
3. Copy example-linearize.cfg to linearize.cfg and update it with your rpcuser/rpcpassword values, along with the maximum block height you want to output (max_height), and the path to your data directory (input), as well as the path to your desired output file (output_file).
4. Run ./linearize-hashes.py linearize.cfg > hashlist.txt followed by ./linearize-data.py linearize.cfg.

Answer 3

 From Mac

1. Open terminal

2. Go to the folder where the blocks are. In this case Bitcoin's folder (it's the same for other bitcoin forks)

   cd "Library/Application Support/Bitcoin/blocks/"
   

3. Run this code that will concatenate all the blk files. Make sure to include all the numbered blocks you find on your Bitcoin's folder. In this case there are 4

   cat blk000*.dat > bootstrap.dat
   

4. Done.

Answer 4

Making your own bootstrap.dat is fairly simple. In windows, drop to your command prompt and use the following command.

Copy /b blk00000.dat+blk00001.dat+blk00002.dat bootstrap.dat

You'll want to keep going with the + blk*.dat files until you reach the last one. To make things easier for you, just run a batch file which can take care of adding them all without having to input each blk*.dat file. Simply copy and paste the code below into a text file and name it something like Create_Bootstrap.cmd and run it from your /blocks folder.

@echo off 
setlocal enableDelayedExpansion 

set BITCOINDIR=%APPDATA%\bitcoin\blocks
for /F %%x in ('dir /B/D/ON %BITCOINDIR%\blk*.*') do (
  IF NOT [!B!] == [] set B=!B!+
  set FILENAME=%BITCOINDIR%\%%x
  set B=!B!"!FILENAME!"
)
copy /b %B% bootstrap.dat

or just use "copy /b blk*.dat bootstrap.dat"

Answer 5

It is somewhat intuitive what files need to be copy and where once you have a full node built. What is not so intuitive is how to bootstrap a full node before you have built your first full node. You first need to know where the files need to be placed for the various 'OSes'. I'm not sure if there are little or big endian issues with copying the raw Blockchain node files between different OSes.

Another approach is to let the protocol handle the problem. Secondary and tertiary nodes build very quickly if they a on your own high bandwidth LAN. Just point a secondary node (by tweaking its bitcoind.conf file) to the primary node that is already built using the "connect=Your_IPv4_LAN_Address". Such builds move very quickly... You might find it to be faster than writing the ledger to a thumb drive and copying it to another computer.

Answer 6

Go to en.blockchaindownload.nl and download the newest chaindownload. On the site is also a guide that tells uou how it works.

Just put that file onto a USB and you can use it over and over. Its free to use and updates the download twice a week. It uses torrents with 2 or 3 (depending on the torrent you pick) webseeds for all the speed you need.