0

I am trying to develop something similar to bitcoin-abe, e.g. import whole blockchain into database.

For database I currently using Redis - I know it will took lots of RAM, but lets suppose that is OK.

Another possibilities could be MySQL, Postgress or even CSV file.

I have bitcoind and I am importing block after block.

For each transaction, I need to look for output of the transactions that "fund" current transaction - I hope you understand what I mean.

I am currently checking with blocks from 2017. Most of blocks have 2000+ transactions. If we assume each transaction have single input, this means I need to look at 2000 more transactions.

At the moment those 2000 lookups are done back to bitcoind, because output database could be csv file.

I can do all this in "lazy" way, but then I will not be able to find if some address is spent.

Is there any faster way to import?

I am using PHP and single block from 2017 is imported for about 1 min.

1

output is spent (e.g. if appear in "vin" to some other transaction), I store an indicator.

Now single block is processed for 1-2 sec.

Implication of this is,
if you want to check address balance, you will need to check each "input" and to compile list of unspent inputs.

Still unsure if this is best way to go.

Update 1

I finally upload the software in github here:
https://github.com/nmmmnu/BlockchainWalker

Update 2

Here are how keys looks like if we want to find the balance for address "1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6":

127.0.0.1:2000> HGETALL a:1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6
 1) "a:1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0"
 2) "0.01270199"

This is funding transaction:

  • 28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0, output 0

This is value:

  • 0.01270199

Then we need to check if this is spent, so we check each input as follows:

127.0.0.1:2000> get t:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0:s
"055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8"

Transaction 055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8, is the transaction where this output is spent.

As each output is spent this means the address 1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6 have balance of zero.


In case we want to check / list a transaction "055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8", we can do:

127.0.0.1:2000> HGETALL t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8
 1) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.-:i:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0"
 2) "1"
 3) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.-:i:39edb8741b701b6da2dbc4e02290e8e78cba244bdbad96da203e41ee2704c525.0"
 4) "1"
 5) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.0:o"
 6) "1Kj76Sxe8c3UK85RAQwwdqScAxaBwAY2eb:0.00500000"
 7) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.0:s"
 8) "49f8bd582439a3b2351f92e0fdb5fcb1032acd42e3ab469c16d805627889ce14"
 9) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.1:o"
10) "14yAJga4ZkULbaMz4LUW5vj8GhQYyzCPoW:0.01118727"
11) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.1:s"
12) "edf55ace1396229c7a91a4b21cd63293c631ae2ddcc5987402b04ed540b9e5d8"

Those "t:xxxxx.-:i:xxxx" are inputs.

Those "t:xxxxx.N:o" are outputs

Those "t:xxxxx.N:s" are indicators, if output is spent or not.


Please note this is not Redis, it is database I developing that have Redis interface. You can check it here:

https://github.com/nmmmnu/HM4

  • How do you handle blockchain reorgs? – Jakub Fedyczak Sep 14 '17 at 18:37
  • honestly, no idea what this is. By try and error, I did this code: github.com/nmmmnu/BlockchainWalker/blob/master/… , it handles all transactions so far, tried 07-2017 to 08-2017, then beginning to end of 2016 – Nick Sep 14 '17 at 18:47
  • I also handle transactions with wrong addresses like 71bbaef28e09d8d6fadd41f053db7768dbb5fa4570f06b961dfc29db3dc00b1d – Nick Sep 14 '17 at 18:48
0

I am using RocksDB and node.js. My database is optimized for tracking address balances and payment history. It needs around 143GB of storage, 4 days to load. Current blocks import below 2s.

I've tried very hard to achieve this with PostgreSQL. It is doable, but requires a lot of hacking and post-import index creating. I've tried levelDB but it is roughly 4 times slower than rocksDB on SSD drive.

I am using custom block parsing library and half-baked full-node implementation.

  • When I said redis, I meant my own database server similar to LevelDB. Is now 4th day of loading, but I believe tomorrow evening will be ready. However question was how to store data in best way. I am using RAID 5 on spinning HDD's. Currently my database use 260 GB, but it have create and expiration time on each of its keys – Nick Sep 14 '17 at 14:58
  • I post big update, please check and let me know if you have thoughts about it :) – Nick Sep 14 '17 at 15:19
  • 1
    total time, less than 6 days. size on disk almost 400 GB. – Nick Sep 16 '17 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.