development of bitcoin-abe like software

Question

I am trying to develop something similar to bitcoin-abe, e.g. import whole blockchain into database.

For database I currently using Redis - I know it will took lots of RAM, but lets suppose that is OK.

Another possibilities could be MySQL, Postgress or even CSV file.

I have bitcoind and I am importing block after block.

For each transaction, I need to look for output of the transactions that "fund" current transaction - I hope you understand what I mean.

I am currently checking with blocks from 2017. Most of blocks have 2000+ transactions. If we assume each transaction have single input, this means I need to look at 2000 more transactions.

At the moment those 2000 lookups are done back to bitcoind, because output database could be csv file.

I can do all this in "lazy" way, but then I will not be able to find if some address is spent.

Is there any faster way to import?

I am using PHP and single block from 2017 is imported for about 1 min.

Answer 1

output is spent (e.g. if appear in "vin" to some other transaction), I store an indicator.

Now single block is processed for 1-2 sec.

Implication of this is,
if you want to check address balance, you will need to check each "input" and to compile list of unspent inputs.

Still unsure if this is best way to go.

Update 1

I finally upload the software in github here:
https://github.com/nmmmnu/BlockchainWalker

Update 2

Here are how keys looks like if we want to find the balance for address "1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6":

127.0.0.1:2000> HGETALL a:1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6
 1) "a:1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0"
 2) "0.01270199"

This is funding transaction:

• 28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0, output 0

This is value:

• 0.01270199

Then we need to check if this is spent, so we check each input as follows:

127.0.0.1:2000> get t:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0:s
"055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8"

Transaction 055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8, is the transaction where this output is spent.

As each output is spent this means the address 1EoMoJK3FJPHg4EwrP31zPVu4iLqmCtQ6 have balance of zero.

---

In case we want to check / list a transaction "055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8", we can do:

127.0.0.1:2000> HGETALL t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8
 1) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.-:i:28c76c9d89f23c1b1f5435d8f4bb5cc66e6cb9d0798f1f0f4293faaac88fb7d0.0"
 2) "1"
 3) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.-:i:39edb8741b701b6da2dbc4e02290e8e78cba244bdbad96da203e41ee2704c525.0"
 4) "1"
 5) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.0:o"
 6) "1Kj76Sxe8c3UK85RAQwwdqScAxaBwAY2eb:0.00500000"
 7) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.0:s"
 8) "49f8bd582439a3b2351f92e0fdb5fcb1032acd42e3ab469c16d805627889ce14"
 9) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.1:o"
10) "14yAJga4ZkULbaMz4LUW5vj8GhQYyzCPoW:0.01118727"
11) "t:055bd8148143c5b05bc2808ccafe54be43b292381449d41cc23462d02d3f85d8.1:s"
12) "edf55ace1396229c7a91a4b21cd63293c631ae2ddcc5987402b04ed540b9e5d8"

Those "t:xxxxx.-:i:xxxx" are inputs.

Those "t:xxxxx.N:o" are outputs

Those "t:xxxxx.N:s" are indicators, if output is spent or not.

---

Please note this is not Redis, it is database I developing that have Redis interface. You can check it here:

https://github.com/nmmmnu/HM4

Answer 2

I am using RocksDB and node.js. My database is optimized for tracking address balances and payment history. It needs around 143GB of storage, 4 days to load. Current blocks import below 2s.

I've tried very hard to achieve this with PostgreSQL. It is doable, but requires a lot of hacking and post-import index creating. I've tried levelDB but it is roughly 4 times slower than rocksDB on SSD drive.

I am using custom block parsing library and half-baked full-node implementation.