0

I would like to do some research of the Bitcoin blockchain. Because i would like to do massive amounts of processing and lookups, I need a fast way to search the blockchain.

Http requests to insight.io just won't cut it...

I know of ABE but it seems no longer maintained and I don't know if it is up to par with the current implementation of the blockchain.

The environment I'm programming in is python.

Any ideas?

2

I held a research period on bitcoin (thanks to this project I became fond of Bitcoin) with the release of the alpha software at December, I had to extract information for the construction of an information graph.

I believe in my opinion that there are really many ways to index information but I believe that the fastest ones are to use the same technique that uses bitcoins.

  • Parser blk files
  • Deserialize the information in a format of your choice and index it in DB
  • query the DB and make a read in the appropriate file.

There are two very difficult things

  • the construction of this indexing system
  • if you want to access the id wallet you need to build a script decompiler or use the bitcoin node

ps: this is my experience, there can be other ways to build this information, which I don't know (mine is just a three-year thesis, my path has just begun)

This is a good parser for blk, one of the few compatible with Segregated Witness.

The abe project still works but his work times seem unacceptable with the current bitcoin dimension

If you have a big pc, this project is very fast (at least they say so)

Update 2021

The blockchain of Bitcoin grows over time and I think that the result of my work is SpyCBlock can be an efficient solution in terms of memory RAM, I don't know in terms of space.

SpyCBlock offers different deserialization methods like JSON, transactions graph, and a deprecated address graph that use Bitcoin Core to make some work. In addition, a method to deserialize in an efficient way the Bitcoin mempool is under development.

Please note that the software is an academics software, developed by only me, maybe can have some mistake and can be difficult to install all the dependencies, but if it will compile, it should be work fine with the integral bitcoin data.

P.S: The development of BlockSci is stopped in 2020, I don't know why, but I think because it is difficult to scale with the actual dimension of Bitcoin blockchain.

0

Depends on what you feel comfortable with.

You can go low level and query the lebeldb directly. The block index (information for every block) is in $DATADIR/blocks/index and the chainstate (information about the currently best known chain) in $DATADIR/chainstate.

An easier way is to have a fully synced bitcoind running locally and query over jsonrpc. From there you slice and dice the resulting JSON to fit your needs.

I haven't done leveldb querying, but I'd expect there to be a significant amount of work involved since it's just a key value store.

3
  • The idea is to try and track coins over long periods of time with an innovative approach. So even bitcoind won't really cut it I think. – Dennis Decoene Apr 7 '19 at 14:05
  • Neither of those LevelDB databases maintained by bitcoind actually contain the information you want. Maybe some more modern indexing software like electrs is more appropriate. – Pieter Wuille Apr 7 '19 at 21:07
  • A Bitcoin equivalent of this XRP project? – user58807 May 7 '19 at 15:40
0

The fastest (simpliest) way is to use my blockchain parser which has just 240 lines of code and able to extract all information from blkXXXXX.dat files to the simple text view. It parse whole the raw bitcoin blockchain database that is stored in those files.

Then you can easily index the data as you need and then make any data manipulations for your academic research. For example, you can build UTXO set by yourself, or make some ECDSA signatures or keys research and so on.

Also, you can learn how the data is stored in blockchain if you check how my parser works with the sequences of bytes.

I'am sure it's the fastest, simpliest and most usefull way to learn the blockchain functionality.

0

If you want to read the raw blkXXXXs, bitcoin-iterate by Rusty Russel is probably the best option.

For more advanced queries, you'll need an indexer. Rust-bitcoin-indexer by dbc and Blockstream's revamped electrs are two very different yet powerful alternatives.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.