I would like to do some research of the Bitcoin blockchain. Because i would like to do massive amounts of processing and lookups, I need a fast way to search the blockchain.
Http requests to insight.io just won't cut it...
I know of ABE but it seems no longer maintained and I don't know if it is up to par with the current implementation of the blockchain.
The environment I'm programming in is python.
Any ideas?
I held a research period on bitcoin (thanks to this project I became fond of Bitcoin) with the release of the alpha software at December, I had to extract information for the construction of an information graph.
I believe in my opinion that there are really many ways to index information but I believe that the fastest ones are to use the same technique that uses bitcoins.
There are two very difficult things
ps: this is my experience, there can be other ways to build this information, which I don't know (mine is just a three-year thesis, my path has just begun)
This is a good parser for blk, one of the few compatible with Segregated Witness.
The abe project still works but his work times seem unacceptable with the current bitcoin dimension
If you have a big pc, this project is very fast (at least they say so)
The blockchain of Bitcoin grows over time and I think that the result of my work is SpyCBlock can be an efficient solution in terms of memory RAM, I don't know in terms of space.
SpyCBlock offers different deserialization methods like JSON, transactions graph, and a deprecated address graph that use Bitcoin Core to make some work. In addition, a method to deserialize in an efficient way the Bitcoin mempool is under development.
Please note that the software is an academics software, developed by only me, maybe can have some mistake and can be difficult to install all the dependencies, but if it will compile, it should be work fine with the integral bitcoin data.
P.S: The development of BlockSci is stopped in 2020, I don't know why, but I think because it is difficult to scale with the actual dimension of Bitcoin blockchain.
Depends on what you feel comfortable with.
You can go low level and query the lebeldb directly. The block index (information for every block) is in $DATADIR/blocks/index and the chainstate (information about the currently best known chain) in $DATADIR/chainstate.
An easier way is to have a fully synced bitcoind running locally and query over jsonrpc. From there you slice and dice the resulting JSON to fit your needs.
I haven't done leveldb querying, but I'd expect there to be a significant amount of work involved since it's just a key value store.
The fastest (simpliest) way is to use my blockchain parser which has just 240 lines of code and able to extract all information from blkXXXXX.dat files to the simple text view. It parse whole the raw bitcoin blockchain database that is stored in those files.
Then you can easily index the data as you need and then make any data manipulations for your academic research. For example, you can build UTXO set by yourself, or make some ECDSA signatures or keys research and so on.
Also, you can learn how the data is stored in blockchain if you check how my parser works with the sequences of bytes.
I'am sure it's the fastest, simpliest and most usefull way to learn the blockchain functionality.
If you want to read the raw blkXXXXs, bitcoin-iterate by Rusty Russel is probably the best option.
For more advanced queries, you'll need an indexer. Rust-bitcoin-indexer by dbc
and Blockstream's revamped electrs are two very different yet powerful alternatives.
