I would like to do some research of the Bitcoin blockchain. Because i would like to do massive amounts of processing and lookups, I need a fast way to search the blockchain.
Http requests to insight.io just won't cut it...
I know of ABE but it seems no longer maintained and I don't know if it is up to par with the current implementation of the blockchain.
The environment I'm programming in is python.
Any ideas?
Depends on what you feel comfortable with.
You can go low level and query the lebeldb directly. The block index (information for every block) is in $DATADIR/blocks/index and the chainstate (information about the currently best known chain) in $DATADIR/chainstate.
An easier way is to have a fully synced bitcoind running locally and query over jsonrpc. From there you slice and dice the resulting JSON to fit your needs.
I haven't done leveldb querying, but I'd expect there to be a significant amount of work involved since it's just a key value store.
I held a research period on bitcoin (thanks to this project I became fond of Bitcoin) with the release of the alpha software at dicembe, I had to extract information for the construction of an information graph.
I believe in my opinion that there are really many ways to index information but I believe that the fastest ones are to use the same technique that uses bitcoins.
There are two very difficult things - the construction of this indexing system - if you want to access id wallet you need to build a script decompiler or use the bitcoin node
ps: this is my experience, there can be other ways to build this information, which I don't know (mine is just a three-year thesis, my path has just begun)
This is a good parser for blk, one of the few compatible with Segregated Witness.
The abe project still works but his work times seem unacceptable with the current bitcoin dimension
If you have a big pc, this project is very fast (at least they say so)
