Getting orphaned Blocks from the Blockchain

Question

I'm very interested in doing some analysis of orphaned blocks, but to do so I'd have to have a good way of accessing them. Blockchain.info seems like a good place to start but I don't want to abuse their API by scraping the entire Blockchain. Another attempt was to use ABE but after parsing nearly all the blockchain from my Disk (several days of work) I am unable to find a single orphaned block, and I'm starting to fear that the satoshi client prunes old orphaned blocks, which would be a pity.

Will I be able to find orphaned blocks with ABE, and if not what alternatives are there?

Answer 1

Orphaned blocks are stored in blkxxxx.dat files forever, though each node will know about different orphaned blocks. Bitcoin will print a list of all known blocks, including orphans, to debug.log if you pass it -printblocktree.

Answer 2

From what I can see, ABE works pretty much like the BlockExplorer in regards to orphaned blocks, which is to say it forgets them. It appears that only Blockchain.info actively stores orphaned blocks, and even displays them in a neat list.

The only other way to analyse orphaned blocks is to keep a history of the last couple blocks and constantly keep checking whether they changed. This is the approach I used for testing a simulated 51% attack on the TestNet in my master thesis. However, orphaned blocks don't occur all that often. You can look into a question on that, or go directly to the stats.

Answer 3

Regarding Abe, it should load orphan blocks from the blockchain if you use the blkfile loader... I haven't looked in detail the use of the orphan_block table but it's empty on both my db's despite the fact I do have orphan blocks in them (I can't say it's a bug yet but it sure looks fishy).

You can search orphan block by hash in Abe, but to find the hashes you need to look directly in the database. This query works for me on MySQL, using binary hash storage (hence the HEX() function); be sure to remove it if you store hashes as CHAR hex strings (used to be the default).

SELECT block_height, HEX(block_hash) FROM block b
  LEFT JOIN block_next bn ON (b.block_id = bn.block_id)
  WHERE bn.next_block_id IS NULL
  ORDER BY b.block_id DESC LIMIT 1,10;

The LIMIT 1,10 is to skip the most recent loaded block which is most likely the tip; it has no block_next_id but it's not orphan neither.

If you use the RPC loader, you can only catch orphan blocks while loading the latest blocks; it won't receive any orphan from bitcoind unless if it hasn't yet been orphaned! You may catch some using abe_loader and eventually with some code I'm still testing that will allow constantly loading mempool and blocks.