3

I have a python app to track and store utxo in a simple/flat structure that looks like this...

from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException 

# track the utxo
trackUtxoList = []

# rpc call 1
rpc_connection = AuthServiceProxy("http://%s:%s@127.0.0.1:8332"%(rpc_user, rpc_password))

for b in range(blockStart, blockLimit): 

    # rpc call 2 (per block)
    blockhash = rpc_connection.getblockhash(b)

    # rpc call 3 (per block)
    block = rpc_connection.getblock(blockhash)

    for txid in block['tx']:
        
        # rpc call 4 (per tx)
        raw_tx = rpc_connection.getrawtransaction(txid,False,blockhash) 

        # rpc call 5 (per tx)
        decoded_tx = rpc_connection.decoderawtransaction(raw_tx)

        for output in decoded_tx['vout']:

            # rpc call 6 (per vout)
            for l in rpc_connection.deriveaddresses(output['scriptPubKey']['desc']):
                trackUtxoList.append({'address':str(l),'value':output['value'],'block':b, 'txid':txid,'isUtxo':True})

        for output in decoded_tx['vin']:

            vinSearch = list(filter(lambda item: item['t'] == output['txid'], trackUtxoList))
            vinSearch[output['vout']]['isUtxo'] = False

The problem is that it runs very slowly for 2 reasons that relate to my 2 questions...

  1. My blocks directory is on an external hard drive because it's too big for my laptop - pretty sure this will impact read-speeds and so I'm wondering if it's possible to somehow split the blocks directory so that for example, I could pull 100k blocks onto my laptop, process that, then delete and copy the next 100k blocks onto my laptop etc. I've already split the blocks directory (on external hdd) from the rest of the data directory (on my laptop) but can't find any examples of people temporarily splitting the blocks directory itself.

  2. I'm making 2 rpc calls per block, 2 per transaction and 1 per vout. To my eye it looks like calls 5 + 6 (as commented in my code example) don't really need to use my rpc_connection since they are manipulations on data already pulled from the blockchain? Whereas the others are essentials/unavoidable? If that's correct, are there any python libraries that are typically used to decode transactions and derive addresses without using an rpc call? I played a little with Buterin's https://pypi.org/project/bitcoin/ but since it's no longer maintained I wasn't sure whether it's my best bet.

Thanks for any thoughts at all on how to optimise my code to parse the entire blockchain

Improve this question

1 Answer 1

Reset to default
4

You're going to see much better performance by doing the parsing of block data yourself directly off disk, though that involves a decent amount of complexity.

If you want to stick with bitcoind RPC calls, I suggest using getblock(blockhash, 2), which will decode all transactions in the block for you in a single result, removing the need for decoderawtransaction.

Lastly, why the roundtrip through deriveaddresses? output['scriptPubKey']['address'] should give you the address directly.

Improve this answer
2
  • wow - thanks so much for the tip on getblock(blockhash, 2) - didn't know that was possible but definitely sounds more efficient. On your second question, sometimes the ['address'] key simply isn't present - in fact normally it isn't - if I could reliably access that field for every transaction I would definitely use it. Not sure if there's any well-known non-RPC methods (or python tools) to derive an address from a descriptor?
    – d3wannabe
    Feb 24 at 16:29
  • 1
    You can just check whether the field exists, before accessing it.
    – Pieter Wuille
    Feb 24 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.