I have a simple python app running through the blockchain as follows...

rpc_connection = AuthServiceProxy("http://%s:%[email protected]:8332"%(rpc_user, rpc_password))
for b in range(blockStart, blockEnd): 
    block = rpc_connection.getblock(rpc_connection.getblockhash(b)) 
    for txid in block['tx']:
        raw_tx = rpc_connection.getrawtransaction(txid)
        decoded_tx = rpc_connection.decoderawtransaction(raw_tx)
        for output in decoded_tx['vout']:

It successfully runs through all blocks from block 1 until block 115,257 where it encounters "BrokenPipeError: [Errno 32] Broken pipe". It errors on transaction 11 (with txid 1c67abba71d570d0f0643b3d4ed93f528a68ef1c6a21e6d1f72535a15b0ce030). Sometimes it errors on the rpc_connection.getrawtransaction line and sometimes it errors on the rpc_connection.deriveaddresses line but both times with a Broken pipe.

What's strange is that if I just run the script for the single transaction it works fine and I notice that the transaction before transaction 11 (i.e. transaction 10 with txid c45b2940e403a1598faaae33a24e385c92b4f051929a49fe3b079ef04b1e2dbc) returns an enormous result for rpc_connection.getrawtransaction() (pages and pages of console ouput) whereas all the others are about 3 lines of console output, and so I'm wondering if perhaps this is related to memory usage or timeouts or some conf setting that I don't have?

Has anyone run into a similar problem before? Also, I'm guessing everyone gets the same enormous result for rpc_connection.getrawtransaction('c45b2940e403a1598faaae33a24e385c92b4f051929a49fe3b079ef04b1e2dbc')? Really appreciate any thoughts on the best way around this problem.

  • What are you doing with the output of your script? Normally, BrokenPipeErrors occur when the process reading your script terminates early, e.g python myscript.py | head -n 100. head will terminate after 100 lines, but your script will still try to write to the pipe, which causes the BrokenPipeError. See stackoverflow.com/questions/26692284/… for more info
    – josie
    Commented Feb 22, 2023 at 11:14
  • thanks @josie - instead of the final "print" line in my example, I'm actually just appending to a list of dicts which once I exit the loop for a given block gets written to a file. The list is about 600k lines large so far which seems relatively small. And I'm just running directly like python3 myscript.py.
    – d3wannabe
    Commented Feb 22, 2023 at 11:46
  • p.s. @josie - I noticed this morning it's actually throwing a new error (but at the exact same line): ConnectionResetError: [Errno 54] Connection reset by peer
    – d3wannabe
    Commented Feb 22, 2023 at 11:49
  • It's difficult to tell what is a python error and what is a bitcoind error. perhaps you can try first running your script on signet. This will have fewer transactions and blocks and should at least tell you if the issue is related to the amount of data.
    – josie
    Commented Feb 22, 2023 at 14:12
  • 1
    just found a hacky fix: if I encounter a transaction with greater than a certain number of vins (around 200 in my experience), then I just reinstantiate the rpc_connection with rpc_connection = AuthServiceProxy("http://%s:%[email protected]:8332"%(rpc_user, rpc_password)) and it works. Not sure if there's a more elegant way to flush/restart the rpc_connection or whether I'm spawning multiple sessions, but at least there's no more connection errors
    – d3wannabe
    Commented Feb 22, 2023 at 17:55


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.