2

Currently, it takes me almost 1.5 hours to obtain all inputs and outputs of a bitcoin block from January using JSON RPC. At this pace, it will take a significant amount of time to go all the way back to genesis. I am starting to think that I am using the wrong API call, or doing something else wrong. I would be greatly thankful if any of you could review the calls that I am currently using (pasted below) and perhaps tell me what I am doing wrong, or if bitcoind is the appropriate client for this.

from bitcoinrpc.authproxy import AuthServiceProxy, JSONRPCException
import json
import requests
import pprint
import time
import os


objectivestart_time=time.time()
pp= pprint.PrettyPrinter(indent=4)



rpc_user='n'
rpc_password='n'

url=("http://%s:%s@1"%(rpc_user, rpc_password))


def req(payload):
    return requests.post(url,data=json.dumps(payload)).json()


def BlockHash_JsonRPCpayload(number):
    dic={

    "jsonrpc":"2.0",
    "method": "getblockhash",
    "params": [number] ,
    "id": 1 

    }
    return (dic)


def Block_JsonRPCpayload(h):
    dic={

    "jsonrpc":"2.0",
    "method": "getblock",
    "params": [str(h)] ,
    "id": 1 

    }
    return (dic)


def rawtx_JsonRPCpayload(tx):
    dic={

    "jsonrpc":"2.0",
    "method": "getrawtransaction",
    "params": [tx, True] ,
    "id": 1 

    }
    return (dic)

def rawvin_JsonRPCpayload(tx,vin):
    block_payload={

    "jsonrpc":"2.0",
    "method": "getrawtransaction",
    "params": [tx,1] ,
    "id": 1 

    }

    request=req(block_payload)
    value=request['result']['vout'][int(vin)]['value']


    return float(value)

BTC_dic={}


hash_payload=[]
start_time=time.time()

block_start=481101
block_end=490000
start=block_start
end= block_start

while end < block_end:
    block_tries=0
    while True: 
        try:    

            end=start+100
            hash_payload=[]
            print ( 'start', 'end', start, end )
            for i in range(start,end):
                hash_payload.append(BlockHash_JsonRPCpayload(i))


            start_time=time.time()
            print ( 'mark 1 ')
            hash_response=requests.post(url, data=json.dumps(hash_payload)).json()
            print('time to get hash batch',start, "_", end, time.time()-start_time)
            print ( 'success in getting hashes for ranges', start, "_ ", end )

            time.sleep(0.4)

            block_payload=[]
            for i in hash_response:
                h=i['result']
                block_payload.append(Block_JsonRPCpayload(h))
            start_time=time.time()
            block_response=requests.post(url,data=json.dumps(block_payload)).json()
            print ( 'time to get block batch ',start, "_", end, time.time()-start_time)
            print ( 'success in getting blocks for ranges', start, "_ ", end )

            time.sleep(0.4)



            for i in range(len(block_response)):


                block=block_response[i]['result']

                height=block['height']
                print('getting', len(block['tx']),'transactions for block ', height)
                BTC_dic[height]=block
                tx_payload=[]
                for tx in block['tx']:
                    tx_payload.append(rawtx_JsonRPCpayload(tx))

                try_errors=0
                while True:
                    try:
                        #print ( len(tx_payload), 'number of transactions in block ', height)
                        start_time=time.time()
                        transaction=requests.post(url,data=json.dumps(tx_payload)).json()
                        time.sleep(0.5)
                        #print ('time to get tx', time.time()-start_time)
                        print ( 'success in getting transactions for block ', height )
                    except:
                        try_errors+=1
                        print ( '!! Try error !!')
                        print ( try_errors, ' transaction tries for block height', height)
                        time.sleep(3)
                        continue

                    break


                for  i in range(len(transaction)):


                    tx_result=transaction[i]['result']
                    vin=tx_result['vin']
                    vout=tx_result['vout']



                    try:
                        fees=0
                        for j in range(len(vout)):
                            fees -= tx_result['vout'][j]['value']

                        for j in range(len(vin)):

                            vout=vin[j]['vout']
                            txid=vin[j]['txid']

                            vin_error=0
                            while True:
                                try:
                                    value=rawvin_JsonRPCpayload(txid,vout)
                                except:
                                    print ( '!! vin error !!')
                                    print ( 'attempt', vin_error )
                                    vin_error+=1

                                    time.sleep(0.4)
                                    continue
                                break 

                            fees+=value
                            tx_result['vin'][j]['value']=value



                        tx_result['fees']=fees
                    except:
                        tx_result['vin'][0]['value']='No Inputs (Newly Generated Coins)'
                        tx_result['fees']=0


                    BTC_dic[height]['tx'][i]=tx_result



                if len(BTC_dic) >= 100:
                    print ( 'len BTC_dic =', len(BTC_dic))


                    for block in BTC_dic:
                        block_fee=0
                        transactions= BTC_dic[block]['tx']


                        #pp.pprint(transactions)
                        for j in transactions:
                            fee=j['fees']
                            block_fee+=fee
                        BTC_dic[block]['block fees']=block_fee



                    block_array=[]
                    for block in BTC_dic:
                        block_array.append(block)


                    file_name='jcampbell_btc_txdata'+str(min(block_array))+'_'+str(max(block_array))+'.json'
                    print ('saving', file_name, 'time', time.time()-objectivestart_time)

                    with open(file_name, 'w') as outfile:
                        json.dump(BTC_dic,outfile)
                    BTC_dic={}




            start=end



        except:
            block_tries+=1
            print(block_tries, 'tries to get data for block range ', start,"_",end)
            time.sleep(5)
            continue
        break 




for block in BTC_dic:
    block_fee=0
    transactions= BTC_dic[block]['tx']


    #pp.pprint(transactions)
    for j in transactions:
        fee=j['fees']
        block_fee+=fee
    BTC_dic[block]['block fees']=block_fee





block_array=[]
for block in BTC_dic:
    block_array.append(block)


file_name='jcampbell_btc_txdata'+str(min(block_array))+'_'+str(max(block_array))+'.json'
print ('saving', file_name, 'time', time.time()-objectivestart_time)
with open(file_name, 'w') as outfile:
    json.dump(BTC_dic,outfile)

Background

If we want BTC to go mainstream, we need a better way of showing Economic Activity. I need you to help me prove that Bitcoin is more than a speculative instrument.

I come from academia (Economics, specifically), and I often find it difficult to convincingly describe to my peers the growth of the Bitcoin ecosystem in numbers that translate to economic value.

Given the rapid growth this market has experienced, when people think about Bitcoin, price is often the first thing the comes to mind. For those of us that for some time have been researching Bitcoin, six thousand US dollars per bitcoin is, irrefutably, an incredible indicator of popularity. However, we need a better metric to show actual economic activity, which has also increased substantially in the past couple of years. If Bitcoin is a peer-to-peer electronic cash system, we need a better indicator of BTC volume between network peers. I have started doing research on Bitcoin Days Destroyed, but I often find this metric difficult to be explained as it does not accurately answers the question: how much value is being exchanged within the network?

I understand that a completely accurate answer to this question is nearly impossible due to Bitcoin's UTXO model, but we need more accurate estimations. Blockchain.info provides estimated transaction volume on a per day basis, but random samples I have processed indicate that their methodology significantly underestimates volume. After months of testing random samples and manually looking at addresses, I believe I have a developed a more accurate methodology to estimate on-chain volume. I plan to share this methodology with the community once I can test my hypothesis using fuller blocks, but the challenge now is to obtain all inputs and outputs from more recent blocks.

I believe a more realistic volume estimation backed by a peer reviewed methodology would increase interest from academia and add credibility to the Bitcoin network. If you are not familiar with JSON RPC, I would appreciate if you could please upvote this post so that more people see it.

Thank you.

-JC

  • Welcome to the site! It's a good question, but we need to keep questions clear so that we can get good answers, so I've edited the question to try to improve clarity. Feel free to update it if you can see any way of improving it. – Highly Irregular Oct 30 '17 at 20:37
1

I like the background information, and the underlying assumptions, with the risk of going into a value based discussion, instead of answering directly, how RPC calls can be improved... :-)

If we want BTC to go mainstream, we need a better way of showing Economic Activity. I need you to help me prove that Bitcoin is more than a speculative instrument.

I come from academia (Economics, specifically), and I often find it difficult to convincingly describe to my peers the growth of the Bitcoin ecosystem in numbers that translate to economic value.

First thing is "who is we"? I met people wo are perfectly ok, that bitcoin is a usable value transfer mechanism, that provides me with some anonymity features, and is apparently censorship resistant. So it wold fulfill already my requirements (values), and in this case bitcoin could be already too mainstream. When greed comes into the game, and people are too limited to create their own group with a clever invention to benefit from, then is it legitimate, to apply "their" rules to "my" bitcoin world? (hypothetical question)

On economics: imho the perspective of economics is mostly gain oriented. Something must have a value, so I can gain from or with it. If there is no intrinsic value, nothing to gain from it: like digging a hole in the desert. And this seems to be what we do with Bitcoin mining. Digging holes... Every other work tries to create something with value, crafting a sculpture, drawing a picture - mostly in the sense of tangible assets. And writing computer programs. In bitcoin world digging holes is to secure the network, this whole idea is not yet understood. That's why it is compared to the tulip bubble in the Netherlands. The value behind is not recognized. Time will be a very good judge, if bitcoin is or is not a bubble.

A provocative question: do "we" really need to convince more people, with a price beyond 6000 US$? I would say no :-) I went into the game at less than 100, so I am ok (and mostly done). It was my toy to play with bitcoins, and I was lucky to benefit from it. But all newcomers probably entered at 5800, or 5900, or even 6000. They are too late, and lost the opportunities. So all these 58, 59 or 6000 entry guys are desperately looking for a way to achieve the same "gain", just without risk and very little effort. I don't see, why "these" guys should have this benefit - they should stay in the economics, invent something on their own, e.g. create s.th. of value for society, sell it to the society, and get this way rich. But I doubt, they are able to do so...

how much value is being exchanged within the network? I understand that a completely accurate answer to this question is nearly impossible due to Bitcoin's UTXO model

not sure what you want to prove here. As we can decompose each block (it is a public ledger!), we can see the transactions inside, and see how much value was moved from a to b - independant of the UTXO model. Simply you don't know, and should never know (anonymity!), how often I tumbled my bitcoins, or even how many addresses I own. A transaction then doesn't mean, I have exchanged value - maybe I just shifted from my hot wallet to my cold storage system, and vice versa. So I come to the conclusion, that your effort is either government (or interest group) oriented, to further control individuals. Bitcoin's design goal was to prevent things like this. With this regard, the question to improve RPC/JSON doesn't make sense... Apologies, as non native speaker I don't want to appear being to arrogant, just my two cents in the economics debate, which is not yet ripe to understand the tidal wave of change coming along...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.