Get transaction data from full node

Question

Background Information

I am building an application that requires the ability to query any bitcoin full node (from a python script using raw tcp sockets) in order to read the OP_Return value listed in the following transaction (https://live.blockcypher.com/btc-testnet/tx/2599dbe540a583ede3512fef9a0f26be718c039ffd4d04d85ff3b339f40e73b1/)

What I have done to date

I have successfully managed to establish the version + verack, however I was faced with two issues.

1. When I send a getdata for the necessary transaction Idit does not return the result. I have identified this is because my initial assumptions that getdata returns the necessary data was incorrect and instead it will only return transaction data for transactions still in mempool.

2. This lead me to my main issue, I was hoping to replicate the initial block download on my python script, starting from the block containing the transaction I'm interested in. However when I do getblocks/getheaders no matter what I seem to put in the hash filter I get a dump of 500 (or 2000) blocks/headers. Below is a hex dump of the requests I am making (just the bitcoin protocol frame):

   • getblocks 0b110907676574626c6f636b00000000450000002a0af9950100000001000000000000592589e55cda6e8a093998e8356ea770d4aaeb7c0f5439b147d7000000000000017a09017d52db538d7a9ddcc48311866d7e5fdbbbec7d0faad5

   • getheaders 0b110907676574686561646572730000450000002a0af9950100000001000000000000592589e55cda6e8a093998e8356ea770d4aaeb7c0f5439b147d7000000000000017a09017d52db538d7a9ddcc48311866d7e5fdbbbec7d0faad5

I am looking for the following help:

1. How can I query for transaction data without using bitcoin RPC if I can't use the getdata message?
2. How do I get my block or headers response to only contain 1 inv item instead of 500/5000

EDIT (4/5/20) The implied requirement which wasn't made clear here was that we needed to do so in a decentralised way using the blockchain RPC api or equivalent.

Answer 1

I don't see any endpoint to get the whole transaction in Blockcypher API documentation, so I used blockstream.info API:

>https://blockstream.info/testnet/api/tx/2599dbe540a583ede3512fef9a0f26be718c039ffd4d04d85ff3b339f40e73b1
{ 
   "txid":"2599dbe540a583ede3512fef9a0f26be718c039ffd4d04d85ff3b339f40e73b1",
   "version":2,
   "locktime":0,
   "vin":[ 
      { 
         "txid":"382bda6ee2bac22afd051104af16146436895ecca382a76c2d66535a837254bc",
         "vout":1,
         "prevout":{ 
            "scriptpubkey":"00144ecdb2867150ff16c7d6e3258adca9ed8bac3ac1",
            "scriptpubkey_asm":"OP_0 OP_PUSHBYTES_20 4ecdb2867150ff16c7d6e3258adca9ed8bac3ac1",
            "scriptpubkey_type":"v0_p2wpkh",
            "scriptpubkey_address":"tb1qfmxm9pn32rl3d37kuvjc4h9fak96cwkpjyyjmk",
            "value":2948920
         },
         "scriptsig":"",
         "scriptsig_asm":"",
         "witness":[ 
            "304402205e9b17758d1daf2a501a1fabe06bdbf15adaadfcf6ff80c4cfb34176bd988c0c02207450185bde5c20e06270b3b7e681e0042072c6af25fc10c62430df0f04ec5bf201",
            "023532fa6b866f37729719a3615ef3a776407048989736ac2b7fafa5678a519df3"
         ],
         "is_coinbase":false,
         "sequence":4294967295
      }
   ],
   "vout":[ 
      { 
         "scriptpubkey":"6a0e3132372e302e302e313a38313831",
         "scriptpubkey_asm":"OP_RETURN OP_PUSHBYTES_14 3132372e302e302e313a38313831",
         "scriptpubkey_type":"op_return",
         "value":0
      },
      { 
         "scriptpubkey":"a914171f697fe358f7d7238b6128930bb1fa7363b44d87",
         "scriptpubkey_asm":"OP_HASH160 OP_PUSHBYTES_20 171f697fe358f7d7238b6128930bb1fa7363b44d OP_EQUAL",
         "scriptpubkey_type":"p2sh",
         "scriptpubkey_address":"2MuMVAFK2gbTDNVBhLtpZfA4Xi62mWh6bU5",
         "value":948920
      },
      { 
         "scriptpubkey":"0014af8abd63a00c40ad75e47870b1b0a1ddb2ba7e66",
         "scriptpubkey_asm":"OP_0 OP_PUSHBYTES_20 af8abd63a00c40ad75e47870b1b0a1ddb2ba7e66",
         "scriptpubkey_type":"v0_p2wpkh",
         "scriptpubkey_address":"tb1q479t6caqp3q26a0y0pctrv9pmket5lnx24twcx",
         "value":1500000
      }
   ],
   "size":248,
   "weight":665,
   "fee":500000,
   "status":{ 
      "confirmed":true,
      "block_height":1612110,
      "block_hash":"000000000000592589e55cda6e8a093998e8356ea770d4aaeb7c0f5439b147d7",
      "block_time":1576024567
   }
}

You can see here that the first output has the data is 3132372e302e302e313a38313831, which decoding to ASCII characters seems like a local URL:

hex = '3132372e302e302e313a38313831'
decoded = binascii.unhexlify(hex).decode('ascii')
print(decoded)

Prints: 127.0.0.1:8181

EDIT:

To get the transaction that interests you without using any external API you will need to run a full bitcoin node with the txindex=1 option. If this transaction is tracked by your wallet, a pruned node would suffice. Then you can get the information about the transaction with this command:

>bitcoin-cli getrawtransaction 2599dbe540a583ede3512fef9a0f26be718c039ffd4d04d85ff3b339f40e73b1 true

You should get something very similar to the answer from blockstream's API I pasted above

Answer 2

The bitcoin p2p protocol does not directly permit transaction lookups this way. Maybe some bloom filtering would minimize the footprint, but it's still relatively complex to implement.

Electrum servers are maybe the next best thing - they do support the lookups you want, but they're less decentralized, and if your application is doing naughty things they could collude and manipulate only your results.

How do I get my block or headers response to only contain 1 inv item instead of 500/5000

I don't think you can limit the headers results (you have to know all the headers to do so) But once you have the header chain, you can send a block locator with start:HASH(n) end:HASH(n+1) and you can download a single block.

Re bloom filtering, if you find nodes with the BLOOM_SERVICE bit you can tell them your filter, and the block data only contains your txs and the false positives allowed by your filter.

Answer 3

It turns out the main issue with the above code was that the the endianness in python was defaulting to my hex payload being mixed up. I simply needed to correct the ordering of the payload. The correct code can be viewed here: https://github.com/dummytree/blockchain-botnet-poc/blob/master/MalwareManager.py#L284-L299