How is the whitepaper decoded from the blockchain (Tx with ~1000x m of n multisig outputs)

Question

The whitepaper is apparently encoded at 54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713, which is an m of n multisig Tx with 947 outputs (just under the scriptsig limit of 20kB!).

Using the Blocktrail Python SDK, I get a list of the outputs as hex using the following Python (2.7) code (NB, the APIKEY, APISECRET parameters are available if required from www.blocktrail.com):

from blocktrail import APIClient
bt_client = APIClient(APIKEY, APISECRET, network='BTC')
txnObj = bt_client.transaction('54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713')
hashes = [(t['script_hex']) for t in (txnObj)['outputs']]   

The resulting list is available here in full and is essentially all pay-to-pubkey-script Txns. An excerpt:

[u'5141e4cf0200067daf13255044462d312e340a25c3a4c3bcc3b6c39f0a322030206f626a0a3c3c2f4c656e6774682033203020522f46696c7465722f466c617465446541636f64653e3e0a73747265616d0a789cad5c4b8b24b911becfafa8b3a1da292925654253d0d55373f06d61c007e39bbd061f0cde8bffbe25c55b5266f61ab3905d419ba54728e28bb76a963777fbcfb77fdf96db7d291f93f3e599f7fafcedefb73fffe1f6aff665fdefb77f7c7bfefce6c2fa166e695bdfd6dbcfbfddfef8c3dd5cf953ae',
.....
u'514130206e200a30303030313832353430203030303030206e200a747261696c65720a3c3c2f53697a652036382f526f6f74203636203020520a2f496e666f20363720413020520a2f4944205b203c43413142304134344244353432343533424546393138464643443436444330343e0a3c4341314230413434424435343234353342454641393138464643443436444330343e205d0a2f446f63436865636b73756d202f36463732454137353134444641443233464142434337413535303032314146370a3e53ae',
 u'51213e0a7374617274787265660a3138323732370a2525454f460a000000000000000051ae',
 u'76a91462e907b15cbf27d5425399ebf6f0fb50ebb88f1888ac',
 u'76a914031c79236ff3017496cf8d9a883f494458f245f288ac']

QUESTION: How is this array of hex data parsed into the bitcoin.pdf? Specific Python framed answers would be appreciated!

Answer 1

This is a fun little puzzle on the blockchain, basically. First, you need to know a little about pdf's and how they're structured, which you can find here.

Second, you'll note from section 3.4.1 that all pdf's start with this string:

%PDF-

In hex, that is 255044462d. And indeed that is in the very first output in the very first bare multisig pubkey:

<e4cf0200><067daf13>**255044462d**312e340a25c3a4c3bcc3b6c39f0a322030206f626a0a3c3c2f4c656e6774682033203020522f46696c7465722f466c6174654465

I haven't figured out what the first 8 bytes are for (♦edit: e4cf0200067daf13 = 2x 4byte little Endian "checksums", see @WizardOfOzzie comment below), but the rest of the bare multisig keys (everything in between 1 and 3 OP_CHECKMULTISIG in each output -- note the last one is a 1 of 1, so it's 1 OP_CHECKMULTISIG) are pieces of data for the pdf and they are in order. If you can put all the hex digits of the bare multisig keys into a single file (no whitespaces) called "fromblockchain.hex", you can run this very simple program to extract the pdf:

contents = open('fromblockchain.hex').read()
bytes = contents[16:].decode('hex')
f = open("bitcoin.pdf")
f.write(bytes)
f.close()

This should create a bitcoin.pdf which is the actual satoshi whitepaper. I've tested this and indeed it is the whitepaper. Good to know it's literally in the blockchain.

Alternatively, if you have bitcoind running on your machine, you can run this python script to grab the bitcoin whitepaper:

import subprocess

# raw = full hex of raw Tx using Bitcoin-cli
raw = subprocess.check_output(["bitcoin-cli", "getrawtransaction", "54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713"])

outputs = raw.split("0100000000000000")

pdf = ""
for output in outputs[1:-2]:
    # there are 3 65-byte parts in this that we need
    cur = 6
    pdf += output[cur:cur+130].decode('hex')
    cur += 132
    pdf += output[cur:cur+130].decode('hex')
    cur += 132
    pdf += output[cur:cur+130].decode('hex')

pdf += outputs[-2][6:-4].decode("hex")
f = open("bitcoin.pdf", "wb")
f.write(pdf[8:-8])
f.close()

Finally checking checksum

sha256sum bitcoin.pdf
b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553  bitcoin.pdf

Answer 2

This is a bash command that can also give you the file:

bitcoin-cli getrawtransaction 54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713 0 00000000000000ecbbff6bafb7efa2f7df05b227d5c73dca8f2635af32a2e949  | sed 's/0100000000000000/\n/g' | tail -n +2 | cut -c7-136,139-268,271-400 | tr -d '\n' | cut -c17-368600 | xxd -p -r > bitcoin.pdf

It doesn't need -txindex, but it does need a full node, not pruned. I suppose it's possible to make a variant that works with gettxout that works with pruned nodes, but that's left as an exercise for the reader.