How is the whitepaper decoded from the blockchain (Tx with ~1000x m of n multisig outputs)

Question

The whitepaper is apparently encoded at 54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713, which is an m of n multisig Tx with 947 outputs (just under the scriptsig limit of 20kB!).

Using the Blocktrail Python SDK, I get a list of the outputs as hex using the following Python (2.7) code (NB, the APIKEY, APISECRET parameters are available if required from www.blocktrail.com):

from blocktrail import APIClient
bt_client = APIClient(APIKEY, APISECRET, network='BTC')
txnObj = bt_client.transaction('54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713')
hashes = [(t['script_hex']) for t in (txnObj)['outputs']]   

The resulting list is available here in full and is essentially all pay-to-pubkey-script Txns. An excerpt:

[u'5141e4cf0200067daf13255044462d312e340a25c3a4c3bcc3b6c39f0a322030206f626a0a3c3c2f4c656e6774682033203020522f46696c7465722f466c617465446541636f64653e3e0a73747265616d0a789cad5c4b8b24b911becfafa8b3a1da292925654253d0d55373f06d61c007e39bbd061f0cde8bffbe25c55b5266f61ab3905d419ba54728e28bb76a963777fbcfb77fdf96db7d291f93f3e599f7fafcedefb73fffe1f6aff665fdefb77f7c7bfefce6c2fa166e695bdfd6dbcfbfddfef8c3dd5cf953ae',
.....
u'514130206e200a30303030313832353430203030303030206e200a747261696c65720a3c3c2f53697a652036382f526f6f74203636203020520a2f496e666f20363720413020520a2f4944205b203c43413142304134344244353432343533424546393138464643443436444330343e0a3c4341314230413434424435343234353342454641393138464643443436444330343e205d0a2f446f63436865636b73756d202f36463732454137353134444641443233464142434337413535303032314146370a3e53ae',
 u'51213e0a7374617274787265660a3138323732370a2525454f460a000000000000000051ae',
 u'76a91462e907b15cbf27d5425399ebf6f0fb50ebb88f1888ac',
 u'76a914031c79236ff3017496cf8d9a883f494458f245f288ac']

QUESTION: How is this array of hex data parsed into the bitcoin.pdf? Specific Python framed answers would be appreciated!

Answer 1

This is a fun little puzzle on the blockchain, basically. First, you need to know a little about pdf's and how they're structured, which you can find here.

Second, you'll note from section 3.4.1 that all pdf's start with this string:

%PDF-

In hex, that is 255044462d. And indeed that is in the very first output in the very first bare multisig pubkey:

<e4cf0200><067daf13>**255044462d**312e340a25c3a4c3bcc3b6c39f0a322030206f626a0a3c3c2f4c656e6774682033203020522f46696c7465722f466c6174654465

I haven't figured out what the first 8 bytes are for (♦edit: e4cf0200067daf13 = 2x 4byte little Endian "checksums", see @WizardOfOzzie comment below), but the rest of the bare multisig keys (everything in between 1 and 3 OP_CHECKMULTISIG in each output -- note the last one is a 1 of 1, so it's 1 OP_CHECKMULTISIG) are pieces of data for the pdf and they are in order. If you can put all the hex digits of the bare multisig keys into a single file (no whitespaces) called "fromblockchain.hex", you can run this very simple program to extract the pdf:

contents = open('fromblockchain.hex').read()
bytes = contents[16:].decode('hex')
f = open("bitcoin.pdf")
f.write(bytes)
f.close()

This should create a bitcoin.pdf which is the actual satoshi whitepaper. I've tested this and indeed it is the whitepaper. Good to know it's literally in the blockchain.

Alternatively, if you have bitcoind running on your machine, you can run this python script to grab the bitcoin whitepaper:

import subprocess

# raw = full hex of raw Tx using Bitcoin-cli
raw = subprocess.check_output(["bitcoin-cli", "getrawtransaction", "54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713"])

outputs = raw.split("0100000000000000")

pdf = ""
for output in outputs[1:-2]:
    # there are 3 65-byte parts in this that we need
    cur = 6
    pdf += output[cur:cur+130].decode('hex')
    cur += 132
    pdf += output[cur:cur+130].decode('hex')
    cur += 132
    pdf += output[cur:cur+130].decode('hex')

pdf += outputs[-2][6:-4].decode("hex")
f = open("bitcoin.pdf", "wb")
f.write(pdf[8:-8])
f.close()

Finally checking checksum

sha256sum bitcoin.pdf
b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553  bitcoin.pdf

Answer 2

This is a bash command that can also give you the file:

bitcoin-cli getrawtransaction 54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713 0 00000000000000ecbbff6bafb7efa2f7df05b227d5c73dca8f2635af32a2e949  | sed 's/0100000000000000/\n/g' | tail -n +2 | cut -c7-136,139-268,271-400 | tr -d '\n' | cut -c17-368600 | xxd -p -r > bitcoin.pdf

It doesn't need -txindex, but it does need a full node, not pruned. I suppose it's possible to make a variant that works with gettxout that works with pruned nodes, but that's left as an exercise for the reader.

Answer 3

The PDF uses Python uploader/downloader scripts present in the Bitcoin blockchain itself

Those scripts are present at:

• uploader: tx 4b72a223007eab8a951d43edc171befeabc7b5dca4213770c88e09ba5b936e17 block 229991 reproduced at: https://gist.github.com/cirosantilli/ade4dde7c2f2f5020d792872681763e8
• downloader: tx 6c53cd987119ef797d5adccd76241247988a0a5ef783572a9972e7371c5fb0cc block 229991 reproduced at https://gist.github.com/cirosantilli/e90bd2e6c3fab25a20898e61e3ab3e90

These are my forks of Ken Shirriff's Gists who presumably found them while researching for: http://www.righto.com/2014/02/ascii-bernanke-wikileaks-photographs.html

Both scripts have the copyright header:

#
# File downloader
# Requires git://github.com/jgarzik/python-bitcoinrpc.git
#
# (c) 2013 Satoshi Nakamoto All Rights Reserved
#
# UNAUTHORIZED DUPLICATION AND/OR USAGE OF THIS PROGRAM IS PROHIBITED BY US AND INTERNATIONAL COPYRIGHT LAW

which is likely a joke and not really written by Satoshi in my opninon.

Both uploader and downloader are uploaded with the uploader encoding, which creates a slight chicken-and-egg bootstrap issue to download the downloader. But the encoding is so simple that you could just download the raw downloader data and fix it up manually.

After downloading the downloader manually, I managed to run it to extract a byte-by-byte match of the PDF present at https://web.archive.org/web/20210418161957/https://bitcoin.org/bitcoin.pdf (sha256 == b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553).

First you have to be running a Bitcoin core server locally. Suppose that your .bitcon/bitoin.conf contains:

rpcuser=asdf
rpcpassword=qwer
server=1
txindex=1

Then I run:

git clone git://github.com/jgarzik/python-bitcoinrpc.git
git -C python-bitcoinrpc checkout cdf43b41f982b4f811cd4ebfbc787ab2abf5c94a
wget https://gist.githubusercontent.com/shirriff/64f48fa09a61b56ffcf9/raw/ad1d2e041edc0fb7ef23402e64eeb92c045b5ef7/bitcoin-file-downloader.py
pip install python-bitcoinrpc==1.0
BTCRPCURL=http://asdf:[email protected]:8332 \
  PYTHONPATH="$(pwd)/python-bitcoinrpc:$PYTHONPATH" \
  python3 bitcoin-file-downloader.py \
  54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713

where tx 54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713 (block 230009) is the transaction in which the whitepaper is encoded.

Encoding used by the downloader script

The downloader is not super strict about the subtleties of the uploader. We can immediately understand its format from the source:

data = b''
for txout in tx['vout'][0:-2]:
    for op in txout['scriptPubKey']['asm'].split(' '):
        if not op.startswith('OP_') and len(op) >= 40:
            data += unhexlify(op.encode('utf8'))

length = struct.unpack('<L', data[0:4])[0]
checksum = struct.unpack('<L', data[4:8])[0]
data = data[8:8+length]

if checksum != crc32(data):
    print('Checksum mismatch; expected %d but calculated %d' % (checksum, crc32(data)),
          file=sys.stderr)
    sys.exit()

so we see that it:

• ignores the two last outputs
• ignores operands
• ignores any integer constant with less than 20 bytes (40 hex bytes)

Then, from the data above:

• the first 4 bytes are the payload length
• the next 4 bytes are a crc32 of the rest of payload data
• the payload starts at the 8th byte

Encoding used by the uploader script

The uploader is a bit more specific. Perhaps it is easiest to understand it by looking at the disassembly of a small upload, e.g. the downloader itself: https://www.blockchain.com/btc/tx/6c53cd987119ef797d5adccd76241247988a0a5ef783572a9972e7371c5fb0cc

First we have a bunch of outputs of type:

OP_1 data OP_3 OP_CHECKMULTISIG

and worth a tiny 0.00000001 BTC which contain the bulk of the data. These cannot presumably be spent, due to the OP_CHECKMULTISIG, but this cannot be proven either.

Then at then end we have two different transactions, which as we saw above the downloader ignores:

• a standard P2PKH OP_DUP OP_HASH160 <hash> OP_EQUALVERIFY OP_CHECKSIG with minimal value 0.00000001 BTC and unspent. TODO why? This corresponds to the part of the code:

  # dest output
  out_value = Decimal(sys.argv[3])
  change -= out_value
  txouts.append((out_value, OP_DUP + OP_HASH160 + pushdata(addr2bytes(sys.argv[2])) + OP_EQUALVERIFY + OP_CHECKSIG))
  

  which shows that you just pass an arbitrary target value and address from your script.

• a standard spent P2PKH change transaction with a considerable value, to recover leftovers from the input. It is sent to a new address owned by you that is created on the fly via the RPC call:

  change_addr = proxy.getnewaddress()
  txouts.append([change, OP_DUP + OP_HASH160 + pushdata(addr2bytes(change_addr)) + OP_EQUALVERIFY + OP_CHECKSIG])
  

Reading the source basically confirms what we see on the disassembly. Some other pointers follow.

At:

data = open(sys.argv[1],'rb').read()
data = struct.pack('<L', len(data)) + struct.pack('<L', crc32(data)) + data
fd = io.BytesIO(data)

we see the:

• payload length struct.pack('<L', len(data))
• crc32

being added to the start of the file.

At:

def checkmultisig_scriptPubKey_dump(fd):
    data = fd.read(65*3)
    if not data:
        return None

    r = pushint(1)

    n = 0
    while data:
        chunk = data[0:65]
        data = data[65:]

        if len(chunk) < 33:
            chunk += b'\x00'*(33-len(chunk))
        elif len(chunk) < 65:
            chunk += b'\x00'*(65-len(chunk))

        r += pushdata(chunk)
        n += 1

    r += pushint(n) + OP_CHECKMULTISIG
    return r

we see the main data encoding loop. This clarifies from the disassembly:

OP_1 data OP_3 OP_CHECKMULTISIG

• the OP_1 is fixed at pushint(1)

• the OP_3 can be less than 3, it encodes how many 65 byte chunks were encoded, with a maximum of 3 chunks present on each output due to fd.read(65*3)

  The last payload transaction can therefore have OP_1, OP_2 or OP_3 depending on granularity.

What else was the uploader script used for and was it popular?

At:

• https://github.com/cirosantilli/bitcoin-strings-with-txids/blob/master/data/satoshi_uploader
• https://github.com/cirosantilli/bitcoin-strings-with-txids#satoshi_uploader

I have indexed every single blockchain transaction that follows the above format, notably having a matching CRC32 in the first 8 bytes.

I have also managed to download and interpret every single one of the transactions as described briefly at: https://cirosantilli.com/cool-data-embedded-in-the-bitcoin-blockchain/illegal-content-of-block-229k

Basically, very soon after the uploader/downloader were uploaded on 229991, besides the whitepaper, several uploads were made intentionally containing what some countries might consider illegal data, although none of them eye-popping for my crazy standards, and as we've seen so far, the Bitcoin blockchain has survived them for the time being as of 2021.

Since the illegal content was uploaded soon after the uploader, I find it exceedingly likely that it was uploaded by the same person who created the uploader.

And after a short flurry of activity, the uploads stopped, and only one single upload was made much later at tx 89248ecadd51ada613cf8bdf46c174c57842e51de4f99f4bbd8b8b34d3cb7792 which caught my attention because it contains an ASCII art of a Buddha between two Yin Yang symbols as described at: https://cirosantilli.com/cool-data-embedded-in-the-bitcoin-blockchain/ascii-art

          69696969                         69696969
       6969    696969                   696969    6969
     969    69  6969696               6969  6969     696
    969        696969696             696969696969     696
   969        69696969696           6969696969696      696
   696      9696969696969           969696969696       969
    696     696969696969             969696969        969
     696     696  96969      _=_      9696969  69    696
       9696    969696      q(-_-)p      696969    6969
          96969696         '_) (_`         69696969
             96            /__/  \            69
             69          _(<_   / )_          96
            6969        (__\_\_|_/__)        9696

Therefore my conclusion is that the uploader and downloader scripts simply never reached considerable popularity.

They are just too over-engineered and clunkier than needed. Why would you need a CRC32 when the blockchain itself is already hashed byte-by-byte to the brim? Perhaps it was intended simply as a marker of "interesting information". But much more natural and dominant later on was to use ASCII strings, or simply magic bytes of certain file formats as markers, see e.g. my comments on more directly encoded images at:

• https://github.com/cirosantilli/bitcoin-strings-with-txids#image-indexing
• https://cirosantilli.com/cool-data-embedded-in-the-bitcoin-blockchain/images