At a block-to-block level, what is actually encoded at a
byte level to encode a file stream?
For details of the specific byte structure of a block, see resources like the Bitcoin Wiki article on blocks. For a transaction, see the article on transactions.
The actual data storage happens within the transactions, within their intput and/or output scripts. Theoretically, any script could be made including arbitrary data, but due to network standards, only certain types of scripts are allowed.
For an in-depth look at data storage techniques, see this paper. (Full disclosure: I am one of the authors.)
Note, when I talk about pushing data to the stack:
0 <= N <= 75): Push the next N bytes onto the stack. The byte value for this is dec:
PUSHDATA1: The next byte indicates the number of bytes after that to be pushed onto the stack. The byte value for this is dec:
PUSHDATA2: The next two bytes indicate the number of bytes after that to be pushed onto the stack. The byte value for this is dec:
Also, each chunk of data pushed with a PUSHDATA element cannot exceed 520 bytes, so bear in mind that any file will necessarily be broken up into chunks of 520 bytes or fewer.
Some ways of storing data include:
Using the coinbase (or generation) transaction (if you're a miner) to store a small amount of arbitrary data in each block you mine. This data takes the place of the inputs that would normally appear in a transaction. The coinbase transaction allows up to 100 bytes of data; however, some of this must be used for special uses, such as the block height and extra nonce.
In outputs with
OP_RETURN. Each transaction may have one
OP_RETURNoutput. Such an output may have a value of zero (since otherwise you're burning money that can never be recovered). The script looks like
OP_RETURN <up to 80 bytes of data>.
OP_RETURNis byte value dec:
0x6a, and the data must begin with a PUSHDATA element.
- In outputs that pay to an address, etc. that recipient address/key/etc. may be replaced with arbitrary data. In that case, take the regular script and replace the address/key/etc. with your data. This creates an output which you cannot spend but which must be tracked in the UTXO set, so it is not advisable to do. An example of this would be to take the P2PKH script
OP_DUP OP_HASH160 <Public Key Hash> OP_EQUALVERIFY OP_CHECKSIGand replace the 20-byte public key hash with 20 bytes of other data. (Note that the hash is pushed onto the stack with a
PUSHDATA(20)element. As bytes, this looks like:
0x 76 a9 14 [your 20 bytes of data] 88 ac.) If you're replacing e.g., a public key, it may be detected that you're not using a real public key (if it doesn't start with
0x04, for instance), but a hash should be indistinguishable from arbitrary data.
- In inputs, using P2SH scripts. Then the data may be stored in the redeem script itself or in the inputs to the redeem script. There are multiple ways to do this, but generally the arbitrary data will be included in a PUSHDATA element, which the redeem script will somehow operate on. One notable method involves the redeem script including
OP_HASH160 <hash of the data to be included> OP_EQUALVERIFY, and the inputs to that script including the data to be stored. This method ensures that the data must be included for the script to return true. Another method is pushing data to the stack, then using
OP_2DROPto remove it from the stack. This is efficient, but it would be possible for an attacker to tamper with the data in a transaction malleability attack.
- You could also store data in e.g., the value field of your outputs, but this is highly inefficient.
Is it 80 bytes minus
op_return, or is the size of that 1 or 2 bytes, bringing the effective size down to 79 or 78 bytes??– TLDRFeb 8, 2019 at 23:56
The maximum size for an OP_RETURN is actually 83 bytes by default. (This is a network "standard" and actual miners can choose for themselves what they will choose to mine.) This gives you effectively 80 bytes of data at most: 1 byte for
OP_RETURN, 1 byte for
OP_PUSHDATA1, 1 byte to indicate 80 bytes of data to be pushed, then 80 bytes of data. This is defined here.– VecnaFeb 9, 2019 at 2:13
You can store arbitrary bytes in a Bitcoin transaction which are not validated but can be confirmed in the blockchain given sufficient fees.
op_return is a Bitcoin script op_code used in output scripts and which, when executed on the script interpreter will return the script run as invalid, thereby rendering the output "unspendable".
Now, because the script-run is aborted, subsequent data in the same output script are never evaluated and can be arbitrary. A valid, confirmed Bitcoin transaction can contain unspendable outputs.
In theory an arbitrary size of data can be appended to
op_return as long as the transaction is still valid (consensus), but node propagation policy usually limits this to 80-bytes. A miner could confirm larger data sizes of course.