0

At a block-to-block level, what is actually encoded at a byte[] level to encode a file stream?

closed as unclear what you're asking by Anonymous, Pieter Wuille, Andrew Chow Feb 22 at 4:57

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    Store files? That’s not a thing in Bitcoin. – Anonymous Feb 8 at 3:09
  • The coinbase transaction? – MCCCS Feb 8 at 4:00
3

For details of the specific byte structure of a block, see resources like the Bitcoin Wiki article on blocks. For a transaction, see the article on transactions.

The actual data storage happens within the transactions, within their intput and/or output scripts. Theoretically, any script could be made including arbitrary data, but due to network standards, only certain types of scripts are allowed.

For an in-depth look at data storage techniques, see this paper. (Full disclosure: I am one of the authors.)

Note, when I talk about pushing data to the stack:

  • PUSHDATA(N) (where 0 <= N <= 75): Push the next N bytes onto the stack. The byte value for this is dec: N
  • PUSHDATA1: The next byte indicates the number of bytes after that to be pushed onto the stack. The byte value for this is dec: 76, hex: 0x4c
  • PUSHDATA2: The next two bytes indicate the number of bytes after that to be pushed onto the stack. The byte value for this is dec: 77, hex: 0x4d

Also, each chunk of data pushed with a PUSHDATA element cannot exceed 520 bytes, so bear in mind that any file will necessarily be broken up into chunks of 520 bytes or fewer.

Some ways of storing data include:

  • Using the coinbase (or generation) transaction (if you're a miner) to store a small amount of arbitrary data in each block you mine. This data takes the place of the inputs that would normally appear in a transaction. The coinbase transaction allows up to 100 bytes of data; however, some of this must be used for special uses, such as the block height and extra nonce.

  • In outputs with OP_RETURN. Each transaction may have one OP_RETURN output. Such an output may have a value of zero (since otherwise you're burning money that can never be recovered). The script looks like OP_RETURN <up to 80 bytes of data>. OP_RETURN is byte value dec: 106, hex: 0x6a, and the data must begin with a PUSHDATA element.

  • In outputs that pay to an address, etc. that recipient address/key/etc. may be replaced with arbitrary data. In that case, take the regular script and replace the address/key/etc. with your data. This creates an output which you cannot spend but which must be tracked in the UTXO set, so it is not advisable to do. An example of this would be to take the P2PKH script OP_DUP OP_HASH160 <Public Key Hash> OP_EQUALVERIFY OP_CHECKSIG and replace the 20-byte public key hash with 20 bytes of other data. (Note that the hash is pushed onto the stack with a PUSHDATA(20) element. As bytes, this looks like: 0x 76 a9 14 [your 20 bytes of data] 88 ac.) If you're replacing e.g., a public key, it may be detected that you're not using a real public key (if it doesn't start with 0x02, 0x03, or 0x04, for instance), but a hash should be indistinguishable from arbitrary data.
  • In inputs, using P2SH scripts. Then the data may be stored in the redeem script itself or in the inputs to the redeem script. There are multiple ways to do this, but generally the arbitrary data will be included in a PUSHDATA element, which the redeem script will somehow operate on. One notable method involves the redeem script including OP_HASH160 <hash of the data to be included> OP_EQUALVERIFY, and the inputs to that script including the data to be stored. This method ensures that the data must be included for the script to return true. Another method is pushing data to the stack, then using OP_DROP or OP_2DROP to remove it from the stack. This is efficient, but it would be possible for an attacker to tamper with the data in a transaction malleability attack.
  • You could also store data in e.g., the value field of your outputs, but this is highly inefficient.
  • Is it 80 bytes minus op_return, or is the size of that 1 or 2 bytes, bringing the effective size down to 79 or 78 bytes?? – TLDR Feb 8 at 23:56
  • The maximum size for an OP_RETURN is actually 83 bytes by default. (This is a network "standard" and actual miners can choose for themselves what they will choose to mine.) This gives you effectively 80 bytes of data at most: 1 byte for OP_RETURN, 1 byte for OP_PUSHDATA1, 1 byte to indicate 80 bytes of data to be pushed, then 80 bytes of data. This is defined here. – Vecna Feb 9 at 2:13
2

You can store arbitrary bytes in a Bitcoin transaction which are not validated but can be confirmed in the blockchain given sufficient fees.

op_return is a Bitcoin script op_code used in output scripts and which, when executed on the script interpreter will return the script run as invalid, thereby rendering the output "unspendable".

Now, because the script-run is aborted, subsequent data in the same output script are never evaluated and can be arbitrary. A valid, confirmed Bitcoin transaction can contain unspendable outputs.

In theory an arbitrary size of data can be appended to op_return as long as the transaction is still valid (consensus), but node propagation policy usually limits this to 80-bytes. A miner could confirm larger data sizes of course.

Not the answer you're looking for? Browse other questions tagged or ask your own question.