1

I'm trying to build a blockchain-based application and I have a problem with understanding some crucial issues. What exactly is a block? I mean, I've seen application where blocks were just String structures and a blockchain was an ArrayList of these String structures. But my question is, does it work like that in real blockchains? Are blockchains just ArrayLists or maybe they are stored in some files that are distributed among a P2P network?

Improve this question
1
  • You can parse the raw database which is stored in blk*****.dat files with blockchain parser and see the structure of blockchain data, so it would be the most fastest way to learn how it works.
    – Denis Leonov
    Commented Sep 1, 2020 at 20:11

4 Answers 4

Reset to default
2

How is data in a blockchain stored

So far as I know, the Bitcoin specifications do not mandate any particular structures for storing the Bitcoin transaction-journal (blockchain) in persistent storage (e.g. Hard disk).

Any Bitcoin node is free to store the transaction-journal in whatever form is convenient. For example it might be decomposed as a set of tables in third-normal form in a relational database in "uncooked" storage outside any filesystem. The node might include in the stored data a complex set of indices for fast retrieval of certain types of data (e.g. Bitcoin-addresses).

Bitcoin Core stores the transaction-journal in groups of consecutive blocks in a series of large files whose format I believe closely matches the form used in the Bitcoin network protocols. But there is no prohibition on other software products doing something quite different.

Are blockchains just ArrayLists

That is just one way of representing the data. Note that the ArrayList is a feature of specific programming languages. It may not exist in some languages and quite different data structures may be more appropriate in that case.

The data itself needs to keep track of blocks and those blocks contain data that references data within other blocks. Hence the name. But there are many different ways of implementing these ideas which could all work in a Bitcoin node.

maybe they are stored in some files that are distributed among a P2P network?

Normally, in my field of work, the term "distributed" implies that not all nodes hold all the data. It implies that there is some planned and coordinated division of data among nodes so that specific nodes are designated to be responsible for specific subsets. This isn't really a fundamental design feature of Bitcoin.

I believe that initially Bitcoin used a model that I would more strongly associate with the word "replicated" than "distributed". Initially all nodes each had a full and complete copy of the entire data-set. There was no system to coordinate the intentional distribution of subsets of data among cooperating storage nodes.

Currently nodes, even full nodes, can discard blocks that are no longer of interest to them. This means that many nodes do not contain a complete set of data. Importantly however, any distribution of the data between nodes is essentially unplanned and uncoordinated.

You could argue that there is distribution of data between SPV clients and the servers they rely on for transaction-journal data. However I'm moderately sure this isn't really the sort of distribution of data that database engineers generally mean when they use the term "distributed".

See also descriptions of Bitcoin Core's storage format in these answers:

Improve this answer
2
  • Thanks, you helped me a lot. But I still have some doubts in my specific situation. I have a website with several forms, that every single form is treated like a transaction. When all forms (transactions) are filled in, a new block containing data from the forms is created. Can you recommend me a good way to store this data?
    – mmaciek
    Commented Sep 2, 2020 at 15:26
  • @mmaciek: Firstly, this isn't the place for that. See area51.stackexchange.com to find places that discuss this kind of software design question. Secondly it depends on application-architecture, platform, toolset, throughput, volume, response-time requirements, skillset, resources, budget and many other things.
    – RedGrittyBrick
    Commented Sep 2, 2020 at 16:53
2

At its core, a blockchain is a linked list with a twist.

In a normal linked list, each item points to the previous item (these are the "links" which allow us to traverse back through the list.) In a computer, these links are references to memory locations, but in a blockchain, they are references to the hashes of preceding blocks.

Most importantly, the block of data that is hashed to produce a blockhash includes within it the blockhash of the previous block. You can't change the previous blockhash inside of the current block without changing the hash of the current block... and this means you obviously can't replace a block in the chain without visibly breaking the chain of blockhashes that come after.

This is the critical difference between a linked list (or ArrayList) and a blockchain:

  • in a linked list, it's easy to replace any element by modifying two elements, it and the element after it, and the change is not visible in later parts of the chain
  • in a blockchain, to replace a single element, you must replace all elements after it, corrupting the whole chain in an obvious visible way

Now, in most blockchain implementation, including bitcoin, what I said above pertains only to block headers, not to whole data blocks. For efficiency, the block headers comprise the cryptographically chained list, and the data is stored in separate chunks. So how are they related?

Each block header also includes within it a hash of one data block. This way, if you're looking at block 2045 of data and want to authenticate it, you merely have to hash it and see if that hash matches the hash in block header 2045. Nobody can give you a false block of data without corrupting the hash and, as noted above, they can't change the hash in the header without corrupting the whole chain of headers.

That's the typically used blockchain structure:

  1. A chain of block headers, each of which includes the hash of the preceding block and also includes the hash of one data block.
  2. A matching collection of data blocks, each of which has a hash that is included in one of the block headers.

(You could dispense with the block header/data block separation, but it can make lots of things more efficient to separate them.)

If you have the first block (sometimes called a "genesis block") and the validation rules (including what hashes are used and how they made), you can validate everything else yourself, without having to trust some other system to give you honest data.

If designing your own blockchain, you obviously have a lot of leeway as to what goes into the data blocks, how they are stored and distributed, etc. The choice of networking and storage protocols are not what make it a blockchain.

So, back to your original question: what is a block? It's just a chunk of data.

How is it stored? However you like, but for Bitcoin, they are just chunks of data disseminated via a P2P network of nodes, and many nodes will store those chunks in a database on disk. In fact, there are several bitcoin implementations which store them in different ways! But they can all agree on what's valid, because of the blockchain.

Improve this answer
0

Are blockchains just ArrayLists or maybe they are stored in some files that are distributed among p2p network.

Yes. The blockchain is stored in files that are shared in every full node of the network. Each full node has a full copy of the blockchain (files). Each file contains a certain amount of blocks.

Each block contains a header and a list of transactions. In the header, you can find information about that specific block, as well as which block it is linked to. Transactions are the atomic unit of the blockchain, and they contain information such as what Bitcoins are being spent, how they are being spent, and to where they are being sent.

If you really want to go into details, just go to the wiki page of Bitcoin:
For block headers:
https://en.bitcoin.it/wiki/Protocol_documentation#Block_Headers
for transactions:
https://en.bitcoin.it/wiki/Transaction

Also, check out this guy's work. He explains very well the Bitcoin blockchain, and how to parse the blockchain into a database to create a block explorer.
https://learnmeabitcoin.com/technical/

Improve this answer
6
  • 1
    Full nodes can be pruned, in which case only recent blocks are stored (old ones are still downloaded, verified, and processed, but thrown away after some time). The "full" aspect refers to validation.
    – Pieter Wuille
    Commented Sep 1, 2020 at 0:06
  • Thanks man. You always give me great inputs and updates. I still have a lot to learn.
    – Oscar Serna
    Commented Sep 1, 2020 at 0:36
  • Ok I almost got it, but one more thing. Every block is an independant file and all these files have some kind of reference to the previous file (block), or there is one blockchain file which is updated every time when a new block is added?
    – mmaciek
    Commented Sep 1, 2020 at 8:08
  • A single file contains many blocks because blocks have a size limit of 1 MB, and the files have a limit of 128MiB. You can find these files in a folder called 'blocks', and they are formatted like this: blkxxxxx.dat (xxxxx is the 5 digit consecutive index of the file). Every block has an id which is calculated through the information in the block, and part of that information is the id of the previous block. In this way, the block knows after what block it comes in the blockchain.
    – Oscar Serna
    Commented Sep 1, 2020 at 17:23
  • I don't know exactly how the latest blocks interact with the blk.dat file being created right at the moment, but I guess it works similar to how rotating-log-files work. But again, I don't really know the exact mechanics of it.
    – Oscar Serna
    Commented Sep 1, 2020 at 17:25
0

If you are an engineer you can have a look at this code here: https://github.com/mrqc/bitcoin-blk-file-reader

It shows how to parse the blk files.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.