2

I am using bitcoinj to listen for payments. I need a way to maintain the entire UTXO set in a local database.

  1. Can bitcoinj be used to download every block (I think it downloads only recent blocks and headers for older blocks)?

  2. I don't need to use bitcoinj or even Java for creating or maintaining this database if a better solution exists. Is there anything like this?

Ideally what I want is a wallet that downloads only the UTXO set without downloading the entire blockchain (due to bandwidth limitations) and maintains it from then onward. This needs to be the entire set, not just for the addresses I own. Additionally I need an API that allows for following queries:

  1. Given a UTXO, check if it is present in the set.
  2. Given an address, get me all the UTXOs for it.

I think that it is currently not possible to validate a given UTXO set without downloading the entire blockchain, so some solution in this regard would also be appreciated.

5 Answers 5

1

Yes, by default bitcoinj downloads full blocks, from the Genesis. Only if you connect a wallet it uses optimizations like checkpoints and filters.

Just downloading the UTXO set (in a trust free way) isn't possible, because the Bitcoin protocol isn't designed for that. You can get UTXOs from block explorers and Electrum servers, but it will be without crypto proof.

1
  • How much time does it take currently with a i7 k7700 CPU and decent bandwidth?
    – Jus12
    May 29, 2017 at 20:56
1

If you put aside the issues of trust, validation of the data, keeping it up to date and so on. If you just want a one-off snapshot for some academic purpose, you could ask a friend who uses Bitcoin Core to give you a copy of their chainstate folder - but you'd have to extract data from the LevelDB data structure.

Other wallets such as Electrum might also have some persistent data storage that mainly consists of UTXO data.

0

As Andreas wrote, Just downloading the UTXO set (in a trust free way) isn't possible. I would like it to be possible, and so I've thought about how to do it...

Each block identifies transactions from previous blocks that were in the UTXO set and are no longer in it and also fully specifies all the transactions that have been added to the UTXO set by the current block. This means that IF you already have a full and verified UTXO set and someone wants to know if a UTXO set as of N blocks ago is valid, you can provide an answer. Whether or not they trust you is a different issue. If your set is valid, then you can undo the additions and subtractions of transactions as indicated by the N blocks, calculate a hash, and then send that hash to the other person. If they get the same hash on their set, it's safe for them to assume that you used the same UTXO Set to get the hash that they used (ie., they match).

There may be software that can do this already. I'd like that. But there's no proof that either of you has the correct UTXO set. I'd like to see that fixed for everyone, but it requires a little bootstrapping. I have a plan for it:

  1. Create software that will render a SHA256 Hash into something that is memorable and unique. This is a bit more complicated than it sounds. Whatever the rendering produces (music, art, animation, a combination, ...) must have (at least) 256 distinct binary properties that will be easily remembered. Anyone could use this software to see if the hash of the UTXO Set at a recent block height produces a memorable rendering.
  2. Make it a thing that sometimes a memorable rendering is produced. That can serve as a global checkpoint for the blockchain. If enough people (judgement call) know about the memorable rendering, then new installations of full nodes could get (from anyone who has it) the UTXO set at that height, compute the hash, render it, and compare the result to the one everyone remembers. I imagine that would be readily available from around the Internet, but it's safer to talk to someone in real life to avoid the danger of Sybil attacks.
  3. Update full node software to provide the UTXO Set as of a block the requesting node specifies (to mak.e the previous step possible for fresh installations)
  4. Update the full node software to allow the user to specify a block height known to have produced a UTXO Set with a memorable hash, ask for the UTXO set as of that block from peers, create the rendering from the hash of the answer, and display it to the user.
  5. Allow the user to indicate that the rendering matches and if the user so indicates, then skip downloading the blockchain (except for any parts that aren't included in the canonical UTXO set - if any) to save bandwidth and disk space, and just start with the UTXO set that came in as the answer.

For the same reason that mined bitcoin that is less than 100 blocks deep cannot be spent (ie., a reorg of 100 blocks is unlikely - possibly not even supported?), this strategy allows us to proceed with a minimum of 100 blocks of history instead of the entire blockchain.

This places the responsibility of ensuring the integrity of the starting point UTXO set on the human being using the full node software. It would have to be experimental for a while, until it proves that blocks zero through X are no longer necessary (because the UTXO set at block height X was memorable enough to be recognizable by mere humans using this new feature), thus reducing the blockchain requirement for an always-increasing amount of storage to an amount of storage that will presumably level off.

Part of the appeal to me of this idea is that the random SHA256 hash of a UTXO set can be rendered into something memorable, and the chances of that happening are low but not zero, just like mining itself. Different rendering algorithms could compete. I just thought of one...

The 256 bits of the hash plus the 32 bits of the block height could be used to indicate which cells in a 16x18 game of life are on. Let the game of life run. Watch it. Remember how it stabilizes. Some games will eventually clear the board. That's memorable. How it clears the board would also be memorable.

I started looking for a canonical UTXO set (ie., one whose structure is standardized, so that the same set of UTXOs would have the same representation as data). Converting the SHA256 hash plus 32-bits of the block height into a Game of Life to see how it plays out can be pretty easily done with Golly or any other software that implements John Conway's "Game of Life."

So I have the same quest as OP, but Andreas has a point. You have to run a full node and create it yourself. However, once you've done that, you could possibly make it available to others (me!). He also points out that there's no way to prove it's correct except by running a full node and repeating all the work of updating it for every transaction. Proving the result is correct without all that work is the goal of the idea I present in this answer.

0

Take at a look at this program. It will dump the entire current UTXO database to a CSV file that you can crossreference. https://github.com/in3rsha/bitcoin-utxo-dump

0

@ Dave Scotese: I agree with your idea of having a generally agreed UTXO set (Not sure why u suggested running the resulting checkpoints hash into a game of life though). Anyway, I think the problem is not that it can't be done rather it comes down to consensus (very similar to the problem solved by blockchain itself). Suppose you did honestly used your algorithm and provided the world with a memorable hash that defines the UTXO set at some block height checkpoint which you place in some repo for download. Now if Mr. A downloads it and checks using his own database and agrees with you. You don't expect Mr. B to agree with you simply because Mr. A agrees. You see, whoever needs to use this UTXO set needs to verify it for himself. It still comes back to that Byzantine Generals problem which blockchain solved. I honestly think for it to work, it has to run on its own separate blockchain so that UTXO set miners can mine it for some kind of financial gain.

And yes I completely agree with you that having such a UTXO set will solve the problem of having to download the entire blockchain before you can validate transactions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.