49
Bitcoind since 0.8 maintains two databases, the block index (in $DATADIR/blocks/index) and the chainstate (in $DATADIR/chainstate). The block index maintains information for every block, and where it is stored on disk. The chain state maintains information about the resulting state of validation as a result of the currently best known chain.
Inside the block ...
14
As someone who was involved in doing that migration at the time, I believe it was the right decision. LevelDB is far from perfect, but I wouldn't know what else to use.
In particular:
BDB is much slower for our usage (large atomic batch writes, small random reads).
There were reports of database corruption as well with BDB, at a time when it was used far ...
12
In the specific example you asked, it does so by querying the LevelDB located in ./bitcoin/chainstate (there is where the UTXO set is stored).
UTXOs are identified by its txid (Little Endian representation) leaded by a c.
Example
Transaction
246c5a81b6ad0dfc0dbc0b2ff5bde65ee1913f75a47d409b8ff8074a27ec1000
is identified in the LevelDB by:
...
7
Redis and LevelDB solve very different problems. We tried using SQLite and its performance was abysmal.
Bitcoin Core needs a database to store the set of unspent transaction outputs (UTXOs). This means we need fast simple reads, and fast batches of random updates.
We don't need a server/client architecture, as we can't have multiple applications accessing ...
7
The fork serves two purposes:
Local modifications that are hard to bring upstream:
Windows support (which is partially based on the existing Windows port, but needed changes for building in MinGW)
Removal of compression support, as it doesn't help, and complicates the build.
Strict control over changes. Given the previous experience with the BDB to ...
7
There is no code for that, and it would be either very slow or nearly useless.
Bitcoin Core assumes it has exclusive access to the -database, and caches data very aggresively. It may take hours before actual data from the chain is on disk.
The data structures are very compact and specifically designed for efficient use in Bitcoin Core. This means only ...
6
The best way to write a parser of your own for the blockchain is to find the source code of one that already exists and from that deduce the precise underlying data structure and how to parse and interpret it. e.g. here in C++ and here in C# - and there are many others.
The data in the blockchain is stored in a custom binary format that is a little tricky ...
4
OK I know I shouldn't really answer my own question but... in the absence of a response to this question, I did a bit of hunting.
Github provided the answer in a file found in the bitcoin-leveldb repository.
The path to a text file containing the information leveldb->doc->table_format.txt
In short, there is no easily describable table like ...
4
This is the Bitcoin Core v0.15.1 source code snippet that defines the serialization of CDiskBlockIndex. The various fields are:
A varint version number of the client who wrote the record (for future extensibility). This is the 87c628 you're seeing.
A varint block height.
A varint status with various flags.
A varint determining the number of transactions in ...
4
The wallet is to a large extent separate from the rest of the node software. One of the reasons for this is because of the (slow) progress towards a working SPV mode. This means the wallet cannot depend on access to the UTXO set, as it won't be available in non-validating nodes.
However, it isn't needed. The wallet maintains an index of all input prevouts ...
4
Sometimes LevelDB has bugs and those bugs sometimes are not fixed in a timely manner in the upstream LevelDB project. These bugs can be problematic for Bitcoin Core so a fork of LevelDB was created to handle these bugs specifically.
Additionally, to ensure security, we want to use a specific version of LevelDB at a specific commit. By having our own fork of ...
3
There’s no way of easily disabling obfuscation, but you could build a version that sets the keys to zero, resulting in no change when the XOR is used. Generally it has no impact on anything, so it’s left enabled even on non-windows systems where it’s generally not useful.
3
The CVarInt format is implemented in serialize.h
As the comment is extensive, I'll just quote it here:
Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make sure the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte ...
3
This happens when your dependencies get twisted up, especially when bcoin has an update where leveldown gets upgraded. You should be good with a npm rebuild leveldown to fix the dependency tree, but if that doesn't work rm -rf node_modules && npm i
3
There are two reasons for having the fork in the first place:
Upstream LevelDB does not support Windows, and the branches that are available elsewhere don't support MinGW (which Bitcoin Core uses for its Windows release builds).
After the scare the project experienced with the BDB locks bug, which resulted in a fork between different versions (see BIP50), ...
3
Yes, Bitcoin Core does do some compression of standard output scripts in order to store the minimal amount of data needed.
Anyway, would I be correct in assuming that you could only get an address from script types 0 and 1 (by base58 encoding the script data)?
Yes
In other words, the chainstate leveldb does not include any witness data to allow you ...
2
The best way to access transaction database for analysis is probably through the blockchain.info API. You can also use the Bitcoin Block Explorer; in fact, this is what a study of the Bitcoin network did.
2
Bitpay's bitcore-node project uses the same datadirectory as Bitcoin Core. You can see this in their sample configuration in their README:
var configuration = {
datadir: '~/.bitcoin',
network: 'testnet'
};
As such, the you can see: What are the keys used in the blockchain levelDB (ie what are the key:value pairs)? for information on what is in this ...
2
(note: I'll be using a different bit numbering from the comment which goes from 0..B instead of 1..2^B, as I find that easier to read myself)
The binary representation of 0x9 separated in functional groups is
bit# ...6543 21 0
---------
0001 00 1
Bit 0 is set because this is a coinbase
Bit 2 and 1 are 0, because vout[0] and vout[1] are ...
2
If you're running master, there was a recent migration that would cause this.
you can run this to fix it:
node migrate/chaindb2to3.js /path/to/chaindb.ldb
That will take a very long time, but if you don't mind pruning your DB at the same time, you can run this instead and it will be much faster:
node migrate/chaindb2to3.js /path/to/chaindb.ldb --prune
...
2
The LevelDB iterator iterates through the keys in lexicographic order. From the docs:
The preceding examples used the default ordering function for key, which orders bytes lexicographically. You can however supply a custom comparator when opening a database. For example, suppose each database key consists of two numbers and we should sort by the first ...
2
Plyvel has Snappy compression enabled by default. Have you tried disabling it when opening the database?
# Open the LevelDB
db = plyvel.DB(".bitcoin/chainstate", compression=None)
With this code my bitcoind does not seem to complain after having opened the database with python.
2
To add on to eponymous's answer, to disable this, comment out this section of code:
https://github.com/bitcoin/bitcoin/blob/50fae68d416b4b8ec4ca192923dfd5ae9ea42773/src/dbwrapper.cpp#L129-L139
if (!key_exists && obfuscate && IsEmpty()) {
// Initialize non-degenerate obfuscation if it won't upset
// existing, non-obfuscated data.
...
2
No, it's vandalism. Some virus signatures were uploaded to the blockchain as a prank: https://www.reddit.com/r/Bitcoin/comments/25otbt/someone_put_a_virus_signature_in_the_bitcoin/
It seems those are some of that cases.
2
is it a great idea to use IPFS for blockchain communication?
Bitcoin nodes form a P2P network, using TCP to communicate bitcoin-specific messages between one another. By using a self-defined protocol, the Bitcoin network's communications can be optimized and made robust against attacks, while not having rely on any external dependencies to do so.
Since ...
1
After investigating I found the x in txout_decompress(x) was not an integer which was causing the difference in answer so during division I typecast it to int and it worked. So the corrected code:
def amount_decompress(x: int):
# x = 0 OR x = 1+10*(9*n + d - 1) + e OR x = 1+10*(n - 1) + 9
if x == 0:
return 0
x -=1
# x = 10*(9*n + ...
1
Database corruption means the state of your node is corrupted, not just data it downloaded. Resuming after state corruption would be like resuming your life after having a lobotomy.
During synchronization your node operates at hundreds of times its normal load: Consider, Bitcoin has existed for over 3000 days yet sync will complete in a day. This tends to ...
1
On the subject of why LevelDB is used, core developer Greg Maxwell stated the following to the ref:
I think people are falling into a trap of thinking "It's a <database>,
I know a <black box> for that!"; but the application and needs are
very specialized here. . . It just so happens that on the back of the
very bitcoin specific ...
1
There is a lot of information available, just not consolidated. After searching for sometime, I found the best answer from this link.
The format of each file is:
4 bytes: Magic bytes
4 bytes: Encodes size N of next upcoming block.
N bytes: Block encoded in standard format (with header)
--
above repeats
Quotes from the original link:
4 | 4 | 80 | TxData | ...
1
The prefix for transactions was changed from b'c' to b'C'.
Is there any specific reason for changing the prefix?
The entry changed to a new data structure. However in order to be able to upgrade previous database versions to the new one, we need to be able to identify which entry corresponds to the old structure and which corresponds to the new structure. ...
Only top voted, non community-wiki answers of a minimum length are eligible
