17

In 2013 a new bitcoin core was released and one of the supposed improvement was migrating from Berkeley DB to LevelDB. According to the release notes at bitcoin.org:

LevelDB, a fast, open-source, non-relational database from Google, is now used to store transaction and block indices. LevelDB works much better on machines with slow I/O and is faster in general.

Similar statements were done by bitcoin core developers:

LevelDB - an open-source rewrite of Google's own database system - was designed for efficiency and consistency on commodity hardware, and outperforms BDB by an order of magnitude in some settings. Preliminary tests with LevelDB show very good results.

However, nowadays there are several comments through the internet critizing LevelDB. According to LevelDB page at Wikipedia:

LevelDB is widely noted for being unreliable and databases it manages are prone to corruption. Academic studies of past versions of LevelDB have found that, under some file systems, the data stored in those versions of LevelDB might become inconsistent after a system crash or power failure. LevelDB corruption is so commonplace that corruption detection has to be built in to applications that use it.

In summary, the complaints about LevelDB are:

  • unreliable and databases are prone to corruption
  • several bugs due to fundamental flaws
  • poor code quality
  • no longer actively maintained (i.e. dead project)

Question 1
Are these complaints about LevelDB correct?

Question 2
What is the final balance of migrating Bitcoin Core from Berkeley DB to LevelDB? Is LevelDB working as expected? This migration was the right choice?

  • 3
    IIRC, Bitcoin uses BerkleyDB 4.8 for wallet files, and LevelDB for blockchain indexes. Informally, I can tell you that blockchain index corruption during crashes is very common, and I usually tell affected users to redownload the blockchain. – Nick ODell Feb 16 '17 at 20:02
  • @NickODell Do you know whether such crashes you`ve mentioned also happened before the migration to LevelDB? – Mark Messa Feb 16 '17 at 20:59
  • Sorry, I don't know. – Nick ODell Feb 16 '17 at 21:23
15

As someone who was involved in doing that migration at the time, I believe it was the right decision. LevelDB is far from perfect, but I wouldn't know what else to use.

In particular:

  • BDB is much slower for our usage (large atomic batch writes, small random reads).
  • There were reports of database corruption as well with BDB, at a time when it was used far less intensively than LevelDB is now.
  • BDB is very painful to upgrade. My impression is that it was designed for setups where a database upgrade only happened with professional supervision. In particular, the write log files created for durability were sometimes not readable by later versions. This is the reason why Bitcoin Core releases have for years stuck with BDB 4.8 for the wallet.
  • BDB has many resource limits that need configuration, where inappropriately chosen values may cause network-wide failures (read up on the March 2013 incident, specifically). The official documentation on this states Reviewing the Lock subsystem statistics is the best way to determine this value. I believe that's not acceptable for our use case.

While database corruptions are reported relatively frequently these days, my belief is that it's mostly hardware failures or driver issues. Bitcoin Core tends to stress disks, memory and CPUs far more than most software, which makes otherwise invisible issues pop up.

  • > "[...] database corruptions are reported relatively frequently these days [...]" Is there an active LevelDB community to provide some feedback/support about such issue? – Mark Messa Feb 17 '17 at 6:19
  • I personally never see corruptions, but I run on relatively new hardware, and use Linux. But I probably resync the chain on average once I week for development purposes. That doesn't mean I don't believe such reports are real, but at least in some cases they turn out to be due to flaky hardware or storage systems - in which case it's hard to blame the database software. – Pieter Wuille Feb 17 '17 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.