What is in the Bitcoin Core LevelDB dbcache? Is it full records or metadata?
This question was asked on IRC by Anonymous.
The Bitcoin Core documentation describes dbcache as the cache for the UTXO database.
Wikipedia describes a cache as a:
software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere.
Greg Maxwell added on IRC:
There are basically two separate uses: the block index which is essentially metadata and the UTXO database.
The dbcache is not really a cache, it's a write buffer and it prevents needing to sync the disk or make random writes. As a cache it doesn't do much and isn't needed for that. It does also act as a cache, but if you nullify that benefit it's only approximately a 10% slowdown even with a huge dbcache.
With a fast SSD (e.g. NVMe) I think the difference between a 400MB cache and a 5GB one is "only" a halving of IBD time (when syncing from LAN peers).
To prevent corruption the database updates must involve synchronizing writes.
We changed how dbcache flushing worked with the explicit intention of making background flushing possible so that it could be concurrently flushing and processing blocks constantly rather than inserting these bubbles where it waits on the disk. Even though the consistency requirements make that possible now (the UTXO database doesn't have to be consistent except there is a record that states that all blocks before it have absolutely positively been applied to the database), actually making the change is still extremely complicated.
Every time you process a block, once the cache fills you could flush out to disk the remaining dirty entries for the oldest remaining blocks until it's back under the limit again.. So essentially flushing every block in the background. Then every once in a while, you do a disk sync and update the record that says where the completely consistent point is, without having to flush anything additional out. So almost no latency spikes at runtime.
This would be good for mining but doing that requires a bunch of machinery to efficiently track things and probably would make sense to replace the dbcache's map with an open hash table of some kind at the same time to cut the malloc traffic down by 10x. I expect there is a factor of 2 IBD speedup waiting from these changes. I'm not sure if it'll ever happen though, it's a big complicated task and any mistake is a consensus bug.
You can even do neat stuff like resolve collisions in the open hash table by displacing the oldest UTXO and once you've gone "far enough" flush. The table could run extremely close to completely full at all times.
I'm personally fond of cuckoo tables. Where every item has a small number of possible locations like two random buckets which each hold 4 items and if those locations are full you pick one, insert the item there, bump what was in that slot into one of its alternative locations.
If the table has a little slack (no more than ~95% full for 2 buckets of 4 items for example) then it's always successful at finding an empty slot after a few kicks. Lookup is extremely fast since you only need to do two random memory accesses. There are a bunch of different designs though most are best when you can keep the table <50% full but the cuckoo design is easy to take up to arbritarily high fullness.
The STL sets that Bitcoin uses are a hash table, but collisions are resolved by having each entry in the hash table be a linked list, so every access involves several pointer chases and every insertion takes a malloc.