How scalable is the BerkleyDB format (the technology used within a wallet), and what are its limitations?
What technology would replace or supplement BerkleyDB to address those scalability challenges?
In the next major release (probably 0.8), the reference client will most likely switch to LevelDB instead of BDB for blockchain data. For wallets, a different solution is needed, but we'll most likely move away from BDB for those too.
BDB has several problems - that's not to say it is bad software, but our use case simply doesn't match what it's designed for. BDB is intended to provide databases with extremely high reliability, in complex multi-proces environments, on high-end systems where database logs are backed up and where upgrades are performed manually by a database administrator. BDB doesn't even guarantee compatibility between its environment files between minor releases.
LevelDB - an open-source rewrite of Google's own database system - was designed for efficiency and consistency on commodity hardware, and outperforms BDB by an order of magnitude in some settings. Preliminary tests with LevelDB show very good results.
For wallets, LevelDB would be overkill and a burden. LevelDB uses an entire directory per database (instead of a single file), and wallets are effectively a simple key-value store that's loaded entirely in memory anyway. I assume wallet.dat will be replaced by some simple but solid hand-rolled format.