In Bitcoin Core, there are two settings,
-maxmempool, defaulting to 450 and 300, respectively.
The mempool can only occupy as much as is configured through
The UTXO set cache can occupy as much as is configured through
-dbcache plus the amount of configured mempool space that is currently unused. The memory usage is however much higher than the equivalent representation on disk - loading the entire UTXO set would require 6-7 GiB at the time of speaking.
The in-memory representations are always larger. On disk they're just packed sequences of bytes. In memory, there are indexes and pointers, and memory allocation overhead - all to speed up access. The blowup factor for UTXO data is larger than for memory.
Since 0.14, the mempool is also saved to disk at shutdown (but not continuously).
So far, there is no prioritization done on the UTXO cache. We just keep all accessed entries around, and when it grows too large, all the modified entries are written to disk. One (very) significant optimization is that when a UTXO is created and then spent, without being written to disk in between, we just remove it from memory.