Are there coins that demand system memory rather than computing power? Which coin brings me the most mining profit if I have a lot of system DDR3 RAM (say 64 GB or more) in my personal computer?
There are memory intensive mining algorithms, but usually the "memory" being discussed is not of the DDR3 variety. That 64 GB of DDR3 RAM you have is an excellent resource for many computing tasks (and enough to make me jealous), but is also dramatically slower than the L1 and L2 cache physically on the chip with your CPU. This is the memory we're talking about.
Memory intensive mining algorithms came about in reaction to GPU (and later FPGA and ASIC) acceleration. Because GPUs have so many cores, each core has access to only a miniscule amount of RAM, often an amount measured in kilobytes. Because Bitcoin's SHA256 mining algorithm has a tiny memory footprint, it runs quite nicely on such hardware. Some say this is a problem, others disagree.
CPUs, on the other hand, have relatively few cores sharing access to, on average, 64 to 128 kB of L1 cache and 2 to 4 MB of L2 cache. By virtue of being on the same die as the CPU cores, this RAM is extraordinarily fast and has amazingly low latency. This, it turns out, is an exploitable difference that memory intensive algorithms take advantage of.
An ideal implementation of one of these "memory intensive" algorithms (for mining) will be tuned such that the RAM footprint required is higher than the few kilobytes available to GPU cores, but lower than the few megabytes typically available in L1/L2. In this way, the mining algorithm can be run at high efficiency on standard CPUs, thus maximizing the hashrate available to secure the network, but is still resistant to GPU/FPGA/ASIC acceleration. The CryptoNote algorithm used by Monero is a good example.
Sadly, few altcoins take such care with their implementation. Most either use so little memory that they are freely accelerated with GPUs or use so much that all mining processes eat into your good old DDR3 and lose efficiency. In short, there is almost certainly a coin out there that can make use of such an obscene amount of system memory, but that's almost certainly a sign that they did it wrong since the goal of such an algorithm should be to maximize hashrate on the target device (CPUs) for the sake of network security while preventing acceleration from non-target devices (GPUs, FPGAs, ASICs).
In any event, all proof-of-work systems currently in use are mathematical in nature. This makes the memory requirements somewhat secondary in nature - you will still require a lot of computing power to mine and it is this computing power that will largely determine your results. In a few cases the extra RAM may help, but your CPU will still be doing the heavy lifting.
Special note: In certain non-mining implementations of such memory intensive algorithms (like the scrypt implementation used in BIP38 encrypted keys) the poor efficiency caused by exceeding available on-chip cache can actually be a positive effect and is often used intentionally. By forcing the process to use slower memory we can throttle the number of decryption attempts an attacker is able to make in a given timeframe, which dramatically reduces the effectiveness of brute force attacks. This is why decrypting your BIP38 key on a phone often takes 30+ seconds - it's a feature, not a bug.
With such fast main memory, I'm sure your machine would be pretty good for mining MemoryCoin whose proof-of-work algorithm, Momentum, requires a lot of RAM. Copied verbatim from its tech page:
Each hash requires 1GB of Memory to perform and encrypts 50GB of data. By contrast, the verification only requires 128K and less than one 10th of a second
Be warned, though, it also requires a fast CPU (from same page: "it has a large AES component – chips with AES-NI instruction sets will fare much better"). Plus, I don't know of many exchanges that deal with it, nor how much it's worth - so can't say for sure if mining such coin would be the most profitable for you or simply the most efficient.
I'm sure your machine would have a slight advantage at mining other CPU-only coins, have you tried Protoshares or Quarkcoin?
Cuckoo Cycle is quite memory intensive, with 67% of runtime spent on main memory access. See https://github.com/tromp/cuckoo for an implementation and whitepaper.
Try to search memory-intensive in whitepapers. Check these results: https://whitepapersindex.com/?q=%22memory-intense%22 I see 14 whitepapers found.
EDIT: URL is 404, try Google Cache.