What is the algorithm used for encrypting the wallet.dat file?
A detailed explanation can be found in the application's README: https://github.com/bitcoin/bitcoin/blob/6b8a5ab622e5c9386c872036646bf94da983b190/doc/README
- The passphrase is converted to a key/iv pair using EVP, with a dynamic number of rounds
- This key/iv pair is used to encrypt a randomly-generated master key, using AES-256-CBC
- The secret part of wallet keys are then encrypted using that master key, again with AES-256-CBC
Peter's answer covers the technical details so I won't repeat any of that. I would add that users should remember the weakest point in Bitcoin-core wallet or any cryptographically strong encryption system is the passphrase. The system converts a passphrase to an AES-256 key and that password derived key is used to encrypt a randomly generated master wallet key. The master wallet key can not be brute forced barring a compromised or flawed PRNG however attackers can attempt to brute force the passphrase itself.
The client adds attempts to harden against this attack by using both a salt (random number) and multiple rounds of hashing. The salt prevents a precomputation attack as the same passphrase with different salts will produce different keys. The use of multiple rounds of hashing slow down brute force attacks because the attacker will also need to perform multiple rounds on each passphrase attempted.
The client uses dynamic key hardening. This means the number of rounds varies based on the computational power of the client (to keep time to encrypt or decrypt below 1 seconds). The more computing power your hardware has the more rounds it performs, and the more difficult it will be to brute force the password. When you change your passphrase the client will adjust the number of dynamic rounds based on your current hardware's computing power. By updating your password periodically you can ensure the password derived key keeps up with Moore's law.
Key hardening through the use of salt and key stretching can only go so far. If your password is "p@ssw0rd!" or some other known commonly used password it will still be trivial to brute force the wallet. To illustrate why lets assume the attacker has hardware which can attempt 100M hashes per second and your wallet is using 100,000 rounds of hashing. This means the attackers hardware (100M hashes) can attempt 1,000 passwords per second. So the wallet does slow the attacker down but if the password is weak then it can't slow it down enough. There are lists of previously compromised/stolen/hacked passwords publicly available with tens of millions of passwords. Without key stretching an attacker could attempt all possible known passwords in less than a second. Key stretching slows that down to a few hours. On the other hand lets say that your password was unknown and strong enough that with a single hash it would on average take one day to brute force. With 100,000 rounds that increases the time to centuries.