Basically, is there any reason not to encrypt your wallet? The only drawback I can think of is forgetting your passphrase and losing access to all your bitcoins.
The benefits far outweigh the drawbacks here, but it's a worthy discussion nevertheless!
Starting with encryption in general: Encrypting the wallet.dat file (and backups etc) with a third-party encryption tool like truecrypt can be time-consuming and requires a lot of hoops to jump through every time you start the client. But this might be more secure and allow you to put the whole client on an encrypted bootable usb stick for portability.
Using the qt's built-in encryption is a lot easier and other than what you mentioned the drawbacks are: The possibility of losing the password. The need to type the password upon starting the client. Waiting for the wallet to encrypt/decrypt itself upon starting/stopping the client. The pain level here is pretty much entirely dependent on how fast your computer can get the job done.
Theoretical possibility of some bug in the encryption algorithm (or something simply goes very wrong during the process that corrupts it) that might render your wallet unusable if it's unable to be decrypted. I don't know if this has ever actually happened--just thinking out loud here. But I suppose creating backups helps safeguard against such a problem. Although then you need to be sure to secure the backups -- which again creates a new series of todos and requires more time and care to maintain. All of this stuff can be automated, however.
Don't seem to be any other potential drawbacks I can think of. Like I said, none of those issues are likely worth arguing against encryption.