I understand that the blockchain is what keeps track of who has what coins and makes sure that everything is correct and secure, but why do I have to download it? Isn't that where the miners come in? Don't they keep track of the blockchain?
Certainly miners, and many other users, keep track of the block chain.
But if you don't download and verify the block chain yourself, what will you do when you want to know whether a transaction is valid? You could ask a miner, but what if they decide to lie to you? You would have to trust them to tell you the truth.
Part of the Bitcoin security model is that you don't have to trust anybody. When you verify the block chain for yourself, nobody can trick you into believing that an invalid transaction is valid, or vice versa.
Bitcoin is trustless, so you need to validate the transaction yourself. The miner are rewarded for protecting the network by working on special hash. A miner that would release a block with an invalid transaction would be rejected by all nodes and would not work... because everyone is validating everything.
The whole blockchain is not required to validate, in fact you need only unspent outputs. Right now there is no way to get that beside download the whole blockchain.
The miners create the blockchain, most of them don't keep track of it directly, just the mining pool operators, and (if there still are some) the solo miners.
Actually, as the recent blockchain fork shows, not even all mining pool operators are reliably checking the blockchain.
So, ultimately, the checking is mostly done by full nodes, which also provide data to SPV wallets. If you are running Bitcoin-Core, you are part of those that provide this verification.
If that's not what you are interested in signing up for, you might be happier with a lightweight wallet.