If there were fewer miners, Bitcoin would be less reliable and you'd have to wait much longer to get the same level of security against a double-spending attack. We need as much mining as possible because mining secures the blockchain.
If I have 10 Bitcoins, I must be able to send them to Jeff. Otherwise, I don't have 10 Bitcoins. If I have 10 Bitcoins, I must be able to set them to Mary. Otherwise, I don't have 10 Bitcoins. But if I could send them to both Jeff and Mary, the system would break. So somehow, sending them to Jeff must prevent me from sending them to Mary too.
Bitcoin does this by having miners pick one of those transactions and heap a massive amount of computations on top of it. Among valid blockchains, Bitcoin prefers the one that likely took the most calculations to do. Miners pile calculations on top of the legitimate block chain to distinguish it from alternate chains. To "undo" a Bitcoin transaction, you must pile more computations on top of it than the rest of the miners combined piled on top of the legitimate chain that includes that transaction. The more mining, the faster you can accumulate sufficient calculations that undoing a transaction is impractical.