The EVM is actually the core of Ethereum. All contracts are written in EVM bytecode. This doesn't mean that Ethereum includes an actual virtual machine in the sense of Hyper-V, Vmware or Xen. It contains something[*] that can execute EVM bytecode (that something can either be interpreter, jit or aot). So, VM in this case is just a name for emulating the bytecode.
Without the EVM, all transaction types would have to be hardcoded (this would limit capabilities, but would also be much safer).
Why do you think that Bitcoin does not have a VM? It actually has. Script is Bitcoin's bytecode, you can read more about technical details here: https://en.bitcoin.it/wiki/Script