... it needs to fetch peers from a centralized source.
Not exactly. Once you are connected to the network, you will begin to receive lists of IP addresses and ports of known clients.
There is a list of seednodes built into Bitcoin. These addresses are only used as a last resort.
After you connect to the network once, you won't need to use the DNS seeds, etc. anymore. The Bitcoin network has a system which allows every network participant to (over time) get the IP addresses of all other network participants. Bitcoin stores this info in a local database(
peers.dat) and contacts these nodes when it next starts.
Are we just ignoring this issue because it'll be hard to gain control
of the servers?
Note that if user have found out that he is under attack, he just needs to connect to one real Bitcoin node. It would be "quite difficult" for an attacker to pull this off, and probably not financially worth the while.