The problem arises from the uneven propagation of information in the network. While the propagation of small blocks is fast, i.e., every miner gets to know a new block approximately at the same time, this is no longer true for large blocks.
Large blocks take a lot of time to be forwarded to every node in the network, since this includes miners, these miners are unaware that there is a new block and they'll continue mining on top of the old parent, i.e., they will work on a competitor of the newly found block. This means that they are actively wasting their computational resources on an obsolete solution if the propagating block gets accepted into the blockchain.
Big miners are likely to have a better connectivity with the Bitcoin network, resulting in them losing less of their computational time due. This in turn puts small miners at a disadvantage since they are now less competitive.
The extreme case would be one strong miner, with close to a majority of the computational resources making such big blocks that everybody else learns about its blocks very late, while it already has spent considerable time finding the followup block.
The uneven propagation is due to both network latencies and verification times of the block that is being propagated, so this is a compound problem from both slow verifiers and high latency/low bandwidth.
For more detail see Information Propagation in the Bitcoin Network (disclosure: I am the author of that paper) and Majority is not Enough: Bitcoin Mining is Vulnerable.