We can have pruned nodes that can verify blocks and transactions so the block size can be increased without leading to the centralization of block verification.
But if we increase block size then block verification time and propagation delay will increase. It will increase the probability of two miners mining a block simultaneously and also delay the competing blockchain branches to resolve the dispute. Probably any increase in block propagation delay will increase the time to resolve disputes exponentially. Based on a website (Can't remember) currently, block propagation delay is 5 seconds to reach 90% nodes which may increase to 15-20 seconds if all transactions were Segwit.
I think block propagation delay is the only limiting factor in increasing block size.
Has anyone done an investigation to find out the relationship between block propagation delay and optimal block size?