What if one block contains many more transactions than the other? Wouldn't it be better to pick the block with more transactions? Also, why is a tie ultimately broken randomly, rather than a deterministic way that helps the network?
Consensus rule dictates that chain work is measured based on the production of a solution which satisfy the difficulty set by consensus. So as long as a produced block satisfy said rule, said produced block would be accepted.
It is technically infeasible to choose base on blocks with more transaction for a variety of reason. First, bitcoin lacks strong consistency (i.e. its impossible for nodes to share a consistent state for its mempools), if block acceptance requires that a block based on the highest number of transaction is accepted, then the blockchain will most likely be orphaned many times as miners compete to produce competing block tips and nodes would discard block tips literally each time a new block with more transactions is received. This results in more partitioning in the network, which leads to a poorer security as less work is focussed on any particular branch of the chain.
It makes for a stabler system if we pick the first that we receive.
The whole point of Bitcoin's blockchain is to determine a fixed order for the transactions. Two blocks being discovered at the same height is an unfortunate incident that cannot be circumvented due to the probabilistic nature of mining. It is not something that we want to occur: Two competing blocks defer the certainty of the block order to a later time but we want it to generally become stable as soon as possible.
Thus, the first block that is seen in order re-establish consensus as quickly as possible and to incentivize miners to immediately switch to mining on the successor block instead of trying to oust already existing ordering.
The current protocol rules for breaking ties is explained here. Breaking ties by number/ value of transactions is not only infeasible as other answers here explain, but is also easily gamed. An attacker could simply move his own coins around to have the most number of transactions. Breaking ties based on time received is better, and the third rule is almost never used.
However, here is research that shows breaking ties by time received does not make for a stabler system. This paper on selfish mining explains the idea. To summarize the intuition here:
- An attacker mining pool may choose to not immediately publish a block it mined, but instead intentionally fork the blockchain. The pool would continue mining on its private blockchain as long it has the lead.
- When the public blockchain approaches the length of the pool's private blockchain, the pool publishes all its private blocks.
- The effect of this strategy is that honest nodes waste hashes on blocks that will never be a part of the final blockchain, while the pool's revenues are disproportionately greater than its hash power.
The paper also shows that even if conditions are extremely unfavorable to the selfish pool, selfish mining would be advantageous to the pool as long it has more than 1/3rd of the total network hash power.
Rule #2 fails here because the selfish pool may set up lots of virtual (dummy) miners whose sole purpose is to peer with other nodes, stop the propagation of honest miners' blocks and instead propagate the selfish pool's blocks when it chooses to publish its private blocks as explained above.
The paper instead proposes that ties should be broken randomly, which would put the selfish pool at a disadvantage (virtual miners and being better connected to the Bitcoin network will not help the selfish pool here).