What happens if, for instance, a block's nonce is so large that the block's size exceeds the limit?
This post addresses the main question.
Don't think it is complete so I will append the following and let someone wiser mark it all.
- A solo miner increments Nonce until it overflows.
- Then it increments extraNonce, which is an integer of arbitrary precision, and resets Nonce.
- extraNonce is located in the coinbase transaction, so changing it alters the Merkle tree.
- extraNonce is reset based on the time.
As @Claris commented, and this post elaborates:
There are many blocks that could be the next one.
For many potential blocks, there is no value of the nonce that will give a hash less than the target.
When that happens, the node must alter something in the block header and try again...
Is there any systematic way of setting this limit without knowing how to break SHA256?
SHA256 has little to do with the validity of a block, having broken SHA256 in any way would not change the consensus rule of the block size. An invalid block with valid proof of work is still invalid, for example if it was larger than the rule specifying limits.
As an aside, no hash function has ever been broken in a way that would impact Bitcoins proof of work. Other than being too small, MD5 is completely suitable for POW despite being embarrassingly broken for any other task.