From the source code:
Numeric opcodes (OP_1ADD, etc) are restricted to operating on 4-byte integers. The semantics are subtle, though: operands must be in the range [-2^31 +1...2^31 -1], but results may overflow (and are valid as long as they are not used in a subsequent numeric operation). CScriptNum enforces those semantics by storing results as an int64 and allowing out-of-range values to be returned as a vector of bytes but throwing an exception if arithmetic is done or the result is interpreted as an integer.
This does not explain why overflowing is allowed, but I guess this has to do with preventing errors (say, 3 billion + 3 billion resulting in a negative number).
As for the 4 byte limitation, I assume this is for safety. If there was no limitation, then because CScriptNum stores numbers as signed int64_t values, overflowing that limit (not the stack item overflow described in the previous paragraph) would result in undefined behavior according to the C standard. This is bad practice in general, but very bad in consensus-critical code. Most nodes run x86 Intel processors, but could you imagine how deeply in trouble we would be if a transaction was valid on Raspberry Pi based nodes and invalid on miner nodes, or valid locally on your smartphone and not on the network? All kinds of bad stuff would happen: hard forks, double spends, you name it.
You could also have problems with very large numbers being added to increase their size past the 520 byte push limit, then hashed repeatedly to consume as much CPU time as possible, resulting in a DoS attack.