Isn't it enough to just have the Input Address, Output Address, Amount and Signature for each transaction?
We don't refer to previous transactions, but rather to specific outputs of previous transactions. When transactions sign over money to a recipient, they create a uniquely identifiable "Unspent Transaction Output" or UTXO. Transaction inputs in turn spend these "UTXO" to fuel transactions. Since addresses can be reused, addresses alone are insufficient to precisely identify the bitcoins that are being spent.
If you only had "Input Address, Output Address, Amount and Signature for each transaction". This would be trivially broken:
Let's say Alice keeps all her funds in address
A1, and pays Bob 0.1BTC in rent to address
Bob pays Mallet to fix Alice's heater from
B1, using the funds that he received from Alice.
When Alice pays Bob next month again from address
A1 to address
B1, Mallet can replay Bob's transaction to himself in order to get paid again.
Bitcoin doesn't track account balances like traditional accounting, it only keeps a record of where BTC came from and which address it is sent to. So, your receiving address doesn't technically have a recorded balance, only a calculated balance for all the transactions received on that address. Once you send Bitcoin, the unspent transaction output you reference when sending becomes spent and the calculated balance for that address you originally received on becomes zero.
You may find this video of assistance:
How Bitcoin Works Under the Hood