Bitcoin uses a 8 bytes value field in the transaction output. However, the total supply of bitcoin is capped at 21M. Theoretically, even if all the 21M bitcoins are transferred to a single address, the output, after converting to satoshis, can be effectively represented with 7 bytes of data. This means, we are adding at least 2 bytes of redundant data (one for the spend address and other for change address) for every transaction stored in the blockchain. Is there a particular reason why a size of 8 bytes was chosen or was this a design flaw that was overlooked by Satoshi? Is it possible to bring a change to this via a soft-fork or does changing this require a hard-fork?
Is there a particular reason why a size of 8 bytes was chosen
In the original Bitcoin source code, many of the primitives used in data structures were simply the in-memory representations of those primitives. For integers, this is a 32 bit (4 byte) little endian integer. However 32 bits is not enough to represent all of the satoshis that exist in Bitcoin. The next primitive that can be used is a long, aka a 64 bit integer. Since this was just serialized as it is represented in memory, the amounts end up being 64 bit little endian values.
Is it possible to bring a change to this via a soft-fork or does changing this require a hard-fork?
The serialiation format used for computing the weight and hashes can only be changed via a hard fork.
However you can transmit a transaction using whatever serialization you want so long as that format contains all of the necessary information to construct the serialization necessary for computing weight and hash. This would not require any fork because it is not a consensus change.
Yes, transaction format has unused bytes. One other example is encoding utxo index in four bytes. Nobody cares. We can not change the format without hard-fork. And we do not want to do it. Live with it.