My understanding of how the blockchain works is that to determine the balance of an address one must find all the transactions that have outputs to that address. So, the only way to be sure you are not missing a transaction is that you have to scan the whole blockchain to determine the balance of an address.
Is that correct?