There is no such thing as a transaction "balance". I am going to assume if I jump right to the answer it won't make much sense so the first two sections are to make sure we are how Bitcoin "really works" (at a very high level abstracted view). We need to speak the same language for the answer to make any sense. For this question the two required concepts in how Bitcoin really works that need to be understood are inputs & outputs, as well as where transactions are recorded.
Inputs & Outputs
Bitcoin works on the concept of discrete inputs and outputs not spending part of a balance. All transactions have as their input a reference to a previous unspent output. The transaction records one or more new outputs (which are referenced in the inputs of some future transactions). Outputs are "spent" when they are referenced in a new transaction. Outputs can only be unspent or spent, they can't be partially spent.
Wallet balance (or address balance) is an abstraction to help us humans and make Bitcoin more like conventional payment systems. Balances are not used at the protocol level. When the wallet indicates your confirmed balance is 1.2 BTC it is saying that the sum of the value of all unspent outputs in the blockchain which correspond to public keys it has the private key for total 1.2 BTC. In other words the wallet is computing the total value of the outputs which it can spend which requires a) the output be unspent and b) the client has the private key necessary to spend it.
Storing Transactions
All confirmed transactions become part of the blockchain. Clients validate transactions and blocks; only valid transactions and blocks are added to the blockchain. For the purposes of validating transactions and computing available balances only part of the blockchain is needed and that portion is called the UTXO (Unspent Transaction Output Set). This is a subset of JUST the unspent outputs. As all future transactions will reference unspent outputs as their inputs only the unspent outputs are needed for validating new transaction and computing the balance (which is another way of saying the total value of outputs which "can" be used in new transactions).
The UTXO is much smaller than the entire blockchain (about 15% of total blockchain right now and that will shrink over time). It may not be immediately obvious but the number of Bitcoins in the UTXO is always equal to the number of mined Bitcoins.
Unconfirmed transactions are not part of the blockchain (or UTXO). Clients maintain an unordered list of all the unconfirmed transactions they are aware of. This list is called the memory pool.
Computing "Balances"
The creation of the UTXO makes computing balances straightforward. The UTXO contains all unspent outputs however your client can only spend the subset of those outputs which correspond to public keys that it has the private key for ("your keys"). This subset has no official name however the RPC call to output this list is listunspent so I will call it the listunspent set.
The wallet uses the listunspent set and memory pool to determine the current balance. In reality wallets compute this in "real time" as changes to the memory pool or UTXO occur (i.e. when the client learns of new blocks or transactions it records them and then makes changes to its reported balance if any). For the sake of brevity I will just show how to get a "snapshot" of the current balance. There is a confirmed balance and an unconfirmed balance, clients often show a reduced-current balance to simplify things for users.
Confirmed Balance
The confirmed balance is the sum of the value of the "listunspent" set. Example: the listunspent (outputs in UTXO which client has keys for) has 20 outputs and combined they have a total value of 1.2 BTC so the confirmed balance is 1.2 BTC.
Reduced-Confirmed Balance
Client usually do not show the "real" confirmed balance because users are more interested in what they have available to spend. Transactions in the memory pool which have inputs that reference outputs in the listunspent set technically have no yet reduced the confirmed balance. In most cases they will eventually confirm so clients will remove these outputs when computing the "confirmed" balance. The "confirmed" balance in most wallets is actually this reduced-confirmed balance. It is done to abstract the way Bitcoin really works. If it wasn't then the confirmed balance would not reduce until a transaction was confirmed which might seem confusing to a user (they have 100 BTC, spend 1, it still shows 100 BTC). A client could break this down into "confirmed balance", "reserved for unconfirmed transactions", and "available balance".
Unconfirmed Balance
An unconfirmed tx "to" the client will not be in the UTXO yet. It will however be in the memory pool. The sum of all the outputs (which the client has keys for) in the memory pool is the pending balance. Those tx will either be confirmed (the tx and output is removed from memory pool and added to the UTXO) or they will be dropped.
Handling Change
Due to the way Bitcoin handles "change" (outputs can be partially spent so the unused portion is sent back to the user) the simplified example above while accurate would be confusing to the user. Most wallets "cheat" and include the unconfirmed change outputs in the "reduced-confirmed" balance. This isn't a requirement however not hiding this aspect from users may lead to some unexpected behavior. Imagine a client with a reduced-confirmed balance of 100 BTC and a pending balance of 0 BTC. The user "spends" 1 BTC however the outputs referenced in the new tx total 5 BTC. This means the new outputs are actually 1 BTC + 4 BTC "change". If balances weren't modified to "hide" the change, after the spend the user would see confirmed balance of 95 BTC and a pending balance of 4 BTC.
