Recall that transactions have inputs and outputs: when an output is listed as the input of another transaction, that output is spent.
So you go through the block chain from the beginning and make an index of unspent outputs. For each transaction output you see, you add it to the index; for each input you see, you delete the corresponding output from the index, because it has been spent.
When you have an index of all unspent outputs, you look for those whose recipient is the address X. Adding up the amounts of all those outputs gives you the "balance" of Bitcoins available to address X.