Version 0.8 of the reference client changed the database format completely, in order to scale better. That meant a one-time rebuild that everyone had to go through, as converting the old database would have been error-prone.
With that change, the transaction index was made optional, and off by default. This is because for normal operation (verifying and processing blocks and transactions, and maintaining a wallet), such an index is not necessary.
-txindex (or txindex=1 in bitcoin.conf) was introduced to re-enable this transaction index (which is necessary for the
getrawtransaction RPC call). It has to be specified when the database is created though, so at first run of a post-0.8 version, or when explicitly rebuilding the index (with
This type of database format change is not expected to happen frequently.