The Lightning Network Whitepaper discusses a situation in which one party of a payment channel broadcasts an outdated commitment transaction. The other party through a breach remedy transaction will get the cheating party's coins.
How does this work though? If the dishonest party broadcasts a transaction to the Bitcoin network, isn't that transaction more likely to be accepted (since it was first) than a subsequent (up to date and legit) transaction?
You are right. If the dishonest party publishes an outdated commitment transaction it is most likely be accepted by the Bitcoin network.
The key trick is that the old state has an output script that gives the other party the opportunity to spend all outputs (even the one that was supposed to belong to the dishonest party) with the so called breach remedy transaction. This mechanism works via a time lock in one output of the commitment transaction and is called a Revocable sequence maturity contract (RSMC)
So if Alice and Bob have a channel with 5 mBTC for Alice and 7 mBTC for Bob. But Alice decides to publish an old commitment transaction in which Alice has 9 mBTC the outputs of that tx look like this :
The breach remedy transaction is the transaction spending the revocable output. So it is not an alternative to the fraudulent commitment transaction but rather a guaranteed way to spend the output of the commitment transaction that is supposed to belong to the dishonest person. It is guaranteed as the timelock prevents the dishonest person from spending their output.
The assumption in your last question is wrong, since there is no exclusivity in the publishing of both transactions. They can be accepted into the ledger and as a matter of fact, the subsequent transaction can only be accepted if the first one has been published on the network.
The first transaction, the Commitment transaction, has two outputs. The first output being the Delivery transaction, a simple transaction that pays to the commitment publisher counterparty (E.g. if Bob publishes his commitment transaction, then Alice can publish the delivery transaction for Bob's commitment transaction). The second output is a Revocable Delivery transaction, which allows Bob to withdraw his money from the channel after a certain amount of blocks have been mined (let's say 1000). I will explain the reason for this lock in a second.
The second output can be superseded by the Breach Remedy transaction. This is what you call the subsequent transaction. This is created in time with the new commitment transaction (to update the channel balance), and it is signed and given to the counterparty so they can claim the completing funds. E.g. Bob gives the signed Breach Remedy to Alice, so if Alice wants to publish this Breach Remedy. So if the Commitment transaction for this Breach Remedy is published, Alice can publish the Delivery transaction that has the money that is rightfully hers and the Breach Remedy transaction assigning her the rest of the money. This Breach Remedy transaction effectively takes the money that was assigned to Bob in the Revocable Delivery Transaction. Bob cannot take this money before Alice because Alice will probably take it before the 1000 blocks have been mined.
Knowing this, Bob wouldn't publish and Commitment transaction for which he has already given away a Breach Remedy. When a Commitment transaction has been compromised with a Breach Remedy, we can call it "outdated" and it should be deleted for risk of being accidentally published and thus losing all of the channel funds.
So simply put:
1) An invalid commitment transaction remains a valid transaction on the bitcoin network until the funding TX output is spent.
Each commitment transaction, once signed will remain valid until the channel funding transaction output (2/2 multisig) is spent by another commitment transaction or closing transaction. Even if a newer state has been negotiated between peers, an older commitment transaction is still "valid" on the Bitcoin network.
However, only one commitment transaction can be confirmed, otherwise the funding tx output would be "double-spent".
The commitment transaction can be invalid (channel) in the contract between channel peers (it represents and old balance), but remain valid (consensus) on the bitcoin network, as it is correctly signed, as long as it does not double-spend.
2) The breach remedy of an older commitment transaction happens after the invalid commitment transaction has been broadcast and confirmed.
The outputs of the invalid (channel) commitment transaction, once confirmed, can be swept by the counterparty. The invalidation happens by revealing a revocation key half to the counterparty, with which all outputs can be swept in a breach scenario, thereby penalising the party broadcasting a false channels state. Both RSMC and HTLC outputs are revoked and swept this way.
