Which transaction would get confirmed in an uncongested network?

Question

Scenario:

1. Mallory sends some bitcoin with a $1 fee
2. Later, Mallory double spends her transaction with a $5 fee
3. A new block is found

In an uncongested network with an average transaction fee of ~20 cents, which transaction would most likely go through?

Answer 1

Transaction selection is the prerogative of miners, and specific behavior in regard to unconfirmed transactions cannot be enforced on the network as it precedes the consensus mechanism (block creation). Therefore, as chytrik states, it's completely up to the miner who succeeds at creating the block.

However, in December 2020, the prevalent transaction selection behavior by miners still appears to be "first-seen safe". Most nodes on the network also do not forward doublespend transactions, so a conflicting transaction would likely need to be submitted directly to a miner that does not adhere to first-seen safe selection. On the other hand, a network participant would not be able to reliably know whether any miners are attempting to include a doublespend until the block is published.

Even so, I'm aware of some businesses that conditionally accept transactions before confirmation. They rely on risk assessment based on factors such as feerate, amount, existence of a customer relationship, and probably more. I do not recommend accepting transactions before confirmation without risk assessment in place.

Note that opt-in RBF (see: BIP125: Opt-in Full Replace-by-Fee Signaling) transactions are explicitly unreliable before confirmation as the sender may overwrite them at-will. Opt-in RBF transactions MUST NOT be accepted before confirmation.

Answer 2

On an uncongested network with an average tx fee of ~20 cents. Which transaction would most likely go through?

The transaction which confirms is the one chosen by the miner which finds the next block. There is no way to know which one they may choose. It may be the first one they saw, or it may be the one with the highest fees, or it may depend on some other factor that is relevant to the miner in question. It is simply up to them.