I am going to interpret this question from a different angle: If a node were to receive a block with two transactions that spend the same input, which transaction will it return to the mempool and which will it discard?
When a node receives a block that contains two or more transactions that spend the same inputs, the block would be marked invalid and thrown away entirely. None of the transactions in that block would be considered confirmed and any transaction that was already in the mempool will remain there. A transaction was not in the mempool but was in the block will not be added to the mempool.
As such, when the node receives this invalid block and validates it, when it determines it to be invalid, it throws away the block and does nothing. It does not change the UTXO set and it does not change the mempool. This means that if any of the conflicting transactions were in the mempool when the block is received, it will remain there. If none were, none of those transactions will be added to the mempool.
So if one of the conflicting transactions paid a higher fee, but the lower fee transaction was already in the node's mempool, the lower fee transaction would stay in the mempool and the higher fee one thrown away with the block.
The code for this behavior is here. ConnectBlock
takes a CCoinsViewCache
which is a cache for the UTXO set. This cache only writes out the changes made to it when its Flush()
function is called. ConnectBlock
will return false because the block is invalid and as such, this function will return here before Flush()
can be called here. So no changes will be made to the UTXO set.
Furthermore, that early return due to an invalid block means that the mempool update will not occur either. The call for the mempool update here will not be reached because of the early return and thus the the transactions that were in the mempool at the time the block was received will remain there, and the ones that weren't in the mempool will be thrown out along with the block.