First of all, what is the goal? The Bitcoin consensus rules already completely eliminate the possibility of an "double spend" within any individual version of history ("chain"). The concern is about individual wallets accepting a transaction as a fullfilled payment, before it is certain that that transaction will end up in the winning chain. So when we speak about problematic "double spends", the term is confusing - no coin is actually spent twice - instead, it is the confusing name we give to "being convinced you were paid while in reality you aren't".
And there is an easy solution for this: waiting until the transaction is sufficiently confirmed before treating it as accepted. This approach is suggested in the Bitcoin whitepaper, together with an analysis that shows how risk of reorganization falls with increasing confirmations, assuming no attacker with a majority of the hashrate exists.
So the whole concern is about being able to accept transactions before they gather sufficient confirmations. And the answer is simply: if you want to be sure, you simply cannot. This is a fundamental problem: global consensus among untrusted participants is hard, and in Bitcoin's case, which relies on economic incentives to see that concensus emerge, you need proof-of-work to build up any guarantee, and that takes time - multiple blocks' worth of time.
If you're thinking of a rule where nodes reject blocks that contain transactions that they believe are fraudulent/doublespends, you're just introducing a recipe for the network to fork. We need a way to make all nodes agree on what transactions are valid, and if we had any other way of doing so without using slow and costly proof-of-work, we would.
It takes approx. 8 seconds for a transaction to propagate through the entire Bitcoin network. If we got rid of the concept of replace-by-fee and replaced it (no pun intended) with first-seen approach wouldn't this significantly reduce the likelihood of a double spend?
That's how things worked before the introduced of opt-in RBF (BIP 125), and still how it does work for transactions that don't signal RBF. However, its promises are widely understood to be an illusion: there were (and are) known ways to construct transactions that propagate slower than others through the network. All an attacker needs to do is create a slow-to-propagate transaction that pays you and broadcast it "close" to you in the network, while broadcasting a fast-to-propagate transaction that pays himself and publish it in many other places on the network at the same time. Most miners will see the second one first, and that transaction will have a higher chance of winning.
And if you look at incentives, you don't even need that. Miners have an incentive to accept the highest-fee transaction, regardless of when they see it. Even if the network doesn't relay it, nothing prevents an attacker from sending a higher-free transaction directly to a miner. They'd be stupid to reject it. Bitcoin software initially (and still, for non-BIP125 transactions) was written to reject such replacements, but as profit margins for miners shrink and the industry becomes more competitive, it seems very unlikely to me that miners will not eventually just accept whatever maximizes their income.
Even if an attacker is running another wallet on the other end of the planet and broadcasts an identical transaction right near the winning mining node, such a double spend transaction would still reach the merchant within < 16 seconds.
That's assuming that the network provides a reasonable guarantee at relaying conflicts. It doesn't. Nodes only relay the transactions they themselves accept. Relaying conflicting transactions so the rest of the network can learn about conflicts has many problems:
- It doesn't necessarily work. If the attacker hands the conflicting transaction directly to a miner, they may not have any incentive to reveal it to the rest of the world before it is mined.
- Done naively, it is a trivial DoS attack: an attacker simply creating a myriad of conflicting transactions would result in the network relaying all of them, at no cost to the attacker except the fee of the one variant that eventually gets mined.
- Done less naively, with precautions in place to prevent said DoS attack (e.g. through rate limiting that kicks in at some point), it becomes trivial to bypass: first produce enough transactions to trigger the rate-limit, and only then perform the attack now the network has decided they've had enough of your conflicts.
- It helps the attacker. If the conflicting transaction pays more, the network relaying it will make it easier for economically rational miners to pick it up.
In fact, there is barely any guarantee that even honest transactions propagate well in adverserial setting. It is even less the case for conflicts.
Would a merchant be able to simply listen out for a double spend transaction and conclude with a reasonable high confidence level that it is very likelihood for a double spend to happen in this case?
It is certainly possible to use heuristics to try to analyze the probability of conflicts, and for sufficiently low-value transactions, it may be fine to treat the occasional risk of a reversed transaction as a cost of doing business. But remember that past performance is not a promise of future success. Just because nobody attacks you doesn't mean it isn't trivial to do so; it may just be that it wasn't ever worthwhile to try.
TL;DR: if you want to be sure about a payment, wait for confirmations
