TL;DR: you don't, you enable RBF and adapt the transaction fee to the block space market (or use a wallet that does).
Best effort(s)
One is interested in the cheapest reliable fee estimation. That is, in estimating the lowest fee that will get their transaction included in a block before a configured deadline.
There are 3 classes of estimates:
- Those based only on the mempool. For instance Electrum's. Or just users looking at Johoe's before making a transaction. It consists in targeting the depth at which you want to add your transaction (below 3 blocks worth of transactions, below 10 blocks, etc..). However looking at the snapshot of the mempool does not give you any guarantee about the time it will take for your transaction to be included in a block. This method is also gameable by miners.
- Those based only on the block chain. It consists in looking at the feerate of the transactions that were included in the previous block. It however against doesn't give you any ETA guarantee, and is even more gameable by miners than looking at a snapshot of the mempool.
- Those based on both the mempool and the block chain. This is the case of the Bitcoin Core fee estimation algorithm. Instead of looking at past blocks or a snapshot of the mempool it will look at the dynamics of transaction confirmation (when was a transaction broadcast, how long did it take for it to confirm, at what feerate, ..) in order to give you more assurance regarding the ETA you can expect by choosing a specific feerate.
However note that no estimation can ever be 100% reliable, it can't predict the future. If after your transaction is broadcast someone wants to fill the blocks up to your target with megabytes of high-fee-paying transactions, they just can.
A guess isn't reliable
Since you cannot reliably estimate transaction fees, you need to adapt your price to the market. However you will only witness the market actions after broadcasting your transaction. This is the reason behind Replace-by-fee (you need a way to bump your fees post-broadcast).
But RBF is opt-in, you need to enable it before performing a transaction. In case you already broadcasted your transaction, you can fall back to CPFP (spend the unconfirmed output with a second, high fee paying, transaction that pays for both --example with Electrum).
Conclusion -- what should i use?
First of all use RBF when possible. Start with a lower estimation and bump the fees as your transaction is approaching your deadline without getting confirmed.
If you are building an application, prefer Bitcoin Core's block chain estimations as:
- Reliability often outweighs cost (stuck transactions really aren't fun).
- With a larger sample of transactions, statistics are on your side.
- If all applications start relying on mempool estimations and fees continue to rise it becomes attracting for miners to try to play games. :)
