What are we trying to predict in fee estimation and why?

Question

There are different estimations used in wallets, explorers and other Bitcoin projects. For example: estimatesmartfee in Bitcoin Core

This is the fee rate distribution right now according to https://btc.bitaps.com

Are different estimations misleading and affect the way fees are used in Bitcoin transactions? Will it be better if we just share mempool stats and user can decide the fee rate accordingly?

If I compare this with BTCUSD orderbook on any exchange, are we trying to estimate at what price buy order will get filled in certain time? Does that make sense?

In Bitcoin, block space is the supply, fee bidding to get the transaction included in a block is demand, miners are sellers who will fill the limit orders however they can include any transactions (not necessary it will be prioritized based on fee rate) so what exactly are we trying to estimate and why?

Market buy orders that fill the sell orders on exchanges can be compared to a transaction using fee rate 310 sat/vByte when highest bid in mempool is 300 sat/vByte. Although things are not instant in Bitcoin and there is a time gap between broadcasting with a certain fee rate and transaction being included in block. There are few other differences like if I try to buy 0.01 BTC right now on Bitfinex at 70,000 it will be executed at 53,450 and in Bitcoin there is no such thing, you can decide any fee rate and even if I use 1000 sat/vByte right now, same will be used and accepted by miners unless there is something that avoids such errors in wallet being used but overall picture looks similar and all this makes estimation even more difficult.

I consider it misleading because lot of users think a transaction with fee rate 1-5 sat/vByte will be included in 1 week or maybe a transaction with X sat/VByte will be included in Y time which is not true. User decide the fee rate and can do bidding, transaction will be included based on demand, supply and miners.

Will it be better if the wallets used this approach?

1. Show mempool stats
2. Leave the fee rate for user to decide
3. RBF every transaction and follow different algorithms for automated bidding

A simple algorithm for automated bidding can be:

Answer 1

What are we trying to predict in fee estimation and why?

Different Bitcoin projects try to predict fee rates for improving UX because its not easier for every user to understand all the technical things involved, look at different charts and decide a fee rate for the transaction.

Are different estimations misleading and affect the way fees are used in Bitcoin transactions? Will it be better if we just share mempool stats and user can decide the fee rate accordingly?

Some fee estimations maybe misleading but not everything. And they are only estimates which does not guarantee the transaction to be confirmed. Mempool stats can be shared in the wallets and most of the wallets already provide option to enter fee rate manually.

If I compare this with BTCUSD orderbook on any exchange, are we trying to estimate at what price buy order will get filled in certain time? Does that make sense?

There are few differences between BTCUSD market and Bitcoin fee market which are shared below.

Fee rates used for bitcoin transactions increase during the bull market and stay low in bear market.

BTCUSD follows cycles which are mentioned as bull and bear above.

The difference in two charts above is BTCUSD making new highs in every bull market and considered as a store of value in long term following logarithmic regression, whereas fee rates chart looks like sine wave and reset to 1 sat/vByte often.

Fee rates used in bitcoin transaction are lower during weekends because of less demand for block space however there can be exceptions and its not necessary that fee rates will always stay low during weekends. The amount of decrease in fee rates during weekends and fee market post weekend also varies. An example with weekends highlighted in last one month:

Other things that affect the fee rates used are technical:

1. Segwit usage: https://transactionfee.info/charts/payments-spending-segwit/
2. Batching: https://bitcointechtalk.com/saving-up-to-80-on-bitcoin-transaction-fees-by-batching-payments-4147ab7009fb
3. UTXO Consolidation: https://bitcoinops.org/en/xapo-utxo-consolidation/
4. Fee estimation used in different wallets: https://blog.bitgo.com/the-challenges-of-bitcoin-transaction-fee-estimation-e47a64a61c72
5. Coin selection algorithms used in different wallets: https://iohk.io/en/blog/posts/2018/07/03/self-organisation-in-coin-selection/
6. Layer 2 usage: https://bitcoinkpis.com/layer2
7. Exchanges: https://b10c.me/mempool-observations/2-bitmex-broadcast-13-utc/

Taproot will be added to this list once its activated and being used by different Bitcoin projects.

Will it be better if the wallets used this approach?

1. Show mempool stats
2. Leave the fee rate for user to decide
3. RBF every transaction and follow different algorithms for automated bidding A simple algorithm for automated bidding can be: https://i.sstatic.net/1SlPv.png

RBF algorithms for automated fee bidding can be tried and users should also manually prefer to use RBF for most of the transactions.

ZmnSCPxj shared few exceptions on bitcoin dev mailing list which should be handled while creating such algorithms:

Again, a race condition can occur --- while the wallet is feebumping a new transaction that includes the new output, a random miner can find a new block that includes the old transaction. Further, of course it is convenient to be able to spend money even while it is unconfirmed.

But the sender of the unconfirmed input might be using the same software as this wallet as well, meaning that the actual transaction output might change as the original spender keeps fee-bumping it over time.

https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-May/018886.html

Jeremy Rubin shared few other ideas in response to my question related to fee estimates and RBF:

In the long term, this can be particularly useful since you can use a separate fee-only wallet to arrange the bumps if your main wallet keys are offline, and you do not disturb any of the on-chain dependants.

I'd like to share with you a draft proposal for a mechanism to replace CPFP and RBF for increasing fees on transactions in the mempool that should be more robust against attacks. A reference implementation demonstrating these rules is available here for those who prefer to not read specs. https://gist.github.com/JeremyRubin/92a9fc4c6531817f66c2934282e71fdf

https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-May/018882.html