1

Apparently the way to find a list of UTXO related to an xpub is to derive all addresses from the master private key and query a node with each and every one of them.

It is possible to generate 4294967296 addresses from a single extended key.

How should a wallet proceed to manage this in an efficient way?

I guess that in an ideal situation, addresses are generated in a sequential order... but my intuition tells me that nothing prevents a wallet from doing otherwise.

5

The balance checks are cut short based on a gap limit.

A gap limit is usually set to 20, and is defined as a consecutive block of addresses with no transactions.

Since HD wallets only give out a new address on explicit request, or automatically when the previously issued address has been used, it is assumed that someone will not leave a block of 20 consecutive addresses unused.

Wallet software will generate a short list of maybe addresses when you load an xpub or other form of seed. It will then check through that, and if it doesn't find 20 consecutive unused addresses, it will generate another short list (so the first list might cover 0-100, then 100-200, etc). until it finds the block of unused addresses.

This assumption naturally falls apart if you are running an exchange, payment service, or other wallet set up where it is very likely that you will have many hundreds of unused addresses. Those systems generally keep track of assigned addresses out of band, and have specialized indexer systems that can track many millions of addresses.

| improve this answer | |
  • The software I am building is about keeping track of funds related to an xpub/ypub/zpub no matter the origin. My questioning is about the fact that nobody can prevent wallets from overriding the gap limit. Anybody can write one pretty easily. So I guess there is no totally reliable way to know how many funds are related to an xpub. – RooSoft Sep 9 at 12:55
  • 2
    If you intend to deal with arbitrary xpubs, then yes, you have no choice but to check every address on every possible account in that xpub, which is not a viable approach (a single xpub will lead to many trillions of addresses). If you are only dealing with xpubs generated by other, sane wallets like electrum, ledger, trezor, etc. then the gap limit is a safe assumption – Raghav Sood Sep 10 at 3:01
  • RooSoft, You're thinking of an xpub like an address, which it is not. It is a brute force issue if someone uses a random address deep in the chain. Fortunately no wallet software will use a random index so utilizing the gap limit approach is the most efficient way to check for UTXOs related to an xpub. – m1xolyd1an Sep 10 at 3:02
1

Not all wallets are the same, but a good wallet should always follow standards. A standard for address-generation generally accepted is BIP44. In this way, you could scan the first consecutive addresses that a BIP44-compliant wallet would create. If the wallet supports Segwit, then you should look at BIP49 and BIP84 as well.

As Raghav Sood pointed out in his answer, most wallets scan a certain amount of unused addresses before they decide to stop scanning for funds. Most of them use 20 as the gap limit (maximum gap in between unused addresses).

Now, to answer your question, you are totally right. Nothing prevents a wallet from not complying with BIP44 standard or with any other standard whatsoever. That is why it is important to know the wallet you are dealing with. However, a well defined protocol/standard helps with the fund-discovery process which is super important for any kind of wallet, for which it is almost certain that every wallet out there possesses a standard for address creation (either BIP44 or one of their own).

Just find out the standard that the wallet you're dealing with is following for address creation, and that would lead you to find the most efficient way to find the UTXO associated to the xpub.

| improve this answer | |
  • Thing is the software being built is about tracking funds from any source the xpub is from without discrimination. I wanted to know if there is a sure way to track all funds without making billions of requests. What I have come to realize is that if a wallet goes beyond the 20 empty derivation gap, some actual wallets might not keep track of those extra UTXOs as we speak. – RooSoft Sep 9 at 13:22
  • 1
    ah ok. Then, I guess the standards, as I mentioned, would work for most cases. And yes, you are right about the gap. Although, you are usually able to fix that number if you notice that the total of your funds is not being reflected. Take into account also the mechanics of wallets for address creation, and what are the odds of an address to be used beyond the 20 gap. In your case, if you want to make sure that you are scanning the total of the funds, then you can use a wider gap. It might seem too much, but that is the most efficient way for scanning. There is no other possible shortcut. – Oscar Serna Sep 9 at 13:59
  • Thank you so much Oscar... helps increase confidence I have in what I am trying to do. – RooSoft Sep 9 at 14:10
0

An electrum wallet (legacy, testnet)

Electrum-XPUB-Addresses

C# code for below windows application that returns 10 addresses from one XPUB using NBitcoin

var pubkey = ExtPubKey.Parse(textBox1.Text, Network.TestNet);
                    

        for (uint i=0; i < 10; i++)
        {
            var Address = pubkey.Derive(0).Derive(i).PubKey.GetAddress(ScriptPubKeyType.Legacy, Network.TestNet);
            richTextBox1.AppendText(Address.ToString());
            richTextBox1.AppendText("\n");
           
        }

WindowsApp-XPUB

The gap limit is the maximum number of consecutive unused addresses in your deterministic sequence of addresses. Electrum uses it in order to stop looking for addresses. In Electrum, it is set to 20 by default, so the client will get all addresses until 20 unused addresses are found.

Electrum will generate new addresses as you use them, until it hits the gap limit. Addresses beyond the gap limit will not automatically be recovered from the seed. To recover them will require either increasing the client’s gap limit or generating new addresses until the used addresses are found. If you wish to generate more than one address, you can use a “for” loop. For example, if you wanted to generate 50 addresses, you could do this: [wallet.create_new_address(False) for i in range(50)]

https://electrum.readthedocs.io/en/latest/faq.html#gap-limit

If I run the above code with i < 50 in for loop I get 50 addresses from the XPUB

gap-limit-1

Gap limit can be changed with: wallet.change_gap_limit(50)

Now I see 50 addresses instead of 20 (excluding change addresses) in electrum wallet same as returned by the code above:

gap-limit-2

| improve this answer | |
  • This does not seem to answer the question. I believe OP understands how addresses can be generated from xpubs, but is asking how wallets manage large numbers of them. – Pieter Wuille Sep 9 at 3:05
  • Edited and added information about gap limit and getting more addresses – Prayank Sep 9 at 3:53
  • @PieterWuille I am writing custom software akin to read only wallets. It accepts xpubs from any source and has no idea how keys have been derived in the past. Thanks for helping, I am a big fan of yours! – RooSoft Sep 9 at 12:00
  • Thanks @Prayank for pointing out how the electrum wallet works... this is a great way to understand that derivation adresses gap thing. – RooSoft Sep 9 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.