how to monitor +20,000 addresses for new received transactions?

Question

I have around 20000 generated addresses which I want to monitor for new transactions. More addresses should probably come. Customers should use every address only once but I would like to monitor already used addresses if some of them decide to transfer coins to "used" addresses.

These addresses are not in the online wallet, I have only addresses in the databases. Private keys are held offline.

That means that I do not want to monitor addresses from the local wallet but the addresses from databases.

A command should executed after a transaction is received (with enough confirmations). There are some monitor services which can send an email or call url after receiving a transaction for an address on the watchlist.

I would like to implement the same feature for my application without depending on external service.

I saw some questions and answers like:

How to know there is a new transaction? or Monitor all transactions on bitcoin network - API

But what would be the most efficient way to do this? Is there a perl/php compatible way to do that?

One of topics from above links to https://bitcointalk.org/index.php?topic=89725.0 where Gavin is talking about some new features but I did not get how could I or whether I can use anything of this to monitor a high number of addresses. Sending every minute a batch RPC call with a few thousands addresses to bitcoind is probably not a solution :-(.

I hope that there is a elegant way to monitor the whole information stream (all incoming transactions and other incoming information) and react if it matches what I need.

Answer 1

I am basically doing the same thing for bitcoinmonitor.net. I have a database of addresses to monitor and get notifications from bitcoind for any incoming transaction.

I maintain a patchset for bitcoind which allows setting an url which will be called with any incoming unconfirmed transaction, including transaction details in a json object. Also for any new accepted block a (different) url can be set up to be called with details on the new block.

So you will get a notification for any incoming unconfirmed transaction, then you can match this with the monitored addresses in your database. If you want to trigger actions after a certain number of confirmations you need to hook up to the new block event.

Check https://github.com/TripleSpeeder/bitcoin for my bitcoind fork. I try to keep it more or less in sync with the current stable official client. This is running stable with bitcoinmonitor.net for around 10 months now.

Answer 2

You could import all of those keys into a wallet using a very long password that you will never use again. Then, you could make bitcoind handle keeping track of all of those addresses and their balances, while not being able to spend any of them. The rest can be handled through the JSON RPC API. It might not be the neatest way of doing things, but it should work.

Answer 3

If number of addresses in your local wallet is not too high, you can search all transactions (in and out) and filter them to seperate addresses which is not in the database. If you have a large number of addresses in the local wallet, this may not be helpful for your case due to very high number of "string comparisons". (In this case, just use another wallet for the ones in the DB).

1. If this is the first time for monitoring, start searching from Block-0 (genesis) "getblockhash n" RPC command will give you hash of n'th block. Use 1 to get hash of the block which is just after the Genesis-Block. Then use "getblockheader"

   getblockhash 1

2. Use the result with "getblockheader" RPC command. This will give you header of a block which includes "previousblockhash" variable.

   getblockheader 'blockhash'

3. Use 'previousblockhash' variable from the result you got at previous step. "listsinceblock" RPC command will return all transactions since given block. (Attention: transactions in the given block are not included!)

   listsinceblock 'previousblockhash'

4. After you finish your job on transactions which are returned from "listsinceblock 'blockhash'", save the last block hash you achieved (and you don't want to search again) into the db or a file.
5. Next time, get the block hash you saved and go to step 3.

This way, you will only monitor new transactions. If you want to update old transactions (number of confirmations etc.), you need to search from the first block every time. In that case, you would just want to use walletnotify or blocknotify.

Answer 4

If you want to just monitor large number public address, you can try blockonomics wallet watcher. It is optimized to watch large number of address. In addition to this:

• You can get email notification of transactions
• Tag Addresses to group them
• See history of transactions

Answer 5

I have written a program in Python3 that allows you to search for any address on bitcoin core whether it belongs to your wallet or not.

Here is the github link:

https://github.com/ORP967/Bitcoin_Core_RPC_par_address

Let me know what you think or if you have any improvements you might have.