BIP 37 automatic filter update not inserting outputs

Question

I'm having a problem with bloom filters when using the getdata message.

As I understand BIP 37, setting the bloom filter on a connection using the loadfilter message with flag BLOOM_UPDATE_P2PUBKEY_ONLY set, the responding client automatically inserts all outpoints (TXID and output index) in the bloom filter of transactions matching the filter. However, when I attempt to do exactly this, the node only ever sends transactions containing outputs TO the address I am filtering for, never transaction that contain outputs FROM the address I am filtering for.

My approach is as follows:

1. Add the following public key to the bloom filter: 1JbFvyaMyHHmvvjc54h4YARq3uoX2ZZ4Qj (randomly picked)

2. Send the filterload message containing the filter (as seen in the wireshark snippet below).

   Bitcoin protocol
   Packet magic: 0xf9beb4d9
   Command name: filterload
   Payload Length: 261
   Payload checksum: 0x2bb02c45
   Filterload message
       Filter
           Count: 251
           Data: 000000040004010000000100000000000000020000400000...
       nHashFunc: 13
       nTweak: 0x1a689ff4
       nFlags: BLOOM_UPDATE_P2PUBKEY_ONLY (0x02)
   

3. Send the getdata message with the hash of all blocks containing transactions to and from the wallet (which I have identified on a block explorer for the purpose of testing this). The inventories are typed MSG_FILTERED_BLOCK.

The node responds with the following messages in order:

MerkleBlock: 000000000000000001201ab2679db7c87a432967c060596bad0ac093dac94111
Tx:......... 187b00588e8d7844c405da9a15d78dbd7ebb9b2887fb5aace6651719fede923e
Tx:......... dbbca0d839435ac77db65378a9607d17b0aa965f3027d84dcf0ee5115dc8d32f
Tx:......... 17c12f45956082a913c0c862d573cf067fa5ee96ba5b954914371da6bef51f7f
Tx:......... 36cce25d34a5da0f7d69253743eae7bb7c576e5543d0eb16db0f72967edb4bc6
Tx:......... 6ea48c6286b1691cd1ecec271a3f52a3fe099467bb6c926b41ba8942e21d4aa6
MerkleBlock: 0000000000000000049ee67931f6d4970a7d06b14b8aadd406223ab65efe5025
MerkleBlock: 0000000000000000019f3fc2c8da43e8a81373ae189ace4caec6c59969f46ff6
MerkleBlock: 000000000000000000f67dad206798db2e6c5dffec84d213397784be894e66fa

The 5 transactions in the list are correct, as the contain outputs targeting said wallet. However, block 0000000000000000019f3... contains transactions with inputs from said wallet, but the merkleblock message of this block is not followed by these transactions. Shouldn't the node add the outputs of these 5 transactions to the bloom filter, and thereby match the later transactions containing inputs pointing at them?

I have confirmed that the node is in possession of the relevant transactions. If i load the filter manually, adding the outputs myself, this works as intended.

Am I misunderstanding how BIP 37 is used?

Answer 1

BLOOM_UPDATE_P2PUBKEY_ONLY means only update the filter with outpoints when the matched transaction is pay-to-pubkey or bare multisig, not p2pkh or others. I don't know why though, since what you're trying to do is very common.

You may also be interested in this work in Bitcoin XT which allows a filter to be updated with any unconfirmed ancestors of a matching transaction (to better assess its confirmation prospects).