Bitcoin Core stalled peer logic during initial block download

Question

I'm trying to track down the cause of unexpected (20% to 85%) IBD slowdowns between my 2020 and 2021 annual performance tests.

I experienced this slowdown with multiple implementations, so I don't think the issue is Bitcoin Core specific, though I do want to understand one potential cause with regard to how Bitcoin Core handles stalled peers.

Several developers have told me that if a peer doesn't respond to a request for a block, a Bitcoin Core node will disconnect from it after 15 minutes. I see here that @pieter-wuille said something similar on a related question.

However, when I look at the "stalled block download" logic in Bitcoin Core it seems to be using a timeout of only 2 seconds.

I have several full debug logs from different syncs I ran and from looking at the logs, these stalled peers appear to get disconnected after 2 seconds; even the ones that are serving headers rather than blocks. Example log:

2021-10-28T13:00:43Z New outbound peer connected: version: 70016, blocks=707043, peer=9 (outbound-full-relay)
2021-10-28T13:00:45Z Ignoring getheaders from peer=9 because node is in initial block download
2021-10-28T13:00:48Z more getheaders (338000) to end to peer=9 (startheight:707043)
2021-10-28T13:00:48Z Protecting outbound peer=9 from eviction
2021-10-28T13:00:50Z more getheaders (340000) to end to peer=9 (startheight:707043)
2021-10-28T13:00:52Z more getheaders (368000) to end to peer=9 (startheight:707043)
2021-10-28T13:00:53Z more getheaders (342000) to end to peer=9 (startheight:707043)
2021-10-28T13:00:54Z more getheaders (370000) to end to peer=9 (startheight:707043)
2021-10-28T13:00:56Z more getheaders (344000) to end to peer=9 (startheight:707043)
2021-10-28T13:00:59Z more getheaders (372000) to end to peer=9 (startheight:707043)
2021-10-28T13:01:00Z more getheaders (346000) to end to peer=9 (startheight:707043)
2021-10-28T13:01:02Z more getheaders (374000) to end to peer=9 (startheight:707043)
2021-10-28T13:01:03Z more getheaders (348000) to end to peer=9 (startheight:707043)
2021-10-28T13:01:04Z more getheaders (437999) to end to peer=9 (startheight:707043)
2021-10-28T13:01:07Z more getheaders (376000) to end to peer=9 (startheight:707043)
2021-10-28T13:01:09Z more getheaders (350000) to end to peer=9 (startheight:707043)
2021-10-28T13:01:11Z more getheaders (439999) to end to peer=9 (startheight:707043)
2021-10-28T13:01:14Z more getheaders (378000) to end to peer=9 (startheight:707043)
2021-10-28T13:01:16Z more getheaders (352000) to end to peer=9 (startheight:707043)
2021-10-28T13:01:18Z more getheaders (489999) to end to peer=9 (startheight:707043)
2021-10-28T13:01:19Z more getheaders (441999) to end to peer=9 (startheight:707043)
2021-10-28T13:01:20Z more getheaders (380000) to end to peer=9 (startheight:707043)
2021-10-28T13:01:22Z more getheaders (354000) to end to peer=9 (startheight:707043)
2021-10-28T13:01:23Z initial getheaders (707100) to peer=9 (startheight:707043)
2021-10-28T13:01:24Z more getheaders (491999) to end to peer=9 (startheight:707043)
2021-10-28T13:01:25Z more getheaders (443999) to end to peer=9 (startheight:707043)
2021-10-28T13:01:26Z Peer=9 is stalling block download, disconnecting

In short, I'm trying to better understand what the worst case scenario of slow / malicious peers could be with regard to impacting IBD performance and how I might diagnose which peers are causing the sync slowdown.

Answer 1

During IBD, there is a "moving window" of 1024 blocks ahead from the current height, from which blocks can be requested from peers for parallel download. When a peer has an open slot for a block request (there is a maximum of 16 blocks assigned to a peer at a time), it is requested to provide a block from this window.

A stalling situation occurs, when there is no such block available to assign to a peer, because all blocks from the current moving window have either been already downloaded or are "in flight", i.e. have been requested but not received. This can happen if some peer sends blocks too slowly or not at all.

In this case the "stalling" peer is determined - the peer that was asked to provide the block that would help us extend our current tip / move the window forward, but hasn't delivered. This peer gets an additional period of 2 seconds (BLOCK_STALLING_TIMEOUT), and if it fails to provide a block, it will get disconnected for stalling and the "bottleneck" block that prevents the window from moving will be requested from another peer.

[Edit]: The requested blocks of the 1024 block window are usually distributed over 10 outbound peers (among them 2 block-relay-only peers) as per the current default. (Blocks aren't requested from inbound peer while in IBD as long as outbound peers exist, although there is currently a proposal to soften this rule: Bitcoin#24171.) This means that a given peer must be considerable slower compared to other peers for a stalling situation to occur.