1

I'm trying to understand at a high level, the local network aspects of calling the bitcoin core api via bitcoinlib in python

So the context around this question is that I wanted to work on some python (for which I have very minimal experience or understanding) so I sat down to attempt to write a blockchain indexer. I run a full node and figured I would see what I could do with the api.

Bitcoinlib offers a class called AuthServiceProxy. You pass the local api url & port to this, it looks like this: rpc_connection = AuthServiceProxy("http://user:[email protected]:8332")

I dug a little bit into this class to find that it's using httlib.HTTPConnection to open the connection itself.

        if connection:
            # Callables re-use the connection of the original proxy
            self.__conn = connection
        elif self.__url.scheme == 'https':
            self.__conn = httplib.HTTPSConnection(self.__url.hostname, port,
                                                  timeout=timeout)
        else:
            self.__conn = httplib.HTTPConnection(self.__url.hostname, port,
                                                 timeout=timeout)

The first thing I attempted to do was to write a loop that would iterate over all the blocks stored locally to create an array out of the hashes of each block.

        while block_counter < total_blocks:
            rpc_connection = rpcConnector.rpcConnect()
            block_hash = rpc_connection.getblockhash(total_blocks)

this is essentially how it is getting called. The rpcConnector is just that AuthServiceProxy call essentially.

So I ran this and printed the output of each block hash just to see how it was working. It worked up until about the 16200th iteration. Then it fails with the error message "[WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted". It fails at this mark each time I run it.

I thought that maybe the connections weren't being closed out correctly so I tried a bunch of different things to try and close the connection within each loop iteration Nothing worked.

The last thing I tried was this: rpc_connection._AuthServiceProxy__conn.close(). I figured since the AuthServiceProxy was just using HTTPConnection I could just call the .close() I stepped through the code to see that it was in fact getting into the .close() method but it wasn't ever hitting close.

    def close(self):
        """Close the connection to the HTTP server."""
        self.__state = _CS_IDLE
        try:
            sock = self.sock
            if sock:
                self.sock = None
                sock.close()   # close it manually... there may be other refs
        finally:
            response = self.__response
            if response:
                self.__response = None
                response.close()

both the Sock and Response were None.

So maybe this is a networking question but it seems like when I call the API it's not keeping a connection open, so if that is the case then my question is: how am I using up the addresses, and why is it always stopping around 16200?

and I guess to close it out, how can I resolve this issue?

2
  • Adding a comment here, I'm closer to narrowing down the problem itself. Navigate to HKEY_LOCAL_MACHINE\SYSTEM \CurrentControlSet\Services\ TCPIP\Parameters . Create a new key for REG_DWORD, and specify the name TcpTimedWaitDelay. Type the minimum decimal value 30 (which is 0x0000001e in hexadecimal). This got me to about 20k iterations. Jan 29 at 21:23
  • I think the ultimate issue is that too many ports are in TIME_WAIT status. The default value is 240 seconds after closure, I changed it to 30 seconds and it seemed to make it a bit better. I don't know how much more I can tune this before something breaks. Any ideas on a potential code solution? Jan 29 at 21:26

1 Answer 1

1

This seems to be an issue with how Windows deals with ports. According to https://help.socketlabs.com/docs/how-to-fix-error-only-one-usage-of-each-socket-address-protocolnetwork-addressport-is-normally-permitted, when a port is released, Windows does not make it immediately available for reuse. Instead it waits for TIME_WAIT which has a default of 240 seconds.

Bitcoin Core's RPC is a one call per connection type of service, so once you connect to it and receive the response, you should be disconnecting. As each outbound connection to the server consumes one port, when the call is done and that port is released, Windows will continue to block use of that port for TIME_WAIT. As you are making many calls very quickly one after the other, you run into a situation where all available ports are in TIME_WAIT and thus you get the error.

There are a few solutions. As mentioned in the linked article, the default number of available ports is just under 4000 (the lower 1024 are reserved, this is standard across all operating systems). However ports are 16-bit ints, so you can actually have up to 65536 ports, so you could increase Windows port range to the actual maximum. You could also try decreasing TIME_WAIT, but that could have negative side effects.

However, you could also do batched RPC requests. This is where a single connection to the RPC server contains multiple RPC requests, and those are all processed sequentially and their results returned in the same response. The results will be as if you called the RPC once per request, but it will reduce the number of actual connections, and thus ports, used by the OS to send the requests. AuthServiceProxy has a batch_() function that you can use for this. You pass it array of tuples each containing the RPC name and the positional parameters for it, and it will construct a single batched request and return an array of results.

2
  • I assume batch will still run into the same issue unless it's making multiple requests on the same port? I'll do some experimenting today, I'll try to just catch the error message and wait 30 seconds and see how that goes Jan 30 at 12:24
  • Batching makes multiple requests within one connection. All of those requests are sent at the same time from the same port, and you get all of their responses at the same time.
    – Ava Chow
    Jan 30 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.