Why is/was the Bitcoin Core HWI written in Python?
The biggest factor was that, at the time, every the major hardware wallet vendor provided a Python library to interact with them. Over time, this has remained true - as new hardware wallets have been created, they usually come with Python libraries (e.g. Bitbox02, Coldcard), but this is potentially attributable to the fact that HWI exists.
Additionally, HWI started out as some scripts for experimentation and mostly as a toy project. Given that I am familiar and comfortable with Python, and Python is very convenient for experimentation, it made sense to use it
However, as with any open source project, some decisions made at the beginning for the sole author's convenience can come back to bite us.
What have the challenges been of having the HWI written in Python?
One of the targets for HWI is to have standalone binaries that can be called by other software. This is actually rather difficult to do in Python. It requires the use of other tooling like PyInstaller to produce those binaries. These binaries are rather inefficient as well since they are self extracting archives that contain a full Python interpreter. This has been problematic on some systems, and some downstream consumers of HWI have opted to repackage the binaries differently in order to better suit their users.
Additionally, none of the tooling to create standalone Python binaries can "cross compile". They can only create binaries targeted for the OS and CPU architecture that the program is being run on. While for Windows this can be worked around with WINE, it is not possible to do so for MacOS, thus requiring the builder(s) to spend a large sum of money in acquiring hardware to create those binaries.
Another major issue has also been reproducibility. One of the objectives for HWI was to make the binaries reproducibly built. This has been rather difficult because of how PyInstaller repackages the running Python interpreter, so Python itself needs to be rebuilt reproducibly on each system that we want to support. This is majorly annoying, given that Python itself is not reproducible on all systems.
Python also has some issues around dependencies. A user may be missing a dependency and never notice it until they decide to run some part of the code that requires a dependency. Then it will start mysteriously failing. This can generate several unnecessary support tickets.
With dependencies are also issues with the dependency graph. Since it's really easy to install a dependency with pip, it's very easy to run into issues where a minimal dependency actually isn't so minimal since it pulls in a ton of other dependencies. This makes it hard to audit the entirety of the code that's actually being run. However this is not unique to Python, as NodeJS and rust can have a similar problem.
Lastly, since Python is an interpreted language that is weakly typed, it's really easy to accidentally write bugs related to types or incorrectly assuming that a particular value will be of some type when it is not. Since the code may be syntactically correct, Python will not give an error until the buggy code is executed, which may not always happen. An example of this is using
a[key] for a list. This will raise a
KeyError when it is executed, but otherwise Python won't tell you that it's wrong.
There seems to be interest and rationale(s) to write another HWI in Rust. Would there be a strong rationale to continue to maintain the Python HWI if there was a Rust HWI? MicroPython is a popular Python implementation (optimized to run on microcontrollers) for hardware wallets. Does Rust have an equivalent and is it widely used?
No, there would not be a strong reason to continue to maintain a Python HWI if a compiled language version sufficiently replicated all of its functionality. This doesn't even require things that use MicroPython or other Python interpreters to really change (also I don't think HWI works in MicroPython).
Compiled languages produce machine code binaries. These can be libraries that have an external API which have bindings in every language. This allows such a project to be used with any other language. These can be automatically generated, and there are several projects that can do this pretty well. They can also be handwritten. If a rust HWI reproduced HWI's Python APIs exactly, it could be used as a drop in replacement. There doesn't need to be a rust equivalent for MicroPython, there just needs to be Python bindings for the rust library.
However, it also wouldn't make sense to have HWI run in MicroPython. It's software for the host, not the hardware wallet.