What's the purpose of using Guix within Gitian? Doesn't that reintroduce dependencies and security concerns?

Question

What's the purpose of using Guix within Gitian? Doesn't that reintroduce dependencies and security concerns that were the purpose of transitioning from Gitian to Guix?

My understanding was that Gitian was reproducible but not bootstrappable and that Guix is an improvement to bootstrappable and reproducible?

This question was asked by robertspigler on IRC.

Answer 1

Gitian achieved reproducibility through the use of a virtual machine to isolate the toolchain used for building. This means that any and all unaudited and untrusted third party binaries execute in a contained environment where they are less likely to pollute and cause damage to the host system.

On the other hand, Guix is intended to be run directly on the host itself, so if there are any unaudited and untrusted third party binaries executed, they could cause damage to the host.

So some developers, notably luke-jr, want to perform Guix builds within a virtual machine to minimize this risk. This could be achieved by installing Guix inside of a virtual machine, but we already have Gitian and the infrastructure around it, so it is also possible to achieve this by running Guix within Gitian and avoid the need for such developers to figure out how to install Guix in a VM, and maintain the VM.

Furthermore, many people who have participated in the reproducible builds process may not have Guix setup yet. By doing Guix inside of Gitian, it will be possible for those people to participate in the new release process with minimal changes. Providing an easy way to transition from Gitian to Guix is the primary motivation for enabling Guix within Gitian.

Of course the recommended method is still to use Guix directly rather than through Gitian. There is a significant performance penalty for using Guix inside of Gitian. Additionally, running Guix directly allows for issues to be more easily debugged and worked through as the toolchain and extra binaries built will still be available to the builder.

Answer 2

fanquake answered this on IRC:

There are a number of benefits that come with using Guix as a release build environment (or just as a build system in general).

Using Gitian, you have to pick a Linux distro to build on. This defines your toolchains, available packages, the glibc your building against etc.

This also means you're subject to choices of the upstream package maintainers. They control how/when package versions are updated, what patches are applied to them, how they are compiled etc. This has downsides.

As an example, our security-check tests rely on a number of -no-* options being present in mingw-w64 ld. These options were patched into the ld in the Ubuntu Bionic binutils-mingw-w64 package, however, the binutils maintainer chose to stop doing that in Ubuntu Focal, which meant our security-check tests would not have been able to run (they have since decided to re-patch them in Hirsute). See #18629 for more details.

In Guix, because we're in full control of our toolchains, we don't have to worry about issues like this, and can just directly apply the patches we want to use, and this is exactly what we've done in #22381, where we started patching the -no-* linker options into our mingw-w64 toolchain. Compiler / linker options or defaults that we may rely on, can no-longer be randomly changing out from underneath us.

To that, some might say "well just don't change the Ubuntu base image, and things won't change". To that, I'd say, I don't want the project to be in a position where we are "stuck", and can't update our base image to use new tools (i.e compilers), because by doing so we'll just break something else, like our security tests.

It would seem smarter to just remove the potential for those kinds of problem entirely, by constructing and using exactly the release build environment we want.

"Well why not just patch / change things in gitian" - I wont say much on this, except for, the gitian environment is not at all setup to do the same kind of patching that we can fairly trivially achieve in Guix. Trying to patch and compile, for example, the mingw-w64 toolchain in a gitian descriptor, would also just turn into a terrible mess.

The same can be said about trying to gitian build anything using a glibc different than what is already available on that version of Ubuntu.

This has backward-compatibility implications, as the version of glibc you're building against, essentially determines your runtime glibc compatiblity. However this can be extended by using the sorts of "workarounds" that we currently have in our glibc-back-compat code.

Now, similar to the situation where you might want to use a newer Ubuntu release (for any number of reasons), doing so will mean that you have to build against a newer glibc (you just get whatever version comes with that release of Ubuntu).

This means that as you use newer versions of glibc, the number of "workarounds" you need to maintain backwards compatibility pile up, get continually more complicated, and even start to leak out of Bitcoin Core code, and into our dependency system. See all the PRs linked in this comment: https://github.com/bitcoin/bitcoin/pull/22418#issuecomment-876379846.

Changes like that leaking into depends are bad, because now normals builders, using depends, are now being subject to the side-effects of patches that are only really necessary in our release build environment. Maybe you'd argue that in that case we should only apply the depends patches when building releases, however then you've got a situation where release builds are even more divergent from "normal"/developer builds, and this means either maintaining an even more complex CI / testing routines, or they just end up less tested (guess which one is more likely to happen).

If that all just sounds like a complicated mess, it's basically because it is. However the solution, using Guix, is actually pretty straight forward.

When we are in complete control of our release environment, we can pick exactly the version of glibc we want to use (even at a per-HOST level), something that we definitely couldn't achieve, in any sort of straight forward fashion, if at all, using gitian.

This is something we've done recently, in #22365. We are now building using glibc 2.27 for the RISC-V HOST, and using glibc 2.24 for all others, while maintaining runtime compatibility with glibc 2.17. You'll note this this is achieved without needing to use any of the work arounds from our glibc-back-compat code (#22405), or any of the other PRs / changes mentioned in the comment above.

These are just two very practical benefits that using Guix provides (there are more), which ultimately all boil down to us being in much greater control of our release build environment. Something I am very happy about, and I think makes a lot of sense for a project like Bitcoin Core.

The fact that there are also many people in the Guix space actively working on bootstrapability is just another big bonus, and even if that doesn't work exactly like some may want it too right now, it is rapidly being improved, month on month.

Guix is also a much more likely pathway to fully bootstrapable Bitcoin Core builds that what gitian could ever provide.

dongcarl stated:

I don't think we're going to run Guix inside Gitian for the releases, but personally I think people are free to do whatever they think make sense if they're committed to maintaining it

luke-jr added:

You don't lose those Guix benefits by running it inside gitian; gitian just provides the isolation needed to avoid regressing on security