3

Bitcoincore.org and Bitcoin.org will let you download a tarball for bitcoin v0.20.1

When they do it

bitcoind: 4ec74161b2a90293926ae8e20a2efbe952bd23b53aeebf051e6a6285ace18271
bitcoin-0.20.1-x86_64-linux-gnu.tar.gz: 376194f06596ecfa40331167c39bc70c355f960280bd2a645fdbf18f66527397

They also say:

reproducing a binary for yourself will provide you with the highest level of assurance currently available

So, it seems like the current method to do this is the "gitian build" maneuver.

When I do it

bitcoind: c5fb850ed9e6afa0af5653e743084912c9bd71fb233acfe2d72738fc319f2181
bitcoin-0.20.1-x86_64-linux-gnu.tar.gz: 277599356bd2df760832c6636797fe5ea5a5c28d929d53635b685f5ac1e4689b

Went through the gitian process with Ubuntu 20.04 and Debian 10. Both produced the same incorrect? tarball, 2 different setups/ 2 Operating Systems, same tarball.

Both my assert files are uploaded here for Debian and Ubuntu. https://github.com/miketwenty1/images/tree/master/bad_asserts


As a pleb I can think 1 of 2 things..

Something is wrong/off with some of my packages / versions / env / OS / setup OR, All the verifiers are lying and bitcoin's been compromised. Help me narrow down the possibilities.

I would like to go through this exercise properly and reproduce the same binaries.. but unsure of the best way forward.. what are the next steps to verifying the build for v0.20.1? I can't seem to reproduce the tarball advertised on bitcoincore . org and bitcoin . org. I can however make a reproducible build which makes me question a lot of things.

Would love if someone could go through their process from scratch and see if they can produce the same tarballs seen in this repo by these people. https://github.com/bitcoin-core/gitian.sigs/blob/master/0.20.1-linux /*/ bitcoin-core-linux-0.20-build.assert's. And share their complete process with me. After reproducing myself I would be happy to make a current walkthrough on the method and update the Docs, which (links below) are dated and broken for nearly all methods.

NOTE: If this is indeed the defacto way to currently do reproducible builds.. it would be nice to have documentation that is precise and current. I understand Ginx is something being worked on.. but would be nice to have a current doc in the interim for people who wants to verify builds. I would also be very satisfied if someone pointed something I was missing.

reference links:
https://gitian.org/
https://github.com/bitcoin-core/docs/blob/master/gitian-building.md
https://github.com/bitcoin-core/docs/tree/master/gitian-building
https://github.com/devrandom/gitian-builder/blob/master/README.md
https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-build.py

reference github issue
https://github.com/devrandom/gitian-builder/issues/235

5

I've done a rebuild of 0.20.1 and I get the same results that you do. This would indicate that a build dependency has updated to produce slightly different results than the version that was in use at the time of the release. The build dependency versions are pinned unlike the actual software dependencies. IIRC this is common for gitian builds and attempting to rebuild even older releases will result in similar mismatches.

There is ongoing work to move to a different reproducible builds system called guix. This build system will pin the exact dependency versions, along with the option to build the entire toolchain from scratch. This should enable fully reproducible builds that can be replicated at any point in time.


Here is a more in depth explanation of this particular mismatch, copied from my response in this GitHub issue.

If you look at the gitian build results, you will see a file named bitcoin-0.20.1-x86_64-linux-gnu-debug.tar.gz. If you untar this file, there will be *.dbg files, e.g. bitcoind.dbg and bitcoin-qt.dbg. These *.dbg files contain the debug data for their respective binaries, i.e. bitcoind.dbg contains debug data for bitcoind.

To ensure that you use the correct dbg file with the correct binary, gcc embeds a checksum of the dbg file within the binary itself. This means that bitcoind contains a checksum of bitcoind.dbg. This is to prevent attempting to debug bitcoind with another dbg file. For example, if you attempted to tell gdb that the bitcoin-qt.dbg file contained the debugging symbols for bitcoind, it would detect it does not and not attempt to load debugging symbols from bitcoin-qt.dbg.

The debugging symbols are initially compiled into the bitcoind binary, but later during the gitian build these are removed and placed into the bitcoind.dbg file. However because they are compiled into bitcoind initially, the debugging symbols have an effect on the build id that gcc embeds into the binary. The build ID is a hash of the compiled binary, including the debugging data.

So the end result is that the published binary contains two commitments to the debug symbols, but does not actually contain them itself.

You can read more about these separate debug files here: https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html

The crux of this issue is that the debug symbols from recent gitian builds differ from the debug symbols that were generated for the original release. This means that both the build ID and the debug symbol checksum that we find inside of bitcoind are different. This then results in the bitcoind hashes being different (as well as all of the other binaries). And of course that causes the tarfile hashes to be different.


Here is the diff of the binaries that diffoscope generates:

--- bitcoind
+++ /mnt/archive/bitcoin/bitcoin-binaries/0.20.1/bitcoin-0.20.1/bin/bitcoind
├── readelf --wide --notes {}
│ @@ -1,15 +1,15 @@
│  
│  Displaying notes found in: .note.ABI-tag
│    Owner                Data size     Description
│    GNU                  0x00000010    NT_GNU_ABI_TAG (ABI version tag)        OS: Linux, ABI: 3.2.0
│  
│  Displaying notes found in: .note.gnu.build-id
│    Owner                Data size     Description
│ -  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 6b464617f7f91fd270ac86f43ef4a58eeeedff19
│ +  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 3a439a31a5157ff7052ed310050df5643a02ea3f
│  
│  Displaying notes found in: .note.stapsdt
│    Owner                Data size     Description
│    stapsdt              0x00000036    NT_STAPSDT (SystemTap probe descriptors)        Provider: libstdcxx
│      Name: throw
│      Location: 0x00000000006e550d, Base: 0x000000000086d140, Semaphore: 0x0000000000000000
│      Arguments: 8@%rdi 8@%rsi
├── readelf --wide --decompress --hex-dump=.gnu_debuglink {}
│ @@ -1,5 +1,5 @@
│  
│  Hex dump of section '.gnu_debuglink':
│    0x00000000 62697463 6f696e64 2e646267 00000000 bitcoind.dbg....
│ -  0x00000010 b25ceebb                            .\..
│ +  0x00000010 114d519d                            .MQ.

As you can see, there are only two differences here, one in the build ID, and one in the .gnu_debuglink section. From the documentation I linked earlier, we can see that this .gnu_debuglink section has the first line is the debug filename followed by enough 0 bytes to pad to a 4 byte boundary. The second line is the 4 byte CRC checksum. And it is this CRC checksum that differs.

So why do the debug symbols differ here? Again, diffoscope can help us a bit.

--- bitcoind.dbg
+++ /mnt/archive/bitcoin/bitcoin-binaries/0.20.1/bitcoin-0.20.1/bin/bitcoind.dbg
├── readelf --wide --notes {}
│ @@ -1,15 +1,15 @@
│  
│  Displaying notes found in: .note.ABI-tag
│    Owner                Data size     Description
│    GNU                  0x00000010    NT_GNU_ABI_TAG (ABI version tag)        OS: Linux, ABI: 3.2.0
│  
│  Displaying notes found in: .note.gnu.build-id
│    Owner                Data size     Description
│ -  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 6b464617f7f91fd270ac86f43ef4a58eeeedff19
│ +  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 3a439a31a5157ff7052ed310050df5643a02ea3f
│  
│  Displaying notes found in: .note.stapsdt
│    Owner                Data size     Description
│    stapsdt              0x00000036    NT_STAPSDT (SystemTap probe descriptors)        Provider: libstdcxx
│      Name: throw
│      Location: 0x00000000006e550d, Base: 0x000000000086d140, Semaphore: 0x0000000000000000
│      Arguments: 8@%rdi 8@%rsi
├── readelf --wide --debug-dump=info {}
│┄ error from `readelf --wide --debug-dump=info {}`:
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│ @@ -85052,36 +85052,36 @@
│      <29607>   DW_AT_decl_line   : 124
│      <29608>   DW_AT_decl_column : 16
│      <29609>   DW_AT_type        : <0x28761>
│      <2960d>   DW_AT_data_member_location: 12
│   <2><2960e>: Abbrev Number: 30 (DW_TAG_member)
│      <2960f>   DW_AT_name        : (indirect string, offset: 0x13aaf): __kind
│      <29613>   DW_AT_decl_file   : 108
│ -    <29614>   DW_AT_decl_line   : 148
│ +    <29614>   DW_AT_decl_line   : 128
│      <29615>   DW_AT_decl_column : 7
│      <29616>   DW_AT_type        : <0x287a6>
│      <2961a>   DW_AT_data_member_location: 16
│   <2><2961b>: Abbrev Number: 30 (DW_TAG_member)
│      <2961c>   DW_AT_name        : (indirect string, offset: 0x7a535): __spins
│      <29620>   DW_AT_decl_file   : 108
│ -    <29621>   DW_AT_decl_line   : 154
│ +    <29621>   DW_AT_decl_line   : 134
│      <29622>   DW_AT_decl_column : 3
│      <29623>   DW_AT_type        : <0x2879a>
│      <29627>   DW_AT_data_member_location: 20
...

There's a lot more output that I haven't included because it's pretty much all of the same.

Now this isn't terribly helpful, but we can see that for a bunch of functions, the line number for that function differs by 20 lines.

To get some more information, I used dwarfdump. This is what it says for the new build for the two functions I show in diffoscope (the functions are __kind and __spins).

0x0002960e:     DW_TAG_member
                  DW_AT_name    ("__kind")
                  DW_AT_decl_file       ("/usr/include/x86_64-linux-gnu/bits/thread-shared-types.h")
                  DW_AT_decl_line       (148)
                  DW_AT_decl_column     (0x07)
                  DW_AT_type    (0x000287a6 "int")
                  DW_AT_data_member_location    (0x10)

0x0002961b:     DW_TAG_member
                  DW_AT_name    ("__spins")
                  DW_AT_decl_file       ("/usr/include/x86_64-linux-gnu/bits/thread-shared-types.h")
                  DW_AT_decl_line       (154)
                  DW_AT_decl_column     (0x03)
                  DW_AT_type    (0x0002879a "short int")
                  DW_AT_data_member_location    (0x14)

As you can see by the given file name, these functions come from libraries installed to the system. These appear to be headers for gcc's implementation of the c++ stdlib.


So what's happened is that libstdc++ has updated in Ubuntu. Whatever updates hapened have moved some code in some header files that Bitcoin Core includes in its use of the c++ stdlib. In turn, compiling with those updated headers results in different debug symbols because function declarations have moved in those header files. This then results in gcc computing a different build ID and a different CRC checksum for the debug symbols. This lastly results in the final binaries being slightly different, which causes the hashes to mismatch.

| improve this answer | |
  • I'm curious as to the choice of guix over Nix - do you have a link to any discussion on irc, github, etc. on why guix was chosen? – Raghav Sood Nov 15 at 10:37
  • @RaghavSood From github.com/bitcoin/bitcoin/pull/15277: "Guix is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on bootstrappability and reproducibility which are attractive for security-sensitive projects like bitcoin." I don't think there has been much discussion about guix vs nix, just that guix is what someone has put the effort into doing, so it's what we get. – Andrew Chow Nov 15 at 17:36
  • Thanks for the detailed info. If it's interesting I have cmp output here comparing both bitcoind files. github.com/devrandom/gitian-builder/issues/235 Luke-jr let me know the files it looked like were in question were: .note.gnu.build-id .gnu_debuglink @AndrewChow you think this is a design flaw or working as intended? – Michael Tidwell Nov 15 at 22:44
  • It is a design flaw of the gitian system, but the gitian system is also working as intended. – Andrew Chow Nov 15 at 23:10
1

As long as the underlying OS image is susceptible to changes, you will not get a reproducible build system that guarantees reproducibility over a long period of time. If the OS can update compiler toolchains, libc versions and whatever tools are used during a gitian build, changes in the future to the output are expected. Since this is the case since gitian is ubuntu based, and patches are still shipped to older ubuntu versions, this is to be expected after some time.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.