This is a report and summary of my understanding for the Reproducible Builds Hackfest during Flock 2023 in Cork.[1][2]
The goal is to have “reproducible builds” for rpms in Fedora and later the rest of the ecosystem. This would allow our users to be able to independently verify that the rpms have not been tampered with (either maliciously or by unreliable hardware): someone can do an independent rebuild of a package and confirm that they get identical binaries when building with the same versions of the compiler and other tools.
Other distributions, in particular Debian, are more advanced in this regard. For Debian, reproducible builds are crucial, because they allow[3] maintainers to generate source packages locally, possibly without any version control, and even to upload locally-built packages for distribution to users. Trust in the contents of both source and binary packages is low.
In Fedora, all packages that are distributed to users are built in the centralized, strongly controlled infrastructure. All source rpms are built from “dist-git”: a git repository which contains the build “recipe” and a cryptographic hash of package sources, so it is relatively easy to verify what changed between package versions, what “inputs” went into a particular source package, and in what environment the binary packages were built.
Because of this strong control over the build process, reproducible builds haven’t been a priority in Fedora. But they would still be a nice feature. For example, let’s image what would happen if the hardware was broken: overheated memory on one builder does bit flips, corrupting output rpms in a subtle way. If we are able to redo any build, it would be fairly easy to verify whether this is the case. If we had a process of doing “shadow” reproducible builds for all packages, we could even detect such cases before any bug reports from users. Similarly, we could detect if a builder machine was compromised in some sort of a supply-chain attack to inject rogue code into the rpms.
Current situation in Fedora
We start with the definition of “reproducible builds” in Debian’s reproducible-builds.org initiative:
A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
In the Fedora ecosystem, we cannot achieve reproducibility by this definition. A fully identical result cannot be achieved, because rpm packages are distributed after signing, with the signature is embedded in the rpm (while Debian uses detached signatures). A rebuild of a package (as distributed to users) will always differ at least by this missing signature.[4][5][6]
In addition, rpm builds inject some information about the build time and place into the outputs: BUILDTIME and BUILDHOST in the rpm header. We opened an issue to allow those to be overriden[7], but right now rpm doesn’t allow this, and the general consensus is that this information is useful and it is not desirable to override it.
We need a different definition (changed parts italicized):
A build is reproducible if given the same source code, build environment and build instructions, and metadata from the build artifacts, any party can recreate copies of the artifacts that are identical except for the signatures and parts of metadata.
I think would still be useful to users, because it would allow them to verify that the build that was done in the official build system is trustworthy, by doing a comparison that ignores the short list of fields which are known to vary.
Rebuilds in koji
How could this work in detail in our build system? The simplest option is to take the source rpm (srpm) as the input. This contains the source code and build instructions.[8]
We also need the same build environment: identical versions of all the programs and data that was available on the system where the rpm was built, and the same configuration.
Luckily, official Fedora rpms are built in a well-defined environment: koji, which uses mock to perform builds. mock uses systemd-nspawn to start a build in an isolated container, which is created using dnf which installs a bunch of packages. Thus, to recreate the build environment we just need the get the list of rpms that were installed into the build environment, and repeat the build using mock. koji records the list of rpms and makes it available via the getBuildrootListing() call. Also, for any build we can download the srpm.
We also need the “configuration”, which means a set of macro definitions. I’m told that koji records this too[9], but I wasn’t able to figure out how to access this data for some specific build.
Are Fedora packages reproducible in this weaker definition? It is hard to answer this question without actually doing a mass re-rebuild. Limited experiments indicate that things are not bad, about which more below, but the true answer is that right now nobody knows.
Thus, during the hackfest we decided to create a service which would listen to notifications about builds and do continuous rebuilds of (some subset of) packages. For a given rebuild, if the results are not “identical” using the definition above, it would use diffoscope to do a comparison[10]. Diffoscope can even generate a fancy HTML page. Those pages would be made available online so that packagers (or anyone interested) can figure out why builds are not reproducible. Ideally, we would have a website similar to Debian’s Overview of various statistics about reproducible builds.
Initial results
I started working on a rebuilder which takes a package name, downloads the srpm from koji and build metadata from koji, downloads all rpms present in the buildroot listing, creates a mock buildroot with this package set, and repeats the build. We need to download packages directly from koji, because we want specific versions of packages. The versions that are distributed via the mirror network are often either older or newer. It would be reasonable to opportunistically download the packages from the mirror network, but I haven’t implemented that yet.[11]
A rebuild of a noarch package:
$ diffoscope original/python3-referencing-0.30.2-1.fc40.noarch.rpm rebuilt/python3-referencing-0.30.2-1.fc40.noarch.rpm --- original/python3-referencing-0.30.2-1.fc40.noarch.rpm +++ rebuild/python3-referencing-0.30.2-1.fc40.noarch.rpm ├── header │ @@ -1,35 +1,35 @@ │ -HEADERIMMUTABLE: 00000040000024810000003f00000007000024710000001000000064000000080000000000000001000003e8000000060000 │ +HEADERIMMUTABLE: 00000040000026390000003f00000007000026290000001000000064000000080000000000000001000003e8000000060000 │ @@ -1,35 +1,35 @@ │ -SIGSIZE: 69207 │ -SIGMD5: 5b0dca488816d0edd42573c5276fc856 │ -SHA1HEADER: 3bf5a9f231f640b1dd476fe0619a137a1760177b │ -SHA256HEADER: bc0da41ce4efb98977cfd02dd5acd5672a85face73031bf69dc813e12df59698 │ +SIGSIZE: 69647 │ +SIGMD5: 8d8d87c797386f2e15fccf4e7a932a8a │ +SHA1HEADER: e6327c0aa283bb72005454efa2ebe1c18e15a574 │ +SHA256HEADER: 48200479463c4ef0179e374b5a29be1856c3a834792eb066435fd31f595caed4 │ NAME: python3-referencing │ VERSION: 0.30.2 │ RELEASE: 1.fc40 │ -BUILDTIME: 1691828680 │ +BUILDTIME: 1691937521 │ BUILDHOST: buildvm-x86-31.iad2.fedoraproject.org │ SIZE: 317872 │ DISTRIBUTION: Fedora Project │ VENDOR: Fedora Project │ LICENSE: MIT │ PACKAGER: Fedora Project │ GROUP: Unspecified │ -COOKIE: buildvm-x86-31.iad2.fedoraproject.org 1691828680 │ +COOKIE: buildvm-x86-31.iad2.fedoraproject.org 1691937521 │ @@ -1072,15 +1081,19 @@ │ -OPTFLAGS: -O2 -g -march=x86-64-v4 │ +OPTFLAGS: -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain- │ +wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 │ +-Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong │ +-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables │ +-fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer │ PAYLOADFORMAT: cpio │ PAYLOADCOMPRESSOR: zstd │ PAYLOADFLAGS: 19 │ PLATFORM: noarch-redhat-linux-gnu │ @@ -1339,15 +1352,15 @@ │ -SOURCEPKGID: 79dfd37bb1e130737dafcee9aaf852a9 │ +SOURCEPKGID: cfe5d663403995705b900598914725f4 │ FILEDIGESTALGO: 8 │ BUGURL: https://bugz.fedoraproject.org/python-referencing │ ENCODING: utf-8 │ PAYLOADDIGEST: │ - 1478fc6cbbaacdd225d33590ef030c34fbe67a25a57bbe5e2ed7cc7d91df2328 ├── filetype from file(1) │ @@ -1 +1 @@ │ -RPM v3.0 bin noarch python3-referencing-0.30.2-1.fc40 │ +RPM v3.0 bin i386/x86_64 python3-referencing-0.30.2-1.fc40
The good new is that the contents are bit-for-bit identical (PAYLOADDIGEST field is unchanged and diffoscope shows no output). The bad news is that the header differers in a bunch of fields.[12]
For this build, BUILDHOST was overridden to match the original build. As mentioned above, there is no mechanism to override BUILDTIME, so that field, as well as the COOKIE field which is a concatenation of BUILDHOST and BUILDTIME, are different in the rebuilt package. Similarly, SOURCEPACKAGEID is different because BUILDTIME and other fields were different in the rebuilt srpm which was used to build this binary rpm. The case of OPTFLAGS is interesting too. The version from koji has -march=x86-64-v4, which might be a bug[13]. We asked rpm maintainers to drop the OPTFLAGS field from the header (it is completely useless, particularly in noarch packages), but this idea was swiftly rejected.[14]
A rebuild of an archful package:
$ diffoscope original/valgrind-3.21.0-8.fc39.x86_64.rpm rebuilt/valgrind-3.21.0-8.fc39.x86_64.rpm --- original/valgrind-3.21.0-8.fc39.x86_64.rpm +++ rebuilt/valgrind-3.21.0-8.fc39.x86_64.rpm ├── header │ @@ -1,454 +1,453 @@ │ -HEADERIMMUTABLE: 000000420000ae450000003f000000070000ae350000001000000064000000080000000000000001000003e8000000060000 │ +HEADERIMMUTABLE: 000000410000ae350000003f000000070000ae250000001000000064000000080000000000000001000003e8000000060000 │ -SIGSIZE: 5048344 │ -SIGMD5: ddd521921b1d2e3e05b763c726e01193 │ -SHA1HEADER: dac581e7868644c3e1ff75f1b188502be164b601 │ -SHA256HEADER: e231673679b1f58aa28d89b4da45b33dd2771a3f68bcc8f21b191d85d2ed5eae │ +SIGSIZE: 5048315 │ +SIGMD5: 9fa13d665b3dcef950be3edae88424bb │ +SHA1HEADER: 7f7674e6362fb7cef90183ec5ccf7013d3a90c66 │ +SHA256HEADER: 7345c9edd3c2c77747a9fb687792cdef879bc7c93176aecebee0f0c9666d1ed2 │ NAME: valgrind │ VERSION: 3.21.0 │ RELEASE: 8.fc39 │ EPOCH: 1 │ -BUILDTIME: 1690047023 │ +BUILDTIME: 1691864859 │ BUILDHOST: buildvm-x86-09.iad2.fedoraproject.org │ SIZE: 30133398 │ ENCODING: utf-8 │ PAYLOADDIGEST: │ - - 4743d82adffc81245d27f1555f9c7adedd26d870e3c0b93e729df08fa286dd8c │ + - 60674af1abf6f4b2a169466d94cc22d7a8a29e964bd7aff3cc2dee717b2ce7ca │ PAYLOADDIGESTALGO: 8 │ PAYLOADDIGESTALT: │ - - 4dd91f04d1839c00953dd182622f67937c28cb0f28cac4076e55a81223d16ffe │ + - c2e1062a9eaaf249b07ebcfddc53d91e2e24cb505f30c07d392eab14bccb8069 ├── content │ ├── ./usr/libexec/valgrind/cachegrind-amd64-linux │ │┄ File has been modified after NT_GNU_BUILD_ID has been applied. │ │ ├── readelf --wide --decompress --hex-dump=.gnu_debuglink {} │ │ │ @@ -1,7 +1,7 @@ │ │ │ │ │ │ Hex dump of section '.gnu_debuglink': │ │ │ 0x00000000 63616368 65677269 6e642d61 6d643634 cachegrind-amd64 │ │ │ 0x00000010 2d6c696e 75782d33 2e32312e 302d382e -linux-3.21.0-8. │ │ │ 0x00000020 66633339 2e783836 5f36342e 64656275 fc39.x86_64.debu │ │ │ - 0x00000030 67000000 919eb441 g......A │ │ │ + 0x00000030 67000000 3fdf2fd9 g...?./. │ ├── ./usr/libexec/valgrind/callgrind-amd64-linux │ │┄ File has been modified after NT_GNU_BUILD_ID has been applied. │ │ ├── readelf --wide --decompress --hex-dump=.gnu_debuglink {} │ │ │ @@ -1,7 +1,7 @@ │ │ │ │ │ │ Hex dump of section '.gnu_debuglink': │ │ │ 0x00000000 63616c6c 6772696e 642d616d 6436342d callgrind-amd64- │ │ │ 0x00000010 6c696e75 782d332e 32312e30 2d382e66 linux-3.21.0-8.f │ │ │ 0x00000020 6333392e 7838365f 36342e64 65627567 c39.x86_64.debug │ │ │ - 0x00000030 00000000 99189dcf ........ │ │ │ + 0x00000030 00000000 2276f857 ...."v.W
The situation with the header is similar. It differs in the places where we expect it to. Interestingly, the contents differ to. The .gnudebuglink section differs in each of the binaries under /usr/libexec/valgrind.[15] This is the kind of bug that we’ll need to fix. Once the payload is identical, the difference should be down to the same list of fields in the rpm header as in the noarch case.
Discussion
I think that once we accept that we’ll need to filter out the (missing) signature when doing rpm comparisons, the fact that some other fields will also be different is not a big issue. It is easier to compare files looking for bit-by-bit identity, but if we have to use a custom comparator to ignore one field (e.g. SIGPGP), having it ignore a few other fields is not a big issue (BUILDTIME, BUILDHOST, COOKIE, OPTFLAGS). I still think being able to override those fields is important, but it’s not a blocker.
For the two packages shown above and a few others I tested, contents are either identical or very close to being identical. This is not an accident! The efforts for reproducibility in other distributions resulted in fixes to various tools that were pushed upstream, and Fedora is benefiting from this work. We also did some preparatory work to set $SOURCE_DATE_EPOCH to the last modification of the changelog[16] and to clamp the modification times of files to it[17]. Together, those changes removed a lot of trivial differences that we’d otherwise have to fight with.
Another way in which we have things easy is that official packages are always built in mock, which means that the build paths are constant. Various tools embed those paths in the outputs. In Debian, many of the outstanding reproducibility issues are caused by this. Embedding of the paths is not great, because the paths are meaningless outside of the build environment, but at least it doesn’t impact reproducibility in our case.
The history of the effort in Debian shows that firstly there are initial infrastructure and tooling issues which impact every build (for example the file modification timestamps), and secondly there are issues which impact broad classes of packages (I expect that the issue with .gnudebuglink will fall into this category), and thirdly there is a long tail of issues which impact ever smaller numbers of packages[18]. I think we’re already between the second and third of those stages, which is great. If we manage to set up a rebuild service, we’ll know more.
Feel free to reply in this thread, or to join us in the Flock discussion room for reproducible builds.
I hope that it is OK to put this text up as a post on discussion.fp.o and more specifically, in the Project Discussion category. Before discussion.fp.o was created, I would have made this into a blog story or a post on fedora-devel. I think that a post here has advantages over a blog post: it will be more discoverable, and it will be easier for other participants in the workshop and anyone reading this to provide feedback. I think it also has advantages over a mailing list post: it is easier to include pictures or formatted output, and it is possible to edit the post after initial publication. I expect that there’ll corrections to make, so the ability to edit is important. ↩︎
I selected#release-engineering-teamtag, because the system wouldn’t allow me to post without selecting anything, and it also wouldn’t allow me to select#flock.
↩︎Allowed? If this changed, please let me know. I wasn’t able to find a clear description. ↩︎
Unless the signing key is leaked
↩︎In principle, a rebuilder could build an identical rpm and then transplant the signature from the original rpm onto the rebuilt rpm, but that wouldn’t match the definition anyway: Debian wants the rebuild to be reproduced without access to the build artifacts. ↩︎
Before you ask: yes, the idea of detached signatures has been raised and rejected (RFE: Detached signature support · Issue #1482 · rpm-software-management/rpm · GitHub). ↩︎
A better, but more complicated option, is to start with the dist-git definition: a spec file and list of hashes of sources and possibly some patches. Long-term, I think this is what we’ll want to do, because this allows us to establish more trust. We can verify that the sources are the same as what upstream provides, and then have a human read the (relatively short) spec file, and look over the patches. Once this is done, we can say “OK, the definition in dist-git is a benign wrapper around the upstream”. But I think this can be left for later: the hard part is making the binary steps reproducible. Once we have that, we can extend the pipeline to the srpm step without too much trouble. Initial experiments show that srpms are generally reproducible, except for some annoying metadata in the form of user and group uids embedded in the srpm. Tickets have been opened to get rid of this (bogus) metadata: RFE: user root:root permissions for files in SRPM · Issue #2604 · rpm-software-management/rpm · GitHub, Source RPMs should have ARCH set to src · Issue #2601 · rpm-software-management/rpm · GitHub. ↩︎
C.f. Use Diffoscope in packager workflows - Fedora Magazine ↩︎
I skipped the issue of macro definitions by just using the
rawhidebuildroot definition to rebuildrawhidepackages. ↩︎Output was edited for clarity. In particular
HEADERIMMUTABLE, which is a quaint copy of some parts of the header that is very very long and differs because the other fields differ, was shortened. ↩︎Again, output was trimmed for clarity. The situation is the same for each of the dozen or so binaries. ↩︎
https://src.fedoraproject.org/rpms/redhat-rpm-config/c/86aae600e62fadc18760d95d1fddd323cf9e9a86 ↩︎
https://fedoraproject.org/wiki/Changes/ReproducibleBuildsClampMtimes ↩︎
Debian’s Known issues related to reproducible builds is a great read. ↩︎