I have been looking a bit more at how things are packaged on Fedora and other distros/OS following that whole xz issue.
I am not sure that I understand fully how packages are built for Fedora and would like to confirm it & ask question re. the presence / absence of PGP keys.
For some packages, eg. keepassxc, there is the signing key as seen here in the folder, the spec file, and the sources.
For some other packages, I could not find anything similar although these packages are security-related and I would think that they are somewhat “sensitive”. Checking on Arch, there are PGP keys filled in on their build system for the packages lacking them on Fedora. Would the build process of Fedora make it less critical?
If I understand correctly, for Fedora, the maintainer would upload the tarball and the source is not downloaded during the compilation process so the PGP key could be checked by the maintainer. This said, the packaging guidelines recommend uploading the PGP keys and there’s no possibility to confirm that the maintainers has checked it? What would typically be the level of review done by a maintainer?
If it’s acceptable to contact the maintainer to ask for the PGP key to be included, what’s the best way of doing so? An email, pull request, bug report, anything else?
It is recommended (and I believe part of the package review process for new packages) that if the upstream provides a PGP signature on their sources that the packager SHOULD check it in the spec file (using the gpgverify macro). AFAIK there has been no consistent effort to go back for older packages and validate that if a package provides an upstream signature to add it to the existing package. It should be noted that not all upstreams provide signatures on their sources. If you can identify upstreams that provide signatures on their sources I would think most packagers would appreciate notification (I would personally email first before a PR, but every packager is likely different in their preferences).
The relevant section of the packaging guidelines is here: Source File Verification. Yes, those signatures should be used when available. I would say the standard approach is to open a bug report if you just ask the maintainer to add those signatures, or a pull request if you are willing to implement the change.
It seems like it’d be good to add a step where we include those files in dist-git (and maybe in the spec file) and have a step in the build process (on the build server) which verifies the signature. That way, it’d happen every time, and if the upstream key changes, the packager would need to notice (and should investigate).
Make this optional at first, and then required after a few releases, but have a flag to say “upstream is not signed” – which in itself would be interesting to track.
Please note that PGP signatures are used to sign packages by the maintainer in Arch. In Fedora we use a different mechanism: the maintainer authenticates to koji using a Kerberos ticket, and the signature is made in an isolated signing system in the build infrastructure.
The checking of PGP signatures on upstream tarball is a completely separate issue…
As Otto wrote, since a few years, we have a rule that those signatures must be checked if upstream provides them. Thus, if you find any such cases, it’d be great to file a pull request against the spec file to add the upstream keyring and check the tarball.
As far as I can see, it is still “SHOULD” not “MUST”.
I think we should go a step further and do the check in the buildsystem rather than optionally in the spec file.
We could add another layer of review[1] with a single repo which holds keys for all Fedora packages. Each package would have:
a key file or a note that upstream does not sign packages or commits[2]
a document or config entry saying where that key came from
optionally, some config (or even a script) describing the various special cases the packaging guidelines cover now
We could give merge-request permissions to provenpackagers, or to a theoretically-active security team, with a rule that the person accepting the commit must themselves verify the origin of the key.
Oh, indeed. For some reason I thought it’s required. But “SHOULD” still means that this expected, unless there are good reasons not to.
I don’t think this is at all feasible. Github tarballs have become very popular, for good reasons, and they do not offer GPG signatures. Similarly, neither crates.io nor pypi.org require GPG signatures. (I think maven central is the only widely used distribution channel that does). So I expect that GPG signatures being available is the exception rather than the norm.
Also, I don’t see why we should centralize this. The problem is that the signatures are rarely available, and that historically we didn’t check them, and that the tooling is very awkward. Adding a heavy-weight process on top with provenpackagers is not going to help.
Quite often, the tag from which the tarball is generated is signed, and this is actually very easy to do, since git tag and git tag -s are almost the same amount of work. We could institute a service which does two things:
for any tarballs listed as Source, check that the download from upstream actually matches what the packager hashed in sources or committed.
for any tarballs in Sources from github and maybe other places, download the repo, build the tarball, and check that the contents of the tarball match the signed tag.
We have some fedora-review check for the first of those things, we could do that periodically for all packages.
Speaking from guilty personal experience, it’s easy to convince oneself that “I’m busy and this is some random dep I don’t even want to be packaging” is a “good reason”. I think we probably should[1] upgrade this to “must”.
Idea is to put it somewhere that involves another person when updating the key, not just at package review time. But it’s just an idea, and it does have drawbacks.
Adding impedance to upgrading packages (when signing keys change or are added) is likely to get more people to want to try to justify skipping the work, or stop upgrading to the latest release as too much overhead. I will note that in some cases the upstream makes it easy to download signing keys, and sometimes there is a lot of work to collect them (I am not justifying how some upstreams work, just stating sometimes it is a lot of annoying work), and I almost tried to convince myself I had a good reason not to include the signing check when I later updated to a version of a package that finally signed their releases (so the original review approval without signing was appropriate, I only noticed it when the new release was signed, and decided to implement the SHOULD as being the right thing to do, even with the extra work involved, but it likely no one would have known if I had not done it).
I simply do not know how many upstreams actually create and sign their releases, but I suspect it is a small subset of the packages in Fedora. I agree with @zbyszek when he points out that one of the most popular distribution points (github) is not especially signing friendly (yes, you can create a release and then create a detached signature and update the release, but it is work most projects tend to not do). Some projects depend on signed tags instead, but that has yet a different workflow.
Picking on some obvious cases of not checking signing, last I knew, the Fedora release linux kernel builds do not verify the available signing, and neither does Fedora builds of glibc. I don’t think systemd signs their releases, but Fedora cannot fix upstream choices.
We do not. We probably should It’s additional hassle during the package release, but if the upload of the signature to github can be nicely scripted, I don’t think there’d be much opposition to that.
My recollection is that there is a way to script the addition of an asset (in this case the detached signature) to a release, but the issue is likely securely creating the detached signature to add, as you may not want github itself to hold both the signing key and passphrase even in the github secrets storage. There are various possible ways to accomplish signing in a more secure way (with greater overhead(s) and less convenience), but those ideas are probably out of scope for this topic.
Github can have my personal signing over my dead body. OK, just joking, when I die, my personal signing key dies with me
More seriously: the signature needs to be made locally and uploaded to github. Thus, either:
tag the release, build a tarball, sign locally, push the tag and tarball and signature to github,
tag the release, build a tarball in the way that github builds it, sign locally, push the tag and signature to github, rely on github autogenerated tarballs being stable.
Lots of interesting responses and answers, thank you!
So I think that the consensus is to notify the packager though the method differ. I’ll use an email before opening a bug report, I’m still getting to grasp with rpm packages and how it all works.
Regarding the signing itself, as far as I know, Fedora and Arch use the same principles with either the %{gpgverify} macro for Fedora or validpgpkeys for Arch. When building with PKGBUILD and the signature does not check out, it fails. I suppose the same would be true of Fedora. The build process seem quite similar, which is why I was comparing the spec and PKGBUILD files. The keepassxc package is interesting to compare see Fedora and Arch.
Ah these are good examples! These are indeed signed on the Arch Gitlab, linux and glibc. I’m not as familiar with Debian and I don’t know how they check packages.
I suppose that this would help mitigate would be if the tarball are replaced by someone else than the developer(s), if the keys are rotated due to security reasons, etc. If Github automatically packages releases and the commits are signed, would that not add an additional layer of certainty as compared to an unsigned tarball downloaded from a third-party website?
Would a maintainer typically go through the changes between two tarballs/releases?
It’s additional hassle during the package release, but if the upload of the signature to github can be nicely scripted, I don’t think there’d be much opposition to that.