Anyone interested in building alternate kernels or binary kernel modules?

Project Discussion
1

TL;DR

If you are interested in building alternate kernel packages or kernel module packages for Fedora, or if you’re interested in testing alternate kernels or kernel modules on Fedora, I would like to hear from you.

Fedora’s kernel works well if you don’t need out of tree drivers

When I look through user forums, one of the problems I see described most frequently is a blank screen on boot. Sometimes this is a first-time setup and the user hasn’t followed all of the documented installation steps, or they haven’t followed the correct list of steps. (In theory, this should be a one-click operation for users that have enabled third party repos, but the last time I checked that doesn’t actually work because rpmfusion only provides AppStream data for their primary repos, not for the NVIDIA driver-specific repo that users can enable as a third party repo.) Sometimes they’ve rebooted the system without waiting for the invisible background process of building the display driver module and the kernel initramfs to complete. Sometimes it fails and there’s simply no indication to the system’s user why. Maybe the disk ran out of space.

Whatever the cause of the problem, I think that this is one of the top reasons that Fedora is often not rated as a “beginner-friendly” distribution.

Fedora’s policies prohibit alternate kernels, and packaging kernel modules. I suspect that this is driven at least in part by requirements imposed by the agreements under which their Secure Boot signing keys are signed by the UEFI 3rd party signing CA. That’s all perfectly reasonable, but it’s also a barrier to any kind of experimentation or improvement.

I am convinced that Fedora users need pre-built kernel modules. As an SRE, I believe that reliable systems build code, test the build, and then deploy the tested build. Systems like akmods and dkms deploy the code first, then build it in place and “test in prod.” It is inevitable that such systems will fail regularly.

The Nova driver will eventually resolve this problem for most NVIDIA users, but there will continue to be users who want out of tree drivers for ZFS, VirtualBox, WiFi drivers that haven’t merged yet, etc.

Ready-to-run signing infrastructure

There’s no shortage of information about how to sign code with pesign, but they’re not always easy to use. Some guides don’t actually work on contemporary releases. Some guides are hardware specific.

The best way to promote a process is to make it as easy as possible. If a process can simply be “fork and build”, it’s much more likely to be adopted and deployed. I’ve developed a Terraform project that you can fork and build to deploy a VPC in AWS in which a forgejo-runner has an HSM with code signing certificates. Users who install the signing certificate in their MOK can use kernels and kernel modules produced on this infrastructure.

Below, you can find the Terraform project, a kernel rpm, a kernel module rpm, an Atomic desktop configuration, and an Atomic desktop container image, all of which can serve as starting points for further development:

If you’ve installed an Atomic desktop, you can try the Fedora Remix:

sudo rpm-ostree rebase ostree-unverified-image:registry:quay.io/gordonmessmer/atomic-desktop/silverblue:43.20260411.0

Various guides to signing code for Secure Boot, for reference

3 Likes
Fedora Silverblue + NVIDIA Proprietary Drivers + Secure Boot
2

I would love to know what @jforbes thinks about this, as our neighborhood Fedora kernel maintainer! Alternatively, @ffmancera might have an opinion too since he now hacks upstream on the Linux kernel for SUSE.

I am curious since you mentioned that the Nova driver for NVIDIA should fix a lot of issues. Building alternate kernels for Fedora seems like a lot of work to me, but I am neither a SRE nor a kernel engineer. If the Nova driver will eventually fix a lot of the worst outcomes for Fedora users with NVIDIA hardware… is it worth the effort?

3

He shared his thoughts on this thread:

4

Hah, I was just reading that topic and saw his comments from three days ago:

1 Like
5

Well, I didn’t say anything before because I am not into committing my time to NVIDIA drivers. But I am fine giving my opinion, although @jforbes opinion is probably more valuable.

Yes as officially packaged but no (?). I mean, I think a good first goal would be a Copr repository. It is a nice way to hack around until the project is well-established and has a nice base of users.

@thl maintains Fedora vanilla kernel in Copr repository[1] if I am not wrong. I use it and it works quite well.

IMHO, this goal is more realistic than an officially supported kernel or kernel-module RPM.

Disclaimer: I am not an expert on Kernel packaging so my knowledge is quite limited.

  1. ↩︎

1 Like
6

+1

I do. @kwizart also has longterm coprs (otherwise I’d have those as well by now, which would be nice to have things in one place, but I do not want to step on his toes): Making sure you're not a bot!

Glad to hear!

One thing that is missing is better secure boot support. Signing them with the official Fedora key of course is out of the question, but would be nice if someone could extent the support for signing nvidia kernel modules automatically (was added to Fedora’s akmods package a few releases ago iirc) to handle kernel packages from copr, too.

1 Like
7

BTW, I agree – unless we finally improve the kernel-module RPM packaging story and make it work properly in dnf (e.g. wait and [at least for a few days] not even warn with kernel package updates in case matching modules packages do not exist yet; that function would be nice for other packages as well that extend Fedora somehow, like mesa-va-drivers-freeworld).

8

That would be a welcome development, though the problem there is much like a lot of other secure boot problems. It isn’t so much a technical one. Coming up with some way to upload a secret key to copr and being able to use that are fairly trivial problems to solve. But it does open up a whole new level of risk. I don’t know that it would be easy to get such a thing approved.

In the same token, existing 3rd party repositories could certainly create their own key, and use that to sign modules. It would require the user’s physical presence at the machine for a one time approval of adding the key, but things would just work then. That part has been talked about for years, but no one ever seems interested. Having Fedora trust such a key out of the gate is a legal no go.

9

OBS-style per-namespace UEFI signing certificates is certainly an option. The signer service COPR uses supports doing it, since it’s the same one that OBS uses. It would require extending COPR to offer a UI to let the user download the public certificate for the namespace, similar to what we have for PGP keys.

But our ability to integrate and expose signing in the build environment is pretty rudimentary at the moment. I wonder if we have somewhere to store these things and expose them in COPR in the first place…

10

I think that question can’t be answered until after the work has been done. When the option is not available, the benefits of the alternative can’t be examined directly, they can only be speculated about. I believe there are many reasons that it might be worth the effort, but I could be wrong!

First, and most fundamentally, the stable release process exists to allow users (both “people who use Fedora” and “people who write out of tree kernel software”) to work asynchronously. It is a process that provides continued support for users who are tracking a release stream, after the introduction of a new release stream. The overlap allows them to test their workloads and use cases, to report regressions, to update 3rd party integrations, and to rebase to a new release stream on their schedule. And when we don’t provide stable-as-in-overlapping releases, the people who need stable-as-in-overlapping releases simply use something that does. In this way, the rolling-release component becomes a filter. Fedora actively selects for users that don’t need formal, external testing practices by not providing the support for formal, external testing. If we ask Fedora users whether stable-as-in-overlapping kernels are needed, they will say “no” because the people who would answer “yes” have gone elsewhere.

Fedora’s rolling kernel is one of the most frequently cited reasons among non-Fedora users for the beliefs that “Fedora is a test environment for RHEL and not a stable platform”, and that “Fedora is suited for intermediate users but not for new users of GNU/Linux systems.” If nothing else, the lack of a stable-as-in-overlapping kernel has a net negative effect on our reputation, even if you think it works well for Fedora’s users.

11

Copr builds of stable vanilla kernels already exist. There’s no problem there, except that they are not signed.

So, the kernels that I have started building are not built in Copr, they are built in a secure, isolated environment in AWS, with an HSM. They provide the security assurances that Copr provides, with the addition of secure signing.

I hope that we have now moved beyond the first goal.

12

Yes, I have set up Secure Boot signing.

akmods can sign kernel modules on the system where they are deployed, but as I said in the post, akmods (and dkms) are fundamentally, necessarily, both insecure and unreliable. They are insecure and unreliable by design.

13

I’m not certain that I understand why no one has seemed interested, but I will note that basically all of the documentation that I’ve read (all of which I think it linked in the post) talks about signing with some kind of smart card or other physical device, which isn’t something that anyone can add to Copr.

That’s why I developed the build infrastructure as an AWS VPC using Terraform. The barrier to entry is now very very low. Anyone can fork this definition, and immediately deploy their own infrastructure to build and sign code. They don’t need to get a specific device from CDW and find a datacenter to host physical machines.

14

There are multiple levels of secure boot signing. Anyone can secure boot sign anything that want to. They simply have to add the key they sign with to their key ring. Again, this is a 1 time thing. And just like what you have set up, a 3rd party repository could have set up something as part of their build infrastructure years ago. I would expect anyone doing this to, at the minimum explain a bit of their infrastructure. What is allowed to be signed, access control etc. This is just so the end user can make an informed decision before they decide to trust this key. This isn’t a technical problem.

As for the physical device, it is certainly an additional step in security, and we do have to meet certain requirements to have our shim signed. Part of those requirements also extend to what we are willing to sign and automatically trust with our own key, as it is part of the chain of trust.

15

For the record, while such a solution would be great, I was aiming for one level lower: do everything locally, just like we do already with locally build kernel-modules. This is less secure, but is way easier to realize, as most of the bits are likely already there.

16

For the record, as I by point did not come across: I suggested to use the code for signing things that is currently part of the akmods package, but adjust and separate it to sign kernels, too.

17

I see little point in that as rpmfusion doesn’t ship compiled modules, it’s too much of a maintenance burden trying to keep up with fedora kernel releases.

18

That is entirely a decision that any 3rd party repository would have to make in regards to effort vs return on investment. I was simply explaining that the option was always there.

19

I don’t think that would be a good time investment. A system that automatically signs executable code, in place on the system where it will run, adds nothing to system security. It defeats entirely the purpose of Secure Boot.

20

I’ve tried to setup copr repository that will rebuild kmod every now and then a new fedora kernel comes up. But so far, this hasn’t seen much interest…. So I’ve dropped it but I can re-instanciate. The problem is that there is a need to lock nvidia-driver version.

Signing external kmod after building is possible for an organisation that would do secureboot with their own keys. (the nvidia-open-kmod is distributed uncompressed for that very much purpose, although it could be possible to uncompressed, sign and recompress the module).

If the fedora infra is capable of importing this pre-built kmod (or building it in koji, but that would be a separate automation) in the secureboot channel either with the default kernel-key or a dedicated kmod-db-key it should be possible to have a dedicated side-repository with the appropriate nvidia-open version.

Now if we are talking about working on and maintaining community pre-built kmod for free, then I’m sure I have words harsh enough for that idea ! RPM Fusion is a community project, so if you want a particular feature you need to come with a sane process and the needed ressources (man, cpu, etc). This is stated clear enough Making sure you're not a bot!

Note: I will be in vacation until mid-May (other side-note: I will be in flock 2026 Prague)