Idea: Ensure to always have a rollback deployment with different kernel

Project Discussion
, , , ,
1

Hey there!

With the current Kernel 6.13 issues, this came into my mind.

rpm-ostree always keeps one old deployment. But it does not keep an older kernel, if you have 2 deployments with the same kernel.

It would be good to detect kernel updates, and pin the current deployment. Then on the next kernel update, unpin the old deployment and pin the new current one, before rebooting.

I dont know how this could be done, it would likely need to be integrated like the system that keeps 2 deployments, but more complex.

Currently this is a big issue, as bugs like the current one, only happening with some Flatpaks, are not always recognized. And deploying an old version removes all rpm-ostree state, which also causes a lot of UX issues.

3 Likes
2

In a different thread there came up this idea @job79

https://bodhi.fedoraproject.org/updates/?packages=kernel

  1. keep track of the bodhi URLs of working kernels. Not sure how to do that.
  2. log the URLs in a file, delete all apart of x that are kept
  3. add the kernel version to that log
  4. have some CLI or GUI tool to execute this command
rpm-ostree override replace $URL

so the question is, how can I get the bodhi URL that corresponds with the currently used kernel version?

1 Like
3

Making it more accessible sounds like a good idea. I think making it work using a simple script is a good first step?

So I did some digging and found a simple HTTP-API that we could use. Couldn’t find much documentation about the API but the python code of the bodhi-client is pretty well documented.

Making an equivalent request as the python code using curl looks like this:

$ curl 'https://bodhi.fedoraproject.org/updates/?packages=kernel&releases=F41&rows_per_page=10'

There are of course many more filters, see the python code for the full list. We would probably want to filter out unpushed or obsoleted updates for example.

The response is pretty detailed, but it contains the update title and url, which is what we would need:

$ curl 'https://bodhi.fedoraproject.org/updates/?packages=kernel&releases=F41&rows_per_page=10' \
      jq -r '.updates.[] | "\(.title) \(.url)"'
> kernel-6.13.5-200.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2025-232314eb1d
> kernel-6.13.4-200.fc41 kernel-headers-6.13.3-200.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2025-f476a8c306
> kernel-6.13.3-201.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2025-41b383769b
> kernel-6.12.15-200.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2025-cca2fcc70c
> kernel-6.12.14-200.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2025-d9b0fecaed
> kernel-6.12.13-200.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2025-573a085a7c
> kernel-6.12.12-200.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2025-d00af34217
> kernel-6.12.11-200.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2025-fb41eca06b
> kernel-6.12.10-200.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2025-2d3a101e1b
> kernel-6.12.9-200.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2025-3b60b81a3d
3 Likes
4

nice! That output is absolutely workable

5

I love this idea! I have run into problems that were kernel related. Thankfully because I was using an atomic system I was able to navigate my way around, but if there was already a fall back option on the previous kernel already pinned, that could be a huge lifesaver! I know that with mainline Linux you can find your way to a new kernel if you know what you’re doing, but for folks who aren’t as strong with troubleshooting, having a pinned deployment lowers the barrier for the bandaid solution.

1 Like
6

Yes we have 2 separate ideas here.

One would just be a list / script to replace the kernel with an older version, as it is true, if you have more than 1 deployment of the same kernel, it probably works.

(I think there could still be edge cases)

The other would be hooking into whatever system rpm-ostree uses to pin and unpin deployments and add a lot more logic to it, which would be more complicated but also have a better result

2 Likes