Kernel - when to upgrade and how to identify relevant "what's new"?

Ask Fedora
1

What’s the consensus on how to upgrade your kernel version?

There are so many considerations to take into account. Security, obviously, should be utmost important, but it’s unclear in “What’s new” for each release what is pertinent to each user.

I found this website that gives relatively good summary of changes for each kernel release: LinuxVersions - Linux Kernel Newbies . But I guess what I’m struggling with ever since I started using Linux years ago is being unable to identify what changes are pertinent to ME versus all other hardware, both new and old. You know how kernel versions keep adding things relevant ONLY to new hardware, never having anything to do with current hardware? Is there a way to filter new changes to kernel based on only hardware existing at a certain kernel version level?

Does anyone have a really extraordinary website to go to for really clever kernel version changes?

I’m also not sure how to split kernel changes in terms of new functionality that affects the way kernel works right now versus new functionality that will only be relevant to me as a user when developers start implementing it. Is there a way to filter changes for this, maybe someone knows?

2

There are so many kernel changes happening that i personally find it too much to follow.

I just upgrade blindly unless i run into a serious issue and have to revert. Then i freeze my working kernel and follow the updates on that issue until i feel safe that it’s fixed.

2 Likes
3

I hope I understand your question.

Kernel development is heavily distributed, and many people & companies are involved that have to test everything to see if it affects their use case or their own code they maintained in /added to the kernel. So there is a lot of indepdenent review and testing from different perspectives. That’s a major origin of the stability and security of the kernel.

All changes can be seen in kernel.org → you can get there all changelogs of any kernels. As far as it concerns Fedora, you find our kernel changelogs in kernel | Package Info | koji → click on the very build to see (among others) its changelog. Be aware that our kernel maintainers at Fedora avoid to change the upstream kernel, but add only what is necessary that it fits Fedora or sometimes to backport patches if they cause trouble and have not yet been backported upstream (e.g., the current patch for the execheap error → that’s usually patches that are already determined to be added to the mainline kernel anyway).

The work on the kernel is done neutrally in the kernel community where all companies and communities involved in kernel development come together. Then, all take the kernel, make their respective changes as far as necessary, review & test what is applicable to them, and also push their results with changes/bugs/issues back to the kernel community → feedback loops.

In short, it is not possible that any one user can have a holistic overview of all that occurs to the kernel (in terms of understand all implications), it contains a chain of trust → at the git repo of Linus Torvalds, all changes towards the next kernel come together, but he trusts the sub-system maintainers to some extent, and I do not think he is able to review and verify every change and every implication that is contained, but relies to some extent on the sub-system maintainers (who are known and reputated people as well), whereas everyone who does any change has to consider at least the related code/sub-systems in detail, and thus single points of failure are mitigated. In order to proceed in development, everyone has then to pull from Linus, and see what he changed and proceed correspondingly. The major consensus that shapes the kernel is effectively the consensus to pull the mainline kernel from Linus and stable patches from Greg K-H.

That is a strong simplification, but I hope it sheds a little light on that it is not realistic to get a complete overview of what is going on in the kernel, you can only focus to use a kernel build that contains widespread testing & is based on code that has undergone review, and that thus brought a lot of knowledge together. As Fedora is a major distro in upstream development, I think we can say that this applies to us.

As a user, the suggestion is to always update the kernel, as old kernels no longer get review/testing in changed circumstances, and thus might become stability/security risks. If you want to stick with one kernel as long as possible and only get necessary stability/security patches, then you need to get a distribution that supports a long term maintained kernel.

2 Likes
4

In the general case each kernel release is an improvement on the previous.
In some cases your specific hardware may see a regression.

I always update without looking at a change log, and that change log is HUGE.
See https://lwn.net/Articles/982034/ that documents what is expected in 6.11 from the 4,072 changes (there will be more before its released).

I do track kernel developments, but from curiousity, not as a way to judge if I should update. Making that judgment would be a full time job!

What do I do? I update once a week in the knowledge that if a regression affects me I can boot from the previous working kernel.

5

A few machines have their own Linux user sites, usually the effort of one user, where such questions are discussed and you can see what issues others have.

The linux kernel is used on a huge range of devices from IOT to big data centers, so many changes won’t impact your system. Kernel updates are often accompanied by new versions of other programs, so many problems stem from other programs that either missed necessary changes or have mistakes in the way kernel changes are handled. If your system does have a problem after a particular kernel update it is much better to identify the problem ASAP because waiting only increases the number of changes that might be the source of the problem so increases the difficulty of isolating a problem. Linux distros generally provide access to older kernels, so you should have a fallback to a working kernel if a new kernel or other updated component causes problems.

From years of experience with Linux, I have found that network, BlueTooth, and sound often have issues with newer kernels. USB dongles usually work when a new kernel breaks support for the internal devices. This has the advantages that:

  • I can report the issue soon after it was introduced
  • I can still use a current kernel with security updates
  • It is easy to check if an update fixes the problem with the internal device

If you encounter a problem, a good bug report made while developers still have the recent changes fresh in their minds increases the chances of getting a fix quickly. There are benefits to other users and you are making things easier for developers who are often very busy. This means becoming familiar with the linux command line so you can generate plain text error summaries suitable for bug reports. A good place to start is Linux Command. The journalctl program (which is run in a terminal) can provide enourmous amounts of detail, but takes some effort to filter out the details relevant to a particular problem. You should take some time to learn how to use journalctl before you are under pressure to solve a problem. You can find many examples in this forum where there are suggestions of journalctl filters to collect data for various problems.

6

In general – always update.

There are security fixes as well as bug fixes and upgrades that affect all users.

If it breaks, then as already noted, boot back to the older working version and file a bug report.

If it does not break then consider yourself one of the very very large majority that benefit from the tireless work of the kernel developers to ensure the best and most secure product possible. Trying to keep up with the thousands of changes with every single kernel update would be more than a full time job.

If you are a kernel developer then the details matter. If not then the function matters and the details are irrelevant.

7

Oh, great, there’s Redhat specific stuff you have to worry about too on top of vanilla kernel updates. faint

This is a diversion. Each kernel release is a pandora box of bugs and security holes. Which is the hole (haha pun) point.

I believe average users should be lagging behind mainline version by at least one year, to allow for any bugs/issues to be identified and fixed. Which is what bugs (pun haha) me - flesh out critical security fixes in kernel updates that DO warrant an update. {what’s the easiest way to apply security patches only? I don’t even know}

It’s just so noice to have a workhorse compiled kernel that just never crashes or causes issues…and then you just look for security fixes and significant security features in new versions…but that’s a nightmare to do!

8

That is not the best choice if the care about security.
It is also not the view of the maintainers of the stable kernels.
They say its better to use the latest release of kernel, after testing your use cases, then to use an older kernel with cherry picked back ported fixes.

I have work with linux in production environments for many years.
Use LTS kernels like Centos ships has been not been an advantage.

I have also shipped product that used Fedora as a base that was a significant advantage, especially getting hardware support and graphics drivers fixes.

9

Security patches are backported to the LTS kernels closely following the mainline kernel, so that is not necessarily an issue. Also most of the security issues in the kernel are related to local vulnerabilities and privilige escalations, which will be less important to consumer and enthusiast environments.

Do you have a source for that perhaps? I would like to read more about this.

I’ve used Debian and Ubuntu LTS in the past, and i have to say that the lack of regressions compared to (semi-)rolling kernel updates was kind of nice. Stuff just always worked. There are a lot of regressions popping up in the mainline kernel all the time.

10

That does reflect my experience, and I’m always on the latest stable kernel, sometime even on the one in testing repository.

2 Likes
11

You could look at cki-project / kernel-ark · GitLab

But it is still going to require digging through the commits.

i.e.,

12

just to clarify another thing.
@soconfused, you are referring to mainline kernel. However, Fedora’s kernel is the latest stable kernel with Fedora-specific patches applied, see https://src.fedoraproject.org/rpms/kernel/blob/f40/f/kernel.spec. It’s not just the raw, mainline kernel.

13

Right. I’m on Fedora-specific kernel with all Fedora’s patches, custom compiled and clamped it at an earlier version. So I’m double so confused. Mainline kernel changelog gives me one set of changes, and then there’s Fedora-specific kernel changes…which may be important enough.

14

There have been far to many cases where back ports have been buggy or incomplete because not everything that the fix depends on was identified for back porting.

It’s what Greg Kroah-Hartman has said in the recent past as reported on lwn.net (I think is the most likely place I read Greg’s thinking).

15

You can become a domain expert yourself or you can decide who you trust.

I choose to trust the linux kernel developers and the Fedora kernel team.

16

As a long-time Linux user, I remember Backports and long-term stable kernels [by Jonathan Corbett (2016)] and even The trouble with backporting fixes [Posted February 26, 2004 by Jonathan Corbet] :

Relatively young software and the new and unknown bugs it is certain to have may turn out to be safer than staying with an older version, which has old and well-documented bugs.