Kernel uninstall failed during F39 to F40 upgrade

Kernel uninstall failed during F39 to F40. This happened on my Raspberry Pi 400 (pftf UEFI firmware) running regular ARM64 F39 server using systemd-boot (installed by Everything installer).

Apr 29 02:08:20 localhost dnf-3[775]:   Erasing          : kernel-modules-core-6.7.9-200.fc39.aarch64        962/1584
Apr 29 02:08:21 localhost dnf-3[775]:   Running scriptlet: kernel-core-6.7.9-200.fc39.aarch64                963/1584
Apr 29 02:08:21 localhost dnf-3[775]: rm: cannot remove '/boot/efi/04415f4205a64c0691dcaf68b2389ae1/6.7.9-200.fc39.aarch64/': Is a directory
Apr 29 02:08:21 localhost dnf-3[775]: /usr/lib/kernel/install.d/50-dracut.install failed with exit status 1.
Apr 29 02:08:21 localhost dnf-3[775]: error: %preun(kernel-core-6.7.9-200.fc39.aarch64) scriptlet failed, exit status 1
Apr 29 02:08:21 localhost dnf-3[775]: Error in PREUN scriptlet in rpm package kernel-core
Apr 29 02:08:21 localhost dnf-3[775]: error: kernel-core-6.7.9-200.fc39.aarch64: erase failed

Might be a typo in F39’s 50-dracut.install, not sure. I checked F40’s script and it’s rm -f ... so shouldn’t be a problem.
Manually remove them now worked.

$ sudo dnf remove kernel-core-0:6.7.9-200.fc39.aarch64
Running transaction
  Preparing        :                                                                                                                                                                                                                      1/1 
  Running scriptlet: kernel-core-6.7.9-200.fc39.aarch64                                                                                                                                                                                   1/1 
  Erasing          : kernel-core-6.7.9-200.fc39.aarch64                                                                                                                                                                                   1/1 
  Running scriptlet: kernel-core-6.7.9-200.fc39.aarch64                                                                                                                                                                                   1/1 

Removed:
  kernel-core-6.7.9-200.fc39.aarch64

EDIT: Nope happened again. It just somehow works on the 2nd try.

$ sudo dnf remove kernel-0:6.8.4-200.fc39.aarch64 kernel-0:6.8.7-200.fc39.aarch64 kernel-core-0:6.8.4-200.fc39.aarch64 kernel-core-0:6.8.7-200.fc39.aarch64 kernel-modules-0:6.8.4-200.fc39.aarch64 kernel-modules-0:6.8.7-200.fc39.aarch64 kernel-modules-core-0:6.8.4-200.fc39.aarch64 kernel-modules-core-0:6.8.7-200.fc39.aarch64
...
  Running scriptlet: kernel-core-6.8.7-200.fc39.aarch64                                                                                                                                                                                   6/8 
rm: cannot remove '/boot/efi/04415f4205a64c0691dcaf68b2389ae1/6.8.7-200.fc39.aarch64/': Is a directory
/usr/lib/kernel/install.d/50-dracut.install failed with exit status 1.
error: %preun(kernel-core-6.8.7-200.fc39.aarch64) scriptlet failed, exit status 1

Error in PREUN scriptlet in rpm package kernel-core
  Erasing          : kernel-modules-core-6.8.4-200.fc39.aarch64                                                                                                                                                                           7/8 
error: kernel-core-6.8.7-200.fc39.aarch64: erase failed

  Running scriptlet: kernel-core-6.8.4-200.fc39.aarch64                                                                                                                                                                                   8/8 
rm: cannot remove '/boot/efi/04415f4205a64c0691dcaf68b2389ae1/6.8.4-200.fc39.aarch64/': Is a directory
/usr/lib/kernel/install.d/50-dracut.install failed with exit status 1.
error: %preun(kernel-core-6.8.4-200.fc39.aarch64) scriptlet failed, exit status 1

Error in PREUN scriptlet in rpm package kernel-core
error: kernel-core-6.8.4-200.fc39.aarch64: erase failed

I have the same issue.
It seems to be a problem in 50-dracut.install.
I am surprised that there aren’t more people who have this problem.

Tracked it down and filled a bugreport

https://bugzilla.redhat.com/show_bug.cgi?id=2278981

2 Likes

Not sure if related but it seems to have gotten worse. The install also errors out, and doesn’t produce initramfs:

  Running scriptlet: kernel-core-6.9.5-200.fc40.x86_64                                                                                                                                                                                  16/16 
dracut[F]: Can't write to /boot/efi/4345316cea7e4da8a0b3b96131f69dce/6.9.5-200.fc40.x86_64: Directory /boot/efi/4345316cea7e4da8a0b3b96131f69dce/6.9.5-200.fc40.x86_64 does not exist or is not accessible.
/usr/lib/kernel/install.d/50-dracut.install failed with exit status 1.
warning: %posttrans(kernel-core-6.9.5-200.fc40.x86_64) scriptlet failed, exit status 1

Error in POSTTRANS scriptlet in rpm package kernel-core

Woke up to an unbootable VPS today, turns out initramfs is missing. Since I was using dnf-automatic on the server, I never knew the issue. Seems to be happening for quite a while. There was just one last bootable kernel entry left.

p.s. The VPS is x86_64 Server Edition installed by Everything installer.

I also had this initramfs issue during a recent F39 to F40 upgrade, it created a real mess on my system. Had to boot into tty and generate the intiramfs manually.
Subsequent update did the same thing, but this time when I generated the initramfs it completely broke my system.

Backup your stuff.

Also thanks for sharing the bug report.

There are at least 4 bug reports about this, and even if there is a clear fix, the maintainer seems to request a pull request before doing anything.

I’m getting a similar issue in the preuninstall scriptlet:

Running scriptlet: kernel-core-6.8.10-300.fc40.x86_64                                              104/135 
rm: cannot remove '/boot/efi/85c872588932423abcdb1bdda9f4ecce/6.8.10-300.fc40.x86_64/': Is a directory
/usr/lib/kernel/install.d/50-dracut.install failed with exit status 1.
error: %preun(kernel-core-6.8.10-300.fc40.x86_64) scriptlet failed, exit status 1

Error in PREUN scriptlet in rpm package kernel-core
error: kernel-core-6.8.10-300.fc40.x86_64: erase failed

I’ve consistently faced this issue over the (at least) 3 kernel updates.
I’ve manually run dnf erase kernel-core-6.X-Y.fc40.x86_64

@vekruse where did you get the information that:

the maintainer seems to request a pull request before doing anything.

Isn’t there some way to bring this issue to the attention of the fedora quality team in general (I’m not familiar with using tags on discourse, edit: @navras I think you might be able to add the tag by editing the original post) ?

I’m seeing issue 2274356 – `dnf erase kernel-core` fails while running the preuninstall scriptlet has been open for 2+ months at this point which is discouraging given that there is a suggested fix (2278981) for a long time with no comment addressing it, or the issue in general.

https://bugzilla.redhat.com/show_bug.cgi?id=2276271#c18

Thank you for adding the context.

I mean, they aren’t wrong that upstreaming (in this case to dracut) changes is not “simple”, but I feel like someone should fix this downstream so that Fedora itself doesn’t fail every single kernel update…

That is obvious. That is also what is said in the Bugzilla reports. It is a simple fix in a simple shell program. That being said, the problem doesn’t affect grub2, only systemd-boot.

Failed again today…

quality-team could we get some attention on this on Fedora side?
Waiting for upstream for something as critical as kernel installs consistently failing for months seems unreasonable.

Two months and eight days after multiple people pointed out where the issue is and proposed solutions and fixes - just one line in a shell script - a maintainer finally accepted it.
This package is maintained by RedHat employees.
The support of systemd-boot was a big advertised feature of fc40.

Such happenings get more common nowadays. Ten years ago i had an issue with a fuse package, tracked it down, made a bug report with a fix and it was deployed within 6 hours.
Over the past decade, the RedHat part of Fedora got slower, while the Community is still as great as it has always been. RedHat turned from a benefit into a burden.

1 Like

I wonder if the purchase of Red Hat by IBM had anything to do with it.

Let me start by saying that in this specific case, I really did share your frustration.
I even opened a topic on Project Discussion to better understand what was going on:

While the issue might be frustrating, I personally found the responses and information I got on that thread very reasonable and explanatory.


Not sure how you came to this conclusion.
Maybe because of some Phoronix click-baity article?

In my experience, systemd-boot support has always been advertised as “experimental” and “niche” on Fedora, and was simply introduced to relieve users that still wanted to set it up from having to manually install and configure the alternative boot process. Don’t forget you had to use the Everything ISO and explicit parameters to set it up, it’s not like it came as a checkbox in Anaconda…

As long as this issue does not affect RHEL directly, this fact is mostly irrelevant.
Many contributions to Fedora might come from RH employees, but they very often - but not always - come from investment of their free time rather than from their working/paid time.

Because of that, one bug fix might be deployed within hours or days, while another might take… well… months, depending on how the maintainer decides to invest their time.

For example, here is another issue relevant to systemd-boot installation I had some time ago, which I have had a great experience with in regards to how the maintainer handled it. I only mention this because this would still be a very recent example when compared with the timeline you used to question if maybe a general trend has been introduced of issues being handled slower.

1 Like

Added arm64, bug-reported

While lots of fedora is Community run. And the Community is great. And i contributed some things too… those core packages are RedHat.

Not sure how you came to this conclusion.
Maybe because of some Phoronix click-baity article?

The package is built from their Red Hat fork from their Red Hat team and the person who wrote the patch that broke the uninstall script after update (the patch was alright back then, it just broke after upstream updates), which you see mentioned in this issue, that is linked in the reply that got marked as the solution here, on May 3rd, is a RedHat employee.
The maintainer who wrote on the bugtracker that they wait for an upstream update, is a RedHat employee.

systemd-boot support is being worked on and advertised for multple releases now. During fc39, the accepted and finished (?) change proposal for clean up of systemd-boot and adding sdubby is owned by a RedHat employee. The same person who made sdubby.

Later, maintainers realized that sdubby isn’t providing all options of grubby and they had to remove the Provides: grubby from the package.
This causes other packages, that need to set kernel arguments and use the official way of RedHat (grubby), to pull in grub as dependency.
Sadly, the employee seems to not be assigned to sdubby anymore, the last commit was 11 months ago.

Of course the Community could now pick up the half-finished work that RedHat produced and fix it. But this is getting increasingly difficult. Like this issue here shows, the solution was there and provided to them for more than two months.

More likely that purchases just illustrates a growing trend:

The problem with opensource software goes far beyond the purchase of Red Hat. The industry has become dominated by large companies that use open source but don’t contribute back in proportion to the benefits they get.

Original developers of important open-source artifacts are often past their “best before” date, and opportunities for contributing to open source as part of a job have disappeared, so the pool of potential new contributors is limited. Years ago, one of my brothers worked for HP. He was given time to work on any project that interested him (I think 20% of his time, e.g., one day a week). I’ve worked in university and government environments. Early in my career sabbaticals for the top job levels were at a rate of 1 year for 5 years of service. In 35 years I actually got one month, and later in my career was asked to use vacation to participate in workshops and conferences that were once considered paid time. Glassdoor has information about sabbaticals. In general, the idea that employees get a fraction of time where they choose what to do has been replaced by a system where you have to get employer approval. There are things like internships for work on open source projects, but I’m not sure the numbers are significant, and again, there is an application process.

I have to agree here. 20-30 years ago, you reported a bug, you got a response from the maintainer and it was fixed. Nowadays, in Debian, which is really a huge surprise as their bug tracking system was awesome, and in Fedora as well, whenever I report a bug, nobody even cares to acknowledge it or answer it therefore nothing is fixed.

As far as kernel is concerned and considering I have computers from 2012-2013, I’m OK with kernels from 5.10 to 6.8.7. All kernels issued after that are problematic. And nobody will ever fix them because of their short shell life. They let the bug expire as the distro become unsupported.

So, in conclusion, there’s no compelling reason to upgrade your kernel if the newer versions are known to have issues and your system is working fine with the current one.

Older kernels only get security updates that don’t depend on major reworking that is done in newer kernels. Many issues with newer kernels require updates to the vendor firmware, so once the vendor stops maintaining the firmware you may be stuck with older kernels and known security issues.

1 Like