Failed upgrade from 38 to 39 -- system is now in between

Ask Fedora
, , ,
1

Hi,

I used dnf upgrade to upgrade from fedora 38 to 39. However, after reboot it caused the dreaded “Oh no, something has gone wrong” issue. Somehow I got the gnome to work again by rolling back some packages, and the system is in a very usable and seemingly stable state. However, the upgrade seems to be a mix of fc38 and fc39, and it reports as fc38 and runs a 38 kernel. /etc/fedora-release is “Fedora release 39 (Thirty Nine).”

So, I’d like to get to a fully fc39 system, and am willing to do it manually, but I’m not sure which way is best.

I tried dnf upgrade --refresh again and that runs without error. After that, to try again, I ran:

$ sudo dnf system-upgrade download --releasever=39
Before you continue ensure that your system is fully upgraded by running "dnf --refresh upgrade". Do you want to continue [y/N]: y
Fedora 39 - x86_64                                                  193 kB/s |  30 kB     00:00    
Fedora 39 openh264 (From Cisco) - x86_64                            6.9 kB/s | 989  B     00:00    
Fedora 39 - x86_64 - Updates                                        168 kB/s |  26 kB     00:00    
google-chrome                                                        24 kB/s | 1.3 kB     00:00    
RPM Fusion for Fedora 39 - Free                                      11 kB/s | 3.6 kB     00:00    
RPM Fusion for Fedora 39 - Free - Updates                            13 kB/s | 3.9 kB     00:00    
RPM Fusion for Fedora 39 - Nonfree                                   22 kB/s | 6.8 kB     00:00    
RPM Fusion for Fedora 39 - Nonfree - Updates                         22 kB/s | 6.9 kB     00:00    
Visual Studio Code                                                   38 kB/s | 1.5 kB     00:00    
No match for group package "gimp-heif-plugin"
No match for group package "python3-dnf-plugin-system-upgrade"
No match for group package "libertas-usb8388-firmware"
No match for group package "iwl135-firmware"
No match for group package "nafees-pakistani-naskh-fonts"
No match for group package "libproxy-duktape"
No match for group package "iwl3160-firmware"
No match for group package "thai-scalable-tlwgtypist-fonts"
[...]
Error: 
 Problem: The operation would result in removing the following protected packages: grub2-tools-minimal, setup, systemd, systemd-udev
(try to add '--skip-broken' to skip uninstallable packages)

So, it seems unfixable via the normal upgrade process. (Note: using --skip-broken and --allowerasing, etc. do not help.)

Also, the system reports as fedora 38, not 39.

rpm -qa | grep fedora-release
fedora-release-identity-workstation-38-37.noarch
fedora-release-workstation-38-37.noarch
fedora-release-identity-workstation-39-36.noarch
fedora-release-common-39-36.noarch
fedora-release-workstation-39-36.noarch

If anyone has any idea how to upgrade the rest of the packages to fc39 I’d appreciate it. If would help just to know how to get the system to believe it is fc39, then perhaps i could manually update rpms.

Please don’t suggest I restore fedora 38 from backup, because a) I’ll probably encounter the same exact error as before when trying to upgrade again and b) I want to move forward, not backward if possible, given the system is stable. Since fc38 is end of life, I need to be on 39 to get updates, etc.

Thank you for any help!

image

2

I’d try booting to runlevel 3 (i.e. add a 3 to the list of kernel parameters when you boot the system), signing in as root, and running dnf distro-sync --releasever=39. Then reboot again and it should work.

2 Likes
3

I would uninstall these two packages. Removing the second one in particular will tell dnf that you’re on F39.

You can see how far along the upgrade got by comparing rpm -qa | grep fc38 | wc -l to rpm -qa | grep fc39 | wc -l. If there are lots of fc39 packages, I’d proceed with a dnf upgrade and then dnf remove --duplicates.

1 Like
4

Thanks. Unfortunately, dnf distro-sync --releasever=39 failed with:

Error: 
 Problem: The operation would result in removing the following protected packages: grub2-tools-minimal, setup, systemd, systemd-udev
(try to add '--skip-broken' to skip uninstallable packages)

Same error I was getting earlier.

5

Thanks for this suggestion. I removed those 2 packages, and indeed dnf thinks it’s on 39. However, dnf upgrade still fails:

$ dnf upgrade
Last metadata expiration check: 0:06:57 ago on Sat 15 Jun 2024 09:37:46 AM EDT.
Dependencies resolved.

 Problem 1: cannot install both libjxl-1:0.8.2-3.fc39.x86_64 from fedora and libjxl-1:0.7.0-6.fc38.x86_64 from @System
  - package webkit2gtk4.0-2.44.1-2.fc38.x86_64 from @System requires libjxl.so.0.7()(64bit), but none of the providers can be installed
  - package webkit2gtk4.0-2.44.1-2.fc38.x86_64 from @System requires libjxl.so.0.7(JXL_0)(64bit), but none of the providers can be installed
  - cannot install the best update candidate for package libjxl-1:0.7.0-6.fc38.x86_64
  - problem with installed package webkit2gtk4.0-2.44.1-2.fc38.x86_64
 Problem 2: cannot install both libjxl-1:0.8.2-3.fc39.x86_64 from fedora and libjxl-1:0.7.0-6.fc38.x86_64 from @System
[...]

I may have to remove a lot of packages until the conflicts go away. Not sure how best to proceed.

It seems the upgrade got pretty far: 3633 packages are fc39 and only 526 are fc38.

Thanks for your help.

6

Update: I managed to get around all the dnf upgrade conflicts and now I’m a lot closer.

One issue is that the fc39 kernel is installed but not in my grub menu. Perhaps I should reinstall the kernel and see if that fixes things? Is that wise to do?

Thanks.

7

You can try.

But I would like to see the output from the command sudo find /boot, because it could confirm one of the common causes of this problem.

1 Like
8 
$ sudo find /boot
/boot
/boot/.vmlinuz-6.8.9-100.fc38.x86_64.hmac
/boot/grub2
/boot/grub2/fonts
/boot/grub2/fonts/unicode.pf2
/boot/grub2/grub.cfg
/boot/grub2/grubenv
/boot/symvers-6.8.9-100.fc38.x86_64.xz
/boot/lost+found
/boot/initramfs-6.8.8-100.fc38.x86_64.img
/boot/initramfs-6.8.9-100.fc38.x86_64.img
/boot/vmlinuz-6.8.9-100.fc38.x86_64
/boot/vmlinuz-0-rescue-3b10a7b109d44abbaa2f8b0048f0788c
/boot/System.map-6.8.8-100.fc38.x86_64
/boot/efi
/boot/efi/EFI
/boot/efi/EFI/BOOT
/boot/efi/EFI/BOOT/BOOTIA32.EFI
/boot/efi/EFI/BOOT/BOOTX64.EFI
/boot/efi/EFI/BOOT/fbia32.efi
/boot/efi/EFI/BOOT/fbx64.efi
/boot/efi/EFI/fedora
/boot/efi/EFI/fedora/grub.cfg
/boot/efi/EFI/fedora/BOOTIA32.CSV
/boot/efi/EFI/fedora/BOOTX64.CSV
/boot/efi/EFI/fedora/gcdia32.efi
/boot/efi/EFI/fedora/gcdx64.efi
/boot/efi/EFI/fedora/grubia32.efi
/boot/efi/EFI/fedora/grubx64.efi
/boot/efi/EFI/fedora/mmia32.efi
/boot/efi/EFI/fedora/mmx64.efi
/boot/efi/EFI/fedora/shim.efi
/boot/efi/EFI/fedora/shimia32.efi
/boot/efi/EFI/fedora/shimx64.efi
/boot/efi/System
/boot/efi/System/Library
/boot/efi/System/Library/CoreServices
/boot/efi/System/Library/CoreServices/SystemVersion.plist
/boot/efi/mach_kernel
/boot/System.map-6.8.9-100.fc38.x86_64
/boot/.vmlinuz-6.8.8-100.fc38.x86_64.hmac
/boot/vmlinuz-6.8.8-100.fc38.x86_64
/boot/config-6.8.9-100.fc38.x86_64
/boot/initramfs-0-rescue-3b10a7b109d44abbaa2f8b0048f0788c.img
/boot/loader
/boot/loader/entries
/boot/loader/entries/3b10a7b109d44abbaa2f8b0048f0788c-6.8.9-100.fc38.x86_64.conf
/boot/loader/entries/3b10a7b109d44abbaa2f8b0048f0788c-6.8.8-100.fc38.x86_64.conf
/boot/loader/entries/3b10a7b109d44abbaa2f8b0048f0788c-0-rescue.conf
/boot/config-6.8.8-100.fc38.x86_64
/boot/symvers-6.8.8-100.fc38.x86_64.xz
9

Nothing bad in the listing. Except you didn’t get the kernel updated.

I expect that sudo dnf update 'kernel*' should fetch version 6.8.11-200.fc39.

10

It’s risky, but in the very rare cases where I’ve run into that “cannot install both …fc<N>… and …fc<N-1>…” message during an update, I’ve used and gotten away with running rpm --nodeps -e <whatever>.fc<N-1>.x86_64 to rip the problematic package(s) from the system. Immediately running dnf update --releasever=<N> would then succeed. If you find yourself in that situation, you would want to:

  1. Make a backup of any important files first.
  2. Do the forced removal and subsequent update while as little of the system is running as possible (i.e. from runlevel 3).
  3. If possible, take a snapshot of your root filesystem so you can revert things if the end result turns out to be worse.

It is very rare in my experience, but unfortunately updates can fail from time to time due to package spec errors or things like power or hardware failures during the update. The good thing about Linux is that it is almost always possible to recover from these situations.

2 Likes
11

Thanks for the tips. Yeah, with Linux I always know it’s fixable and I usually know how, but this time was new for me, and I’ve been on fedora since 11 or so. :slightly_smiling_face:

1 Like
12

Update: reinstalling the fc39 kernel worked. All is well now! Thanks to everyone for the help!

3 Likes
13

For future readers, I’d like to summarize what I’ve learned from this experience.

I found that, even after I got up and running and reinstalled the fc39 kernel so it appeared in the grub menu, I still had a lot of fc38 packages installed, which were mostly duplicates. So there was more work to do; dnf distr-sync --releasever=39 was the answer, but not before I had to clear one obstacle.

So, here’s the summary:

  1. If you get the dreaded “Oh no! Something has gone wrong.” screen, you can try a virtual terminal to fix the problem. however, in my case no virtual terminals were running, so the best thing to do is reboot, go to the grub menu, select the kernel you want and hit ‘e’. Then add a " 3" at the end of the kernel options list, which will put you into run level 3 and bypass any graphical GDM issues.

  2. If dnf distro-sync --releasever=39 does not work due to “Problem: The operation would result in removing the following protected packages: systemd, systemd-udev
    (try to add ‘–skip-broken’ to skip uninstallable packages)”, you must use sudo rpm -e --nodeps <package>.fc38.<arch> on all duplicate packages until distro-sync works. See here for more info.

  3. Distro-sync will remove any existing duplicates. I was unable to use it, as @glb had originally suggested, due to not forcibly removing the conflicting duplicates in the above step. Had I known I would have been able to recover sooner.

  4. If the fc39 kernel is not showing up in your grub menu, reinstall it with dnf.

Hope this helps, and thanks again to everyone who helped.

2 Likes
14

If you have duplicates, you should try

sudo dnf  remove --duplicates

and run

sudo dnf check

to check if you have duplicates.

This can happen if for some reason the update process was not allowed to finished, for example due to power failure or battery ran empty.

15

Yes, you’re right. But I found that dnf remove --duplicates failed with the same “Problem: The operation would result in removing the following protected packages: systemd, systemd-udev
(try to add ‘–skip-broken’ to skip uninstallable packages)” that dnf distro-sync did.

The key was removing the problematic duplicates manually with rpm -e --nodeps ... before running either dnf remove --duplicates or dnf distro-sync.

And, of course, in my case, the update was not allowed to finish due to the “Oh no! something went wrong …” issue.

If this happened to me all over again, I’d do the manual removal with rpm -e --nodeps, followed by a dnf distro-sync followed by a dnf remove --duplicates.

1 Like
16

This doesn’t match your description of the events

So did you get “Oh no” during upgrade or after reboot? If the latter, the dnf upgrade should have been completed before rebooting.

17

I got “Oh no” at some point after issuing sudo dnf system-upgrade reboot command. I had left it while it was installing all the packages after the initial reboot and only saw the “Oh no” screen when I came back. I assumed it had rebooted automatically again but I wasn’t there to watch so I can’t know for sure.

18

Then the “Oh no” was cased by the failure of the upgrade and not the other way around. Perhaps the real cause could be seen in one of the dnf logs /var/log/dnf.rpm.log, /var/log/dnf.librepo.log or /var/log/dnf.log, but that might be like looking for the haystack with the needle. The output of dnf history info xxx where xxx is the transaction number of the upgrade. It can be found by running dnf history | more. I don’t know if this is worth the effort.