Fixing half completed system upgrade

rabarbra · July 3, 2024, 10:17am

I was upgrading Fedora 38 to 40 and during the upgrade there was a “soft lockup” error and the upgrade seemed to be stuck. After waiting a while I decided to power off the system. The system booted fine, though the grub entry still shows f38. The about this system panel shows Fedora 40 but the kernel is reported to be 6.8.9-100.fc38.x86_64

I can log in, but dnf offline-upgrade seems to still look for f38 software (though only libdeflate is found for update. I had updated prior to upgrade so not surprising.). Discover shows Fedora 40 as the default source.

Additionally openssh fails to start the daemon due to the following

(sshd)[7242]: sshd.service: Referenced but unset environment variable evaluates to an empty string: OPTIONS
 sshd[7242]: OpenSSL version mismatch. Built against 30000090, you have 30200010
 systemd[1]: sshd.service: Main process exited, code=exited, status=255/EXCEPTION

I suspect this is due to incomplete updating. Is there a way to redo the system-upgrade or to at least figure out all the conflicts that are left?

ankursinha · July 3, 2024, 10:37am

This is tricky, because one does not know if the rpm databases etc. were updated at all to note what packages were installed/updated/removed.

I’d first do a:

rpm -qa | sort

to see if there are duplicate packages. That’ll give us some idea of where the upgrade failed.

One can try dnf distro-sync --releasever=40 to try and “sync” the packages, but that may or may not work depending on the state of the system.

Perhaps start with the distro-sync and see how that goes?

ankursinha · July 3, 2024, 10:37am

Added f40, upgrades

rabarbra · July 3, 2024, 11:36am

Thanks. The output of rpm -qa | sort seems to be mostly duplicates with only a few from f38 without a f40 equivalent.

Running dnf distro-sync results in a number of conflicting pacakges, mostly around sddm packages provided by Plasma 6.

Trying to run dnf distro-sync --allowerasing produces the error that this would remove dnf, grub2-tools-minimal, plasma-desktop, sudo, systemd, systemd-udev

Would --skip-broken be a good choice?

vekruse · July 3, 2024, 11:48am

It most likely would be, as first the new package versions are recorded in the rpm databas as the package are unpacked and installed. After all that, the information of the old versions are cleared from the database.

To check

dnf check --duplicates

To fix

dnf remove --duplicates

See man dnf for details of those commands.

ankursinha · July 3, 2024, 11:49am

Cool, so that indicates that most of the packages were updated where the F40 versions were installed, but the F38 versions weren’t uninstalled.

Could you list the output of distro-sync just so we can see exactly what packages are causing issues?

The last time I had to do this, I followed more of a “surgical” method where I manually dealt with various packages to ensure that they were updated to the F40 version and the F38 version removed.

I wouldn’t use skip-broken just yet. Ideally we should be able to fix all the packages, we’ll just have to do it bit by bit.

rabarbra · July 3, 2024, 11:52am

Here’s the full output

sudo dnf distro-sync --releasever=40
Last metadata expiration check: 0:06:28 ago on Wed 03 Jul 2024 01:28:19 PM CEST.
Error: 
 Problem 1: package mozjs78-78.15.0-10.fc38.x86_64 from @System requires libicui18n.so.72()(64bit), but none of the providers can be installed
  - package mozjs78-78.15.0-10.fc38.x86_64 from @System requires libicuuc.so.72()(64bit), but none of the providers can be installed
  - libicu-72.1-2.fc38.x86_64 from @System  does not belong to a distupgrade repository
  - problem with installed package mozjs78-78.15.0-10.fc38.x86_64
 Problem 2: problem with installed package sddm-x11-0.20.0-1.fc38.noarch
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.0.3-2.fc40.noarch from updates-modular
  - package sddm-wayland-plasma-6.0.3-2.fc40.noarch from updates-modular conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora-modular conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.0.3-2.fc40.noarch from updates-modular
  - package sddm-wayland-plasma-6.0.3-2.fc40.noarch from updates-modular conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora-modular
  - package sddm-wayland-plasma-6.0.3-2.fc40.noarch from updates-modular conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from updates-modular
  - package sddm-x11-0.21.0-4.fc40.noarch from updates-modular conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.0.3-2.fc40.noarch from updates-modular
  - problem with installed package sddm-wayland-plasma-6.1.1-1.fc40.noarch
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora-modular
  - package sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora-modular conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora
  - package sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora-modular conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora-modular
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora-modular conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora-modular
  - package sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora-modular conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from updates-modular
  - package sddm-x11-0.21.0-4.fc40.noarch from updates-modular conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora-modular
  - sddm-x11-0.20.0-1.fc38.noarch from @System  does not belong to a distupgrade repository
  - package sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora
  - package sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora-modular
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora-modular conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora
  - package sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from updates-modular
  - package sddm-x11-0.21.0-4.fc40.noarch from updates-modular conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.1.1-1.fc40.noarch from updates
  - package sddm-wayland-plasma-6.1.1-1.fc40.noarch from updates conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora-modular conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.1.1-1.fc40.noarch from updates
  - package sddm-wayland-plasma-6.1.1-1.fc40.noarch from updates conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora-modular
  - package sddm-wayland-plasma-6.1.1-1.fc40.noarch from updates conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from updates-modular
  - package sddm-x11-0.21.0-4.fc40.noarch from updates-modular conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.1.1-1.fc40.noarch from updates
  - package sddm-wayland-plasma-6.1.1-1.fc40.noarch from @System conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.1.1-1.fc40.noarch from @System
  - package sddm-wayland-plasma-6.1.1-1.fc40.noarch from @System conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora-modular
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora-modular conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.1.1-1.fc40.noarch from @System
  - package sddm-wayland-plasma-6.1.1-1.fc40.noarch from @System conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from updates-modular
  - package sddm-x11-0.21.0-4.fc40.noarch from updates-modular conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.1.1-1.fc40.noarch from @System
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)

rabarbra · July 3, 2024, 12:00pm

Ok, this shows what you describe, but dnf remove --duplicates fails with the error

Problem: The operation would result in removing the following protected packages: grub2-efi-x64, grub2-pc

Also the output lists many packages like the following Installed package tex-preview-13.3-3.fc40.noarch not available. does that indicate that these fc40 packages were not actually installed?

ankursinha · July 3, 2024, 12:11pm

Are you using the modular repos at all? Otherwise we can disable those and see how that goes?

Did you use --releasever=40 with your commands? Otherwise it’s trying to find the f40 package in the F39 repos.

Was fedora-release one of the duplicated packages? From what I remember that’s where dnf etc. get the releasever from.

rabarbra · July 3, 2024, 12:21pm

Ahh, yes --relesasever=40 was the issue with that, and yes fedora-release-{common,identity,kde} were duplicated.

I disabled fedora-modular and updates-modular, which shortens the output a bit, but this is now the output from distro-sync (remove --updates produces the same list of package conflicts)

Last metadata expiration check: 0:50:41 ago on Wed 03 Jul 2024 01:28:16 PM CEST.
Error: 
 Problem 1: package mozjs78-78.15.0-10.fc38.x86_64 from @System requires libicui18n.so.72()(64bit), but none of the providers can be installed
  - package mozjs78-78.15.0-10.fc38.x86_64 from @System requires libicuuc.so.72()(64bit), but none of the providers can be installed
  - libicu-72.1-2.fc38.x86_64 from @System  does not belong to a distupgrade repository
  - problem with installed package mozjs78-78.15.0-10.fc38.x86_64
 Problem 2: problem with installed package sddm-wayland-plasma-6.1.1-1.fc40.noarch
  - package sddm-wayland-plasma-6.1.1-1.fc40.noarch from @System conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.1.1-1.fc40.noarch from @System
  - package sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.0.3-2.fc40.noarch from fedora
  - package sddm-x11-0.21.0-4.fc40.noarch from fedora conflicts with sddm-greeter-displayserver provided by sddm-wayland-plasma-6.1.1-1.fc40.noarch from updates
  - package sddm-wayland-plasma-6.1.1-1.fc40.noarch from updates conflicts with sddm-greeter-displayserver provided by sddm-x11-0.21.0-4.fc40.noarch from fedora
  - problem with installed package sddm-x11-0.20.0-1.fc38.noarch
  - sddm-x11-0.20.0-1.fc38.noarch from @System  does not belong to a distupgrade repository
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)

rabarbra · July 3, 2024, 1:07pm

I manually removed sddm-x11 and mozjs78 and the problems with conflicting packages or unavailable dependencies go away, however, the problem with protected packages still remains

Problem: The operation would result in removing the following protected packages: dnf, grub2-tools-minimal, plasma-desktop, sudo, systemd, systemd-udev

ankursinha · July 3, 2024, 1:47pm

Let’s try and fix the fedora-release packages first so that the system at least tries to get packages for the correct Fedora version.

What does

sudo dnf --releasever=40 update fedora-release\*

say?

Also worth trying:

sudo dnf --releasever=40 reinstall fedora-release\*

(which may trick the system into reinstalling the f40 ones and uninstalling the f38 ones).

Yeh, this is because the deps are all messed up at the moment. It needs to remove some f38 packages, but they’re protected and it doesn’t know how to handle it. So we’ll have to do these bits manually.

Obligatory note: if you have a different /home partition or a recent backup of your user data, it’ll be quicker to fresh install and re-mount the partition/restore from your backup. Fixing the system is possible in theory, but one cannot say how much tinkering it’ll require.

rabarbra · July 3, 2024, 2:13pm

Does it make sense that running dnf remove --duplicates --releasever=40 would list all packages with the fc40 version replacing fc38 and the download size of 5.9GB of packages? (I would think these are already installed and thus the size would be smaller, but maybe that’s just the interface?) I haven’t run the process, as I was unsure.

sudo dnf --releasever=40 update fedora-release\* just said everything is up-to-date.

sudo dnf --releasever=40 reinstall fedora-release\* replaced the old f37-f38 version with f39-f40. However, (as expected?) distro-sync still errors with the protected packages problem.

I have everything in btrfs subvolumes inside of an encrypted luks container. Is it possible to reinstall just the system in this case or or will I need to repartition and then copy home and configs from backup after creating the users? I guess I would also need to determine all the extra software that I use which I have installed over the years.

Will flatpaks be preserved just by cloning home?

hamrheadcorvette · July 3, 2024, 2:32pm

Hey there, could you do us a favor, and provide the output of :

cat /etc/os-release && cat /etc/fedora-release in preformatted text </>.

If both point to 40 as the release, please do dnf -y update --allowerasing --best

ankursinha · July 3, 2024, 3:47pm

Cool.

Does running dnf commands without providing the releasever argument now look for f40 repos and all that?

Yeh, I think that’ll be expected. The man page says:

“To ensure the integrity of the system it reinstalls the newest package”.

Was that throwing any errors, otherwise, that’ll be one to run. It should remove all the f38 packages and re-install all the f40 ones ensuring that they’re correctly installed. That should clean up most of the mess.

I’m not sure about this—not on btrfs yet.

But yes, you’ll have to install the extra packages you collected over the years.

User installed flatpaks should be (ones installed with --user), but not systemwide ones I don’t think.

rabarbra · July 10, 2024, 1:27pm

After removing another package that was in conflict gcc-gdb... I was able to run remove --duplicates and distro-sync without any special options. This seems to have downloaded things, and openssh is running fine. However, the kernel is still stuck at 6.8.9-100.fc38.x86_64 even after explicitly installing 6.8.7-200.fc40 with dnf and running dracut.

Also after rebooting my normally 1920x1080 display only supports 640x480, but I haven’t really tried to solve this issue yet.

rabarbra · July 10, 2024, 1:29pm

cat /etc/os-release && cat /etc/fedora-release
NAME="Fedora Linux"
VERSION="40 (KDE Plasma)"
ID=fedora
VERSION_ID=40
VERSION_CODENAME=""
PLATFORM_ID="platform:f40"
PRETTY_NAME="Fedora Linux 40 (KDE Plasma)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:40"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f40/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=40
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=40
SUPPORT_END=2025-05-13
VARIANT="KDE Plasma"
VARIANT_ID=kde
Fedora release 40 (Forty)

As noted in my response to @Ankur, I have been able to run the dnf commands without any special options, the F40 packages seem to be installed, but the kernel is still stuck on fc38.

hamrheadcorvette · July 10, 2024, 3:44pm

Can you provide the output of dnf list installed kernel

If all you need to do is add the latest kernel then updating the kernel shouldn’t be a problem?

Topic		Replies	Views
Upgrade failed, mismatched packages \| F40 \| Ask Fedora upgrades , f40	6	431	October 26, 2024
Upgrade from 39 -> 41 Failure Ask Fedora installation , f39 , f41	5	103	February 26, 2025
Problem when upgrading from fedora 38 xfce to fedora 40 Ask Fedora f38 , upgrades , dnf , f40	33	952	May 26, 2024
My System Upgrade Seems Stuck Ask Fedora upgrades , f39 , f41	4	80	March 9, 2025
Reboot during F38->F39 upgrade, how to fix it? Ask Fedora f38 , f39	1	21136	February 18, 2024

Fixing half completed system upgrade

Related topics