Fc37 is upgraded to fc39, but dnf is stuck at fc37

I have upgraded fc37 to fc39 following the standard instructions. Here is a history snapshot:

  985  dnf upgrade --refresh
  986  dnf system-upgrade download --releasever=39
  987  dnf system-upgrade reboot
  988  ls -alF /etc/yum.repos.d/ 
  989  ls -alFrt /etc/yum.repos.d/ 
  990  df
  991  man df
  992  df -T
  993  dnf install remove-retired-packages
  994  remove-retired-packages 37

It booted into a Gnome session. At the last command, at deleting various fc37 packages X suddenly died. I rebooted to no X. Using a console I typed startx. It opened a KDE session. It seems that removing a fc37 package damaged Gnome. I found that the running programs: the kernel, gcc, evince etc. belong to fc39. However, dnf and rpm refer to fc37. The following files all refer to fc39:
/etc/fedora-release ; /etc/redhat-release ; /usr/lib/os-release ; /usr/lib/fedora-release

Where does dnf take the release number 37 and how can I re-configure it to 39?
Help will be much appreciated. Thanks!

Where do you see the 37? Please show the command and its output.

Check the output:

hostnamectl status
dnf config-manager --dump-variables
dnf repolist
dnf repoquery --extras
1 Like

after you upgrade did you do sudo dnf distro-sync if not do it

1 Like

Hi folks,

Thanks for the replies!

I attach the results of the commands suggested by Vladislav:

xxx@localhost:~$ hostnamectl status
     Static hostname: localhost.localdomain
     Transient hostname: fedora
     Icon name: computer-desktop
     Chassis: desktop 🖥
     Machine ID: 9103d... skipped
     Boot ID: 7e280cb5...  skipped
     Operating System: Fedora Linux 39 (MATE-Compiz)   
     CPE OS Name: cpe:/o:fedoraproject:fedora:39
     OS Support End: Tue 2024-11-12
     OS Support Remaining: 7month 1d
     Kernel: Linux 6.8.4-200.fc39.x86_64
     Architecture: x86-64
     Hardware Vendor: Dell Inc.
     Hardware Model: Inspiron 3847
     ......
     
xxx@localhost:~$ dnf repolist
repo id                                                              repo name
Dropbox                                                              Dropbox Repository
adobe-linux-x86_64                                        Adobe Systems Incorporated
fedora                                                               Fedora 37 - x86_64
fedora-cisco-openh264                                 Fedora 37 openh264 (From Cisco) - x86_64
rpmfusion-free                                               RPM Fusion for Fedora 37 - Free
rpmfusion-free-updates                               RPM Fusion for Fedora 37 - Free - Updates
rpmfusion-nonfree                                        RPM Fusion for Fedora 37 - Nonfree
rpmfusion-nonfree-updates                        RPM Fusion for Fedora 37 - Nonfree - Updates
updates                                                           Fedora 37 - x86_64 - Updates

xxx@localhost:~$ dnf repoquery --extras
Adobe Systems Incorporated                                              
Dropbox Repository                                           
Fedora 37 - x86_64 - Updates                                                 
RPM Fusion for Fedora 37 - Free - Updates                
RPM Fusion for Fedora 37 - Nonfree - Updates                                                                                 
astronomy-bookmarks-0:1-23.fc32.noarch
xxdiff-0:4.0.1-10.fc37.x86_64
xxdiff-debuginfo-0:4.0.1-10.fc37.x86_64

**Command suggested by Igor ** I try dnf distro-sync on a package cheese:

root@localhost:/home/xxx# dnf list --releasever=39 cheese
Last metadata expiration check: 0:15:13 ago on Wed 10 Apr 2024 07:38:56 PM EDT.
Installed Packages
cheese.x86_64                                        2:43~alpha-2.fc37     @fedora                                                                    
Available Packages
cheese.x86_64                                        2:44.1-1.fc39                 fedora                                                                               
root@localhost:/home/xxx# cheese --version
Cheese 44.1
root@localhost:/home/gen# dnf distro-sync cheese
Last metadata expiration check: 1:12:23 ago on Wed 10 Apr 2024 06:44:49 PM EDT.
Dependencies resolved.
Nothing to do.
Complete!
root@localhost:/home/gen# dnf list cheese
Last metadata expiration check: 1:13:16 ago on Wed 10 Apr 2024 06:44:49 PM EDT.
Installed Packages
cheese.x86_64                                                                           2:43~alpha-2.fc37      @fedora                                                                     
root@localhost:/home/gen# cheese --version
Cheese 44.1

It looks like the cheese exacutable installed belongs to fc39, but dnf thinks that it belongs to fc37.
Perhaps, when X died on the remove-retired-packages 37 command, the command was killed and something was corrupted in the package database.
dnf distro-sync --releasever=39 cheese prompted a change of glibc and other libraries to fc39. I am wary at this time that it would crash the OS completely.

Try this way:

sudo dnf distro-sync --refresh --releasever 39 --allowerasing

I tried the suggested command to a bizarre effect.
# dnf distro-sync --refresh --releasever=39 cheese
It said that it replaced several fc37 packages by the fc39 ones. Indeed:

gen@localhost:~$ dnf list installed | grep fc39
cheese.x86_64                                     2:44.1-1.fc39                       @fedora                   
cheese-libs.x86_64                              2:44.1-1.fc39                       @fedora                   
glibc.i686                                              2.38-17.fc39                        @updates                  
glibc.x86_64                                         2.38-17.fc39                        @updates                  
glibc-all-langpacks.x86_64                 2.38-17.fc39                        @updates                  
glibc-common.x86_64                        2.38-17.fc39                        @updates                  
glibc-devel.x86_64                              2.38-17.fc39                        @updates                  
glibc-gconv-extra.i686                        2.38-17.fc39                        @updates                  
glibc-gconv-extra.x86_64                   2.38-17.fc39                        @updates                  
glibc-headers-x86.noarch                  2.38-17.fc39                        @updates                  
glibc-langpack-en.x86_64                   2.38-17.fc39                        @updates                  
glibc-langpack-ru.x86_64                    2.38-17.fc39                        @updates

Then I ran
# dnf distro-sync --refresh --releasever 39 --allowerasing

It has downloaded 3500 fc39 packages, ran transaction, cleanups etc. An finally it came to exactly the initial situation: dnf and rpm see no fc39 packages installed. They see only the fc37 packages. The versions of the executable programs: the running kernel, gcc etc belong to fc39.
I have not noticed any problem in /var/log/dnf.log (a piece is enclosed):

2024-04-11T21:27:49-0400 DEBUG Upgraded: xorriso-1.5.6-5.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Upgraded: xpdf-1:4.04-10.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Upgraded: xprop-1.2.5-4.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Upgraded: xrandr-1.5.2-3.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Upgraded: xrdb-1.2.2-1.fc39.x86_64
......
2024-04-11T21:27:49-0400 DEBUG Installed: vlc-plugin-lua-1:3.0.20-9.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Installed: vlc-plugin-notify-1:3.0.20-9.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Installed: vlc-plugin-pipewire-3-1.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Installed: vlc-plugin-visualization-1:3.0.20-9.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Installed: vlc-plugins-base-1:3.0.20-9.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Installed: vlc-plugins-freeworld-3.0.20-1.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Installed: vlc-plugins-video-out-1:3.0.20-9.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Installed: webkitgtk6.0-2.44.0-2.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Installed: xdg-desktop-portal-gnome-45.1-1.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Installed: xpdf-libs-1:4.04-10.fc39.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: kernel-6.3.8-100.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: kernel-6.5.7-100.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: kernel-core-6.3.8-100.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: kernel-core-6.5.7-100.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: kernel-modules-6.3.8-100.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: kernel-modules-6.5.7-100.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: kernel-modules-core-6.3.8-100.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: kernel-modules-core-6.5.7-100.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: kernel-modules-extra-6.5.7-100.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: python3-pyside2-1:5.15.7-1.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: python3-shiboken2-1:5.15.7-1.fc37.x86_64
2024-04-11T21:27:49-0400 DEBUG Removed: python3-slip-0.6.4-28.fc37.noarch
2024-04-11T21:27:49-0400 DEBUG Removed: python3-slip-dbus-0.6.4-28.fc37.noarch
2024-04-11T21:27:50-0400 INFO Complete!
2024-04-11T21:27:50-0400 DDEBUG Cleaning up.
2024-04-11T21:27:50-0400 DDEBUG /var/cache/dnf/updates-7da5a72b0a306ada/packages/corosynclib-3.1.8-1.fc39.x86_64.rpm removed
2024-04-11T21:27:50-0400 DDEBUG /var/cache/dnf/updates-7da5a72b0a306ada/packages/gupnp-1.6.6-1.fc39.x86_64.rpm removed
2024-04-11T21:27:50-0400 DDEBUG /var/cache/dnf/updates-7da5a72b0a306ada/packages/kf5-pimcommon-akonadi-23.08.5-1.fc39.x86_64.rpm removed

I ran the cheese replacement again, and after that dnf reported again that cheese, glibc and a few other libraties come from fc39. Somehow the change of everything restored the obsolete fc37 reference, but the change of one package produced the correct one.
One thing has been fixed as a result of this exercise - Gnome started working again. Still, no X window manager on reboots, though it seems that the level 5 is set.

That last list shows a lot of DEBUG entries. which really is not ideal for an regular user.

Do you by chance have the kernel-debug (and variants) installed?

Please show the result of dnf list installed | grep debug

I have only a few debug packages:

root@localhost:/home/gen# dnf list installed | grep -i debug
debugedit.x86_64                                                5.0-7.fc37                        @updates                  
elfutils-debuginfod-client.x86_64                 0.189-3.fc37                        @updates                  
elfutils-debuginfod-client.i686                      0.190-1.fc37                        @updates                  
elfutils-debuginfod-client.x86_64                 0.190-1.fc37                        @updates                  
perl-debugger.noarch                                  1.60-494.fc37                        @updates                  
xxdiff-debuginfo.x86_64                               4.0.1-10.fc37                       @@commandline             
xxdiff-debugsource.x86_64                          4.0.1-10.fc37                       @@commandline             

dnf log is very verbose and adds this word DEBUG or DDEBUG to about half of all lines. I do not know what it means. Perhaps, it is a level of logging. It is not a new feature - I now see the same thing in the logs from 2019.

I noticed that rpm --rebuilddb produced an error, and a window message flashed saying something about selinux. I found a note about such a case
https://discussion.fedoraproject.org/t/problem-with-rpm-rebuilddb-after-dnf-system-upgrade-from-f35-to-f36/78152
The installation manual recommends to relabel selinux by
fixfiles -B onboot and reboot. I did that and without any other action, miraculously, rpm and dnf started seeing fc39 as the default release. They showed that about 15% of packages still came from fc37. I ran again:

      # remove-retired-packages 37  # - removed a few
      # dnf distro-sync

These commands seemed to be doing right things, installing fc39 and removing fc37. BUT after that dnf and rpm went again to the fc37 default, showing all the installed packages as coming from fc37, while they (at least many of them) come from fc39. I tried
# rpm --rebuilddb - it did not show errors, but did not help either.

After doing all the above then try the sudo dnf distro-sync --refresh command again to verify the system is fully up to date with f39.

root@localhost:/home/gen# dnf distro-sync --refresh
Adobe Systems Incorporated                                                                                                   8.6 kB/s | 2.9 kB     00:00    
Dropbox Repository                                                                                                            13 kB/s | 1.5 kB     00:00    
Fedora 37 - x86_64                                                                                                            63 kB/s |  24 kB     00:00    
Fedora 37 openh264 (From Cisco) - x86_64                                                                                      12 kB/s | 989  B     00:00    
Fedora 37 - x86_64 - Updates                                                                                                 201 kB/s |  23 kB     00:00    
RPM Fusion for Fedora 37 - Free                                                                                               10 kB/s | 3.6 kB     00:00    
RPM Fusion for Fedora 37 - Free - Updates                                                                                     26 kB/s | 3.3 kB     00:00    
RPM Fusion for Fedora 37 - Nonfree                                                                                            18 kB/s | 6.8 kB     00:00    
RPM Fusion for Fedora 37 - Nonfree - Updates                                                                                  16 kB/s | 6.3 kB     00:00    
Error: 
 Problem: The operation would result in removing the following protected packages: dnf
(try to add '--skip-broken' to skip uninstallable packages)

I am not sure what it may do. dnf and rpm (the database in /etc/lib/sysimage/) think that all the packages belong to fc37. Would it remove all them? There is no need to remove dnf, since it comes from fc39. Another option:
# dnf distro-sync --refresh --releasever=39
suggested to upgrade everything:

Transaction Summary
===================================
Install   168 Packages
Upgrade  3721 Packages
Remove     13 Packages

Total download size: 4.6 G
Is this ok [y/N]: n
Operation aborted.
Total download size: 4.6 G
Is this ok [y/N]: n
Operation aborted.

I have done it once (described above) but the database returned to the wrong state.
Again, all (or nearly all) the installed packages come from fc39. The database is confused.

Another observation described above: after “replacing” a particular package with:
# dnf distro-sync --refresh --releasever=39 cheese
the database remembers that this package comes from fc39.

I think you meant /lib/sysimage/. Here:

% ls -l /lib/sysimage/rpm
total 12
lrwxrwxrwx. 1 root root 36 Mar 22 10:27 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite
lrwxrwxrwx. 1 root root 40 Mar 22 10:27 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm
lrwxrwxrwx. 1 root root 40 Mar 22 10:27 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal
% ls /var/lib/rpm/
rpmdb.sqlite  rpmdb.sqlite-shm  rpmdb.sqlite-wal
% ls -l /var/lib/rpm/
total 396836
-rw-r--r--. 1 root root 406327296 Apr 12 18:33 rpmdb.sqlite
-rw-r--r--. 1 root root     32768 Apr 13 14:36 rpmdb.sqlite-shm
-rw-r--r--. 1 root root         0 Apr 12 18:33 rpmdb.sqlite-wal

The behaviour suggests the `releasever` "default" value is still at 37.  Here:

% python3 -c ‘import dnf, json; db = dnf.dnf.Base(); print(json.dumps(db.conf.substitutions, indent=2))’
{
“arch”: “x86_64”,
“basearch”: “x86_64”,
“releasever”: “39”,
“releasever_major”: “39”,
“releasever_minor”: “”
}

This looks like the rpm database has not been proper migrated.

/var/lib/rpm should be a symbolic link to ../../usr/lib/sysimage/rpm and the files in /usr/lib/sysimage/rpm should be real files and not symbolic links.

1 Like

Can you please try this :

cat /etc/os-release

cat /etc/fedora-release

If it is Fedora 37 then try :

vim /etc/os-release and change VERSION_ID=39

After this you should update the system dnf -y distro-sync

They all refer to Fedora 39.

1 Like

:thinking:

@eugen55
I am really short on time right now, I will try my best to get back to you in about an hour. But this is all fixable.

Just out of curiosity, could you try

dnf clean all then
dnf updgrade --allowerasing --skip-broken --best --releasever=39 ??

sorry i deleted this twice.

Thanks for the suggestions!
Indeed, something is fishy with the locations of the database.
I have:

root@localhost:/home/gen# ls -alF /usr/lib/sysimage/rpm/
total 194924
drwxr-xr-x. 2 root root      4096 Apr 13 13:47 ./
drwxr-xr-x. 7 root root      4096 Apr 13 01:44 ../
-rw-r--r--. 1 root root 199557120 Apr 13 13:47 rpmdb.sqlite
-rw-r--r--. 1 root root     32768 Apr 13 19:28 rpmdb.sqlite-shm
-rw-r--r--. 1 root root         0 Apr 13 13:47 rpmdb.sqlite-wal
-rw-r--r--. 1 root root         0 Apr 13 13:47 .rpm.lock
root@localhost:/home/gen# ls -alF /var/lib/rpm
rpm/       rpm-state/ 
root@localhost:/home/gen# ls -alF /var/lib/rpm/
total 326984
drwxr-xr-x.  2 root root      4096 Feb 26  2023 ./
drwxr-xr-x. 77 root root      4096 Jul 20  2023 ../
-rw-r--r--.  1 root root         0 Apr 13 01:11 .migratedb
-rw-r--r--.  1 root root 334786560 Apr 11 21:35 rpmdb.sqlite
-rw-r--r--.  1 root root     32768 Apr 13 01:26 rpmdb.sqlite-shm
-rw-r--r--.  1 root root         0 Apr 11 21:35 rpmdb.sqlite-wal
-rw-r--r--.  1 root root         0 Feb 26  2023 .rpm.lock

So, the database is duplicated - no symlinks. About the dates: I started the fc39 installation on Apr 6th. In /usr/lib/sysimage there are older versions:

root@localhost:/home/gen# ls -alF /usr/lib/sysimage/
total 68
drwxr-xr-x.  7 root root  4096 Apr 13 01:44 ./
dr-xr-xr-x. 89 root root 40960 Apr 13 01:11 ../
drwxr-xr-x.  2 root root  4096 Apr 13 13:47 rpm/
drwxr-xr-x.  2 root root  4096 Feb  6 19:00 rpmold.17976/
drwxr-xr-x.  2 root root  4096 Feb  6 19:00 rpmold.29940/
drwxr-xr-x.  2 root root  4096 Nov 13 10:51 rpmold.4015460/
drwxr-xr-x.  2 root root  4096 Feb  6 19:00 rpmold.97571/
root@localhost:/home/gen# ls -alF /usr/lib/sysimage/rpmold.17976/
total 8
drwxr-xr-x. 2 root root 4096 Feb  6 19:00 ./
drwxr-xr-x. 7 root root 4096 Apr 13 01:44 ../
lrwxrwxrwx. 1 root root   34 Apr 13 01:09 .migratedb -> ../../../../var/lib/rpm/.migratedb
lrwxrwxrwx. 1 root root   36 Apr 13 01:09 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite
lrwxrwxrwx. 1 root root   40 Apr 13 01:09 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm
lrwxrwxrwx. 1 root root   40 Apr 13 01:09 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal
lrwxrwxrwx. 1 root root   33 Apr 13 01:09 .rpm.lock -> ../../../../var/lib/rpm/.rpm.lock
root@localhost:/home/gen# ls -alF /usr/lib/sysimage/rpmold.97571/
total 8
drwxr-xr-x. 2 root root 4096 Feb  6 19:00 ./
drwxr-xr-x. 7 root root 4096 Apr 13 01:44 ../
lrwxrwxrwx. 1 root root   34 Apr 11 20:59 .migratedb -> ../../../../var/lib/rpm/.migratedb
lrwxrwxrwx. 1 root root   36 Apr 11 20:59 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite
lrwxrwxrwx. 1 root root   40 Apr 11 20:59 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm
lrwxrwxrwx. 1 root root   40 Apr 11 20:59 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal
lrwxrwxrwx. 1 root root   33 Apr 11 20:59 .rpm.lock -> ../../../../var/lib/rpm/.rpm.lock
root@localhost:/home/gen# ls -alF /usr/lib/sysimage/rpmold.4015460/
total 8
drwxr-xr-x. 2 root root 4096 Nov 13 10:51 ./
drwxr-xr-x. 7 root root 4096 Apr 13 01:44 ../
lrwxrwxrwx. 1 root root   34 Nov 30 20:13 .migratedb -> ../../../../var/lib/rpm/.migratedb
lrwxrwxrwx. 1 root root   36 Nov 30 20:13 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite
lrwxrwxrwx. 1 root root   40 Nov 30 20:13 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm

The symlinks in the older databases’ directories all refer to the same files /var/lib/rpm/. In one directory (rpmold.4015460) the links were created before the fc39 upgrade. Others seem to be created during and after the change. What is the right place for the symlinks - in /usr/lib/sysimage/rpm or in /var/log/rpm ? The files rpmdb.sqlite in these two places are different. One in /var is bigger and older, but both were created after the upgrade.

Indeed, it looks like some installation scripts have failed to complete. I wonder it it can be fixed or it would be better to install fc39 as a new system. It has been a while since I did it the last time, configuring the software raid etc.

Thanks! I added a lengthy comment on this matter.

Thanks! I added a lengthy comment on this matter. The first question is in what directory the symlinks should stay in fc39.