Fc37 is upgraded to fc39, but dnf is stuck at fc37

With what you posted it would appear there are duplicates, but they are not. Look at the size of the 2 different files named rpmdb.sqlite. One is about 200 MB and the other is about 330 MB with 2 days difference.

As far as your question about which files should be the actual files and which are not, this is what I see on my F39 system

 $ ls -alF /usr/lib/sysimage/rpm/
total 432700
drwxr-xr-x. 2 root root      4096 Feb  6 18:00 ./
drwxr-xr-x. 4 root root      4096 Jul 20  2023 ../
-rw-r--r--. 1 root root         0 Feb 25 15:49 .migratedb
-rw-r--r--. 1 root root 443039744 Apr 13 11:31 rpmdb.sqlite
-rw-r--r--. 1 root root     32768 Apr 13 21:48 rpmdb.sqlite-shm
-rw-r--r--. 1 root root         0 Apr 13 11:31 rpmdb.sqlite-wal
-rw-r--r--. 1 root root         0 May 13  2023 .rpm.lock


$ ls -alF /var/lib/rpm
lrwxrwxrwx. 1 root root 26 Apr 13  2023 /var/lib/rpm -> ../../usr/lib/sysimage/rpm/

I would guess that the rpmold.XXXX files in /usr/lib/sysimage/ are leftovers from incomplete or failed upgrades and that those may be the cause of the inconsistencies that you see.

Unless rpm is actually performing a task it should remove the .rpm.lock file so that indicates a failure to complete a task and the lock file was left because it failed to complete properly. That also would explain the rpmold.XXXX directories where the old database copies are placed during upgrades and removed once the upgrade completes.

I’ve only skimmed the thread, but I don’t think anyone has mentioned this: the release version is controlled by the fedora-release package (or fedora-release-workstation, etc.). Please check rpm -qa | grep fedora-release.

If you see both 37 and 39 versions of those packages, you can try manually removing the 37 ones, and then remove the rest of the duplicate packages by following the instructions in: Upgrading Fedora Using DNF System Plugin :: Fedora Docs

This seems to happen when the upgrade is interrupted.

1 Like

The upgrade happens in stages.

  • The new directory /usr/lib/sysimage/rpm is created with symlinks to the old rpm database files. The file /var/lib/rpm/.migratedb is created.
  • At the next boot, the service rpmdb-migrate.service is run to do the actual upgrade. This runs the script /usr/lib/rpm/rpmdb_migrate and the result is the symbolic links are replaced by real files and the old directory /var/lib/rpm is replaced by a symlink.

My suggestion is to move /var/lib/rpm to /var/lib/rpm.old and create the symlink like this

    ln -sfr /usr/lib/sysimage/rpm /var/lib/rpm
    touch /usr/lib/sysimage/rpm/.rpmdbdirsymlink_created

Then see what running

dnf distro-sync --releasever=39

would do.

By the way, the migration should already have happened in fc37.

rpm thinks that a fc37 package is installed, while in fact it came from fc39:

root@localhost:/home/gen# rpm -qil fedora-release-identity-matecompiz
Name        : fedora-release-identity-matecompiz
Version     : 37
Release     : 17
Architecture: noarch
Install Date: Mon 30 Oct 2023 12:36:14 AM EDT
Group       : Unspecified
Size        : 1482
License     : MIT
Signature   : RSA/SHA256, Thu 19 Oct 2023 03:32:26 PM EDT, Key ID f55ad3fb5323552a
Source RPM  : fedora-release-37-17.src.rpm
Build Date  : Thu 19 Oct 2023 01:13:57 PM EDT
Build Host  : buildvm-x86-23.iad2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://fedoraproject.org/
Bug URL     : https://bugz.fedoraproject.org/fedora-release
Summary     : Package providing the identity for Fedora MATE-Compiz Spin
Description :
Provides the necessary files for a Fedora installation that is identifying
itself as Fedora MATE-Compiz.
/usr/lib/os-release
/usr/lib/swidtag/fedoraproject.org/org.fedoraproject.Fedora-edition.swidtag

root@localhost:/home/gen# cat /usr/lib/os-release
NAME="Fedora Linux"
VERSION="39 (MATE-Compiz)"
ID=fedora
VERSION_ID=39
VERSION_CODENAME=""
PLATFORM_ID="platform:f39"
PRETTY_NAME="Fedora Linux 39 (MATE-Compiz)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:39"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=39
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=39
SUPPORT_END=2024-11-12
VARIANT="MATE-Compiz"
VARIANT_ID=matecompiz

Thanks! I made the symlinks.

Then see what running

dnf distro-sync --releasever=39

would do.

It would download and replace all 3500 packages. It would be the third or fourth time. I will do it hoping that at the end it will produce a correct database. BTW, I made a flash drive with fc39-live and found that there are no instructions for its use. But this is another story.

Thanks to all who gave their advice!

I ran $ dnf distro-sync --releasever=39. It replaced all 3500 packages, apparently from fc39 to the same fc39, but registered it properly in the database. The default release version has become 39.
NB: I have already did it a few days ago, to the same results. But after that I ran

$ dnf install remove-retired-packages
$ remove-retired-packages 37

After that, or some other manipulation I can not recall now, the default went back to 37. So, that time it did not work.
This time the /var/lib/rpm directory is a link to /usr/lib/sysimage/rpm. So, hopefully, the configuration will be stable.
The files indicating an uncompleted status .migratedb and .rpm.lock are still there:

gen@localhost:~$ ls -alF /usr/lib/sysimage/rpm/
total 384092
drwxr-xr-x. 2 root root      4096 Apr 14 14:04 ./
drwxr-xr-x. 7 root root      4096 Jul 20  2023 ../
-rw-r--r--. 1 root root         0 Apr 14 14:04 .migratedb
-rw-r--r--. 1 root root         0 Apr 14 12:28 .rpmdbdirsymlink_created
-rw-r--r--. 1 root root 393265152 Apr 14 14:11 rpmdb.sqlite
-rw-r--r--. 1 root root     32768 Apr 14 22:00 rpmdb.sqlite-shm
-rw-r--r--. 1 root root         0 Apr 14 14:11 rpmdb.sqlite-wal
-rw-r--r--. 1 root root         0 Apr 13 13:47 .rpm.lock

I think it is good enough for me at this time. I will be watching for other issues since the installation happened to be not fully consistent.
Once again, thank you all for your help !

The only one in that list that does not exist on my system is the .rpmdbdirsymlink_created file. All the others are in place on my system

I don’t know if remove-retired-packages can be trusted. I would prefer to use dnf list extras or dnf list --extras to get a list potentially retired and then decide for each of them what to do. For example kernel modules built by akmods would be on that list.

That file was suggested by Villy, probably as a note for oneself.

I believe my problems started when remove-retired-package 37 command removed something needed for X, causing the desktop to die which killed the scripts. The scripts did not finish properly, leaving an inconsistent configuration. I should have run these commands from the console, not from X. Perhaps, the scripts could have been made more robust.
In general, I had become too relaxed after a few smooth installations during the last years.

Linux is normally quite robust. It tracks whether a file is in use, and the contents of files are not removed while being used. I have seen errors when many “deleted” files are still in use and a user tries to replace them with new versions.