If dnf is broken you can repair it by folloing those steps, then you can use dnf history rollback <transaction-id> , if you know which packages are bad dnf downgrade <package> or even do a dnf distro-sync for synchronizing your system with the latest versions of packages available in the current Fedora release.
There’s literally a bunch of stuff you can do to get your system back to the way it was.
If you roll back the system, you are taking it backwards to a known good update. So at that point it does not matter what or which is the issue, the updates revert to that time. dnf distro-sync brings you up to the current release of Fedora.
Please note: You are doing this in a Live Environment. So using the Live USB to mount your broken system, fixing dnf and continuing to fix your system with the Live USB’s dnf commands. So it would look something like this :
dnf --installroot=/mnt reinstall dnf
dnf --installroot=/mnt history rollback <transaction_ID>
journactl should have messages related to the lack of space with a high priority.
The first thing to do is to use the Live System to free up some space. Then you can try to rollback to a known good configuration. Your issue may have been fixed in a recent update, so once you get to a working configuration you can try updating again.
So, after manually removing some files, and have more that 20GB empty in total (one partition for /root and /home) I run:
liveuser@localhost-live:/run/media/liveuser/fedora_localhost-live/root/var/lib$ date && sudo dnf --installroot=/ distro-sync
and the message I get multiple times is
Error Summary
-------------
Disk Requirements:
At least 1254MB more space needed on the / filesystem.
Even after removing more files, the message remains…
Seems like DNF reads this disk space somehow differently than other tools like df
(obviously 20GB plus are more than enough for the above task and transactions, one 1GB + is required)
p.s. the dnf rollback was NOT possible, since the transaction 730 cannot be seen by the live USB now…
Where did you mount your / to? I do not see it here. If you properly mounted the / it should look like this dnf --installroot=/mnt/< path-to-my-/ > , the /mnt directory is commonly used as a safe mount location.
The command you are displaying is potentially showing you / of the Live USB to which does not have enough space.
Error Summary
-------------
Disk Requirements:
At least 1254MB more space needed on the / filesystem.
One of the risks when messing with filesystems is confusion over which device is used in a given operation. It can be helpful to consult the Gnome Disks GUI to double check command-line results.
I should also mention that running disks near capacity increases wear and/or decreases performance on both SSD’s and HDD’s and increases the risks associated with running out of space. You may want move the nearly full disk to an external case and install a larger SSD (but be aware that heat can be a concern for large SSD’s).
Hey @hamrheadcorvette indeed, I noticed that even some minute after my last post.
I was offline thus I did not reply earlier…
And I retried with the correct path, where the full command is
Of-course the disk-space issue was not there any more, since the one I mentioned referred to the live system trying to have the dnf distro-sync command for itself!!
Unfortunately due to the above confusion, I run the dnf history rollback 730 later than the dnf distro-sync
So now booting again with editing GRUB and adding option 3, the system will still not boot - the same services keep being failed.
Please also check the output of dnf history rollback 730
liveuser@localhost-live:/run/media/liveuser/fedora_localhost-live/root/var/lib$ date && sudo dnf --installroot=/ distro-sync
Error Summary
-------------
Disk Requirements:
At least 1254MB more space needed on the / filesystem.
liveuser@localhost-live:/run/media/liveuser/fedora_localhost-live/root$ date && sudo dnf --installroot=/run/media/liveuser/fedora_localhost-live/root history rollback 730
Sat Dec 30 01:27:07 PM EST 2023
Last metadata expiration check: 0:06:47 ago on Sat 30 Dec 2023 01:20:21 PM EST.
Transaction history is incomplete, before 733.
Transaction history is incomplete, after 732.
Error: The following problems occurred while running a transaction:
Cannot find rpm nevra "akmod-nvidia-3:545.29.06-1.fc39.x86_64".
Cannot find rpm nevra "filezilla-3.66.1-1.fc39.x86_64".
Cannot find rpm nevra "gjs-1.78.1-1.fc39.x86_64".
Cannot find rpm nevra "gstreamer1-plugins-bad-free-libs-1.22.8-1.fc39.x86_64".
Cannot find rpm nevra "gstreamer1-plugins-bad-free-opencv-1.22.8-1.fc39.x86_64".
Cannot find rpm nevra "gstreamer1-plugins-bad-free-1.22.8-1.fc39.x86_64".
Cannot find rpm nevra "gstreamer1-plugins-bad-freeworld-1:1.22.7-1.fc39.x86_64".
Cannot find rpm nevra "gstreamer1-plugins-ugly-free-1.22.8-1.fc39.x86_64".
Cannot find rpm nevra "gstreamer1-plugins-ugly-1:1.22.7-1.fc39.x86_64".
Cannot find rpm nevra "java-17-openjdk-headless-1:17.0.9.0.9-1.fc39.x86_64".
Cannot find rpm nevra "kmod-nvidia-6.6.8-200.fc39.x86_64-3:545.29.06-1.fc39.x86_64".
Cannot find rpm nevra "libde265-1.0.14-1.fc39.x86_64".
Cannot find rpm nevra "librados2-2:18.2.1-1.fc39.x86_64".
Cannot find rpm nevra "librbd1-2:18.2.1-1.fc39.x86_64".
Cannot find rpm nevra "libssh-config-0.10.6-1.fc39.noarch".
Cannot find rpm nevra "libssh-0.10.6-1.fc39.i686".
Cannot find rpm nevra "libssh-0.10.6-1.fc39.x86_64".
Cannot find rpm nevra "lynis-3.0.9-4.fc39.noarch".
Cannot find rpm nevra "mozjs115-115.4.0-1.fc39.x86_64".
Cannot find rpm nevra "php-cli-8.2.13-1.fc39.x86_64".
Cannot find rpm nevra "php-common-8.2.13-1.fc39.x86_64".
Cannot find rpm nevra "php-gd-8.2.13-1.fc39.x86_64".
Cannot find rpm nevra "php-pdo-8.2.13-1.fc39.x86_64".
Cannot find rpm nevra "php-process-8.2.13-1.fc39.x86_64".
Cannot find rpm nevra "php-xml-8.2.13-1.fc39.x86_64".
Cannot find rpm nevra "python-unversioned-command-3.12.1-1.fc39.noarch".
Cannot find rpm nevra "python3-devel-3.12.1-1.fc39.x86_64".
Cannot find rpm nevra "python3-libs-3.12.1-1.fc39.x86_64".
Cannot find rpm nevra "python3-tkinter-3.12.1-1.fc39.x86_64".
Cannot find rpm nevra "python3-3.12.1-1.fc39.x86_64".
Cannot find rpm nevra "qpdf-libs-11.6.3-1.fc39.x86_64".
Cannot find rpm nevra "qpdf-11.6.3-1.fc39.x86_64".
Cannot find rpm nevra "stress-ng-0.17.01-1.fc39.x86_64".
Cannot find rpm nevra "tigervnc-icons-1.13.1-9.fc39.noarch".
Cannot find rpm nevra "tigervnc-license-1.13.1-9.fc39.noarch".
Cannot find rpm nevra "tigervnc-server-minimal-1.13.1-9.fc39.x86_64".
Cannot find rpm nevra "tigervnc-1.13.1-9.fc39.x86_64".
Cannot find rpm nevra "tor-0.4.8.9-1.fc39.x86_64".
Cannot find rpm nevra "xorg-x11-drv-nvidia-kmodsrc-3:545.29.06-1.fc39.x86_64".
Cannot find rpm nevra "xorg-x11-drv-nvidia-libs-3:545.29.06-1.fc39.i686".
Cannot find rpm nevra "xorg-x11-drv-nvidia-libs-3:545.29.06-1.fc39.x86_64".
Cannot find rpm nevra "xorg-x11-drv-nvidia-3:545.29.06-1.fc39.x86_64".
liveuser@localhost-live:/run/media/liveuser/fedora_localhost-live/root1$ date && sudo dnf --installroot=/run/media/liveuser/fedora_localhost-live/root distro-sync
Sat Dec 30 01:27:44 PM EST 2023
Last metadata expiration check: 0:07:23 ago on Sat 30 Dec 2023 01:20:21 PM EST.
Dependencies resolved.
Nothing to do.
Complete!
Looks like the only last option is to downgrade dracut…
After that, I do not think the system is more recoverable… Thankfully since day 1 of the issue there is no data loss, and there is full bash terminal access…
I would boot to the live usb then mount the installed system under /mnt and use a chroot environment to recover.
su to become root mount <your root filesystem> /mnt to mount the device for i in sys proc run dev; do mount -o bind /$i /mnt/$i ; done to mount the needed portions of your live system. chroot /mnt to enter the mounted system as your own environment. mount -a to add the additional file systems that would normally be mounted on the running system.
At this point you could run the commands given as if you are running the actual installed system. The only relevant difference is that you would still be running the kernel from your live system which may interfere with certain commands, but not with most commands.
You are going to have to experiment a little here but, a good course of action would be to clean up some of the cache in hopes that helps.
Go back into your LiveUSB and mount your broken system.
dnf --installroot=/run/media/liveuser/fedora_localhost-live/ clean all
dnf --installroot=/run/media/liveuser/fedora_localhost-live/ distro-sync
That should get you back on track.
Please Note : You are mounting your / in a previous command I saw you had :
dnf –installroot=/run/media/liveuser/fedora_localhost-live/root distro sync
If you do this through the LiveUSB GUI and you are mounting / you should see your broken system folders under /run/media/liveuser/fedora_localhost-live/ and there is no need to point dnf to the /root directory. If you have doubts, and you used the GUI to mount the filesystem then you can check it through the GUI, and see all the folders under /.
The root of the filesystem / is not the same as /root
I had a similar, or maybe the same issue. After updating from 6.6.6 to 6.6.8, I stopped getting the encrypted password prompt. Luckily I was able to boot into the previous kernel w/o issue. I tried booting back & forth multiple times and 6.6.8 would fail each time with messages about systemd-udev failing to load.
I’m not sure which of these two fixed it for me but it was surely one of them …
for p in $(rpm -qa | grep 6\\.6\.8); do sudo dnf -y reinstall $p; done
sudo dnf list extras showed I had webex installed. It had custom files in /etc/udev/rules.d … so I removed the package.
Afterwards, my system booted 6.6.8-200 without issue.
@jtingiris The issue here is more so about running out of storage space for an update. The update then getting borked and going through mounting and updating the borked system from a LiveUSB.
The thread details this.
Two weeks later, issue not solved.
In my case, for some unexplained (?) reason, the actual folder on which the command worked was: /run/media/liveuser/fedora_localhost-live/root and NOT the plain /@hamrheadcorvette .
After that, for some reason I can only login again through terminal only with root, and not my user (user1)! This can be a deal breaker, since I need to run some commands as user1, then backup my system… and eventually re-install… It appears that the above dnf operations, affected the user1 (its permissions or maybe more things).
Running su - user1
gives me
su: warning: cannot change directory to /home/user1: Permission denied.
su: dailed to execute /bin/bash: Permission denied
Also I re-run dnf update an hour ago and the update worked fine. However the same services keep failing to start and GUI will not boot. Only root through bash terminal.
Any advice (or link to other post) on how to bring user1 back?
I still can access all files in the system including user1 files.
Based on a quite “official” blog on Linux permissions, my impression is that I should use chmod to restore the access rights of user1 to himself (while being logged in as root)…
