Fedora 37, Login Loop after upgrade from Fedora 36

nguyenvinhlinh1 · December 10, 2022, 2:51pm

Hi there, I have just updated my fedora from Fedora 36 to Fedora 37. However, I cannot login in multi-user.target eventhough I did provide correct username & password for my personal user and root.

I used to think that it’s because of Selinux, I add enforcing=0 in the GRUB menu. But it does not work, that login loop still appears.
In addition, I also try rescue mode in the GRUB menu, it asked me root password, I am 100% sure that I provided correct password. but error happened.

/usr/sbin/sulogin terminated by signal SEGV

Please help me, it’s a really good chance that I can learn to recover similar RHEL7 & 8 issues in the future.

computersavvy · December 10, 2022, 3:03pm

If you did not use sudo as your normal user to create a password for the root account you will not be able to log in as root on fedora workstation. The default config is that the root account is locked.

You should be able to log in at the terminal window shown in your first image with your regular user account name and password

In emergency mode you MUST have a root password assigned or you will not be able to log in.

nguyenvinhlinh1 · December 10, 2022, 3:21pm

Hi, Thank for your reply

In the first image, I also did try to login with my regular user. The login loop still happend, in the recording, I only post login with root user.

nguyenvinhlinh1 · December 10, 2022, 3:37pm

my root account is enabled and I know its password well. In the image, I did enter password properly, but the output is

/usr/sbin/sulogin terminated by signal SEGV

After that, login screen come back, it ask me root password again.

computersavvy · December 10, 2022, 4:38pm

That is more helpful since we now know (most of) the full details.

I would guess that something happened during the upgrade to cause this, so could you please tell us how you did the upgrade and the details of how it failed if known.
Was the upgrade done using dnf or gnome-software? If using dnf was it from a terminal login or was it a terminal window from the desktop?
When did the failure occur?
What messages did you see on screen if any?

It seems quite possible that recovery may require booting to a live media then doing a recovery live in that environment. We need as much info as possible to determine how to proceed.

Can you boot with an older kernel? or do you not even see the grub menu?

nguyenvinhlinh1 · December 10, 2022, 6:13pm

I have tried booting with old kernel F35, F36 from GRUB menu. Login loop still happen.

Blockquote
Was the upgrade done using dnf or gnome-software? If using dnf was it from a terminal login or was it a terminal window from the desktop?

I was using DNF to upgrade from F36, referenced from https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/

This problem overall is really tricky cause I can’t read the log.

computersavvy · December 10, 2022, 7:21pm

That reference is the proper way to do an update, but you did not answer whether it was done from the desktop via a terminal window or while at a terminal login.

I assume you do see the grub menu since you said you tried an old kernel from F35 & F36.

This means that you should boot to a live media to move forward with recovery.
You can use either F36 or F37 install media on a USB to boot to the live system then mount the root file system of the currently installed OS. We will need to know what that file system is. Is it btrfs or ext4?
The command lsblk -f should give us the needed info for that once you are booted to the live media. Please post commands & outputs as preformatted text </>.

If you know that it is btrfs then you can use sudo btrfs subvolume list / to display all the subvolume names so we can move forward with that.

nguyenvinhlinh1 · December 11, 2022, 2:59am

Thank, let me try, I am at GMT+7 so I can’t reply that fast.

That reference is the proper way to do an update, but you did not answer whether it was done from the desktop via a terminal window or while at a terminal login.

I was done via terminal window, but before I run sudo dnf system-upgrade reboot , I change runlevel to multi-user.target with systemctl set-default multi-user.target

This means that you should boot to a live media to move forward with recovery.
You can use either F36 or F37 install media on a USB to boot to the live system then mount the root file system of the currently installed OS. We will need to know what that file system is. Is it btrfs or ext4?

As I remember it’s btrfs volume while installation. but I need to prepare live usb first. Plz w8!

nguyenvinhlinh1 · December 11, 2022, 5:20am

Hi, this is the command outputs.

$ lsblk -f

NAME        FSTYPE   FSVER            LABEL                 UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0       squashfs 4.0                                                                                        
loop1       ext4     1.0              Anaconda              8fe0120d-7a70-4e73-8185-f8521c2529e0                
├─live-rw   ext4     1.0              Anaconda              8fe0120d-7a70-4e73-8185-f8521c2529e0    1.8G    75% /
└─live-base ext4     1.0              Anaconda              8fe0120d-7a70-4e73-8185-f8521c2529e0                
loop2                                                                                                           
└─live-rw   ext4     1.0              Anaconda              8fe0120d-7a70-4e73-8185-f8521c2529e0    1.8G    75% /
sda         btrfs                     Disk_2                564ce8bd-1fc7-4d9e-9b94-556e549c4756                
sdb                                                                                                             
├─sdb1      vfat     FAT32                                  0684-45BC                             581.4M     3% /run/media/liveuser/0684-45BC
├─sdb2      ext4     1.0                                    c3917e42-d97e-4b7b-97f8-23d0079eae38  624.7M    29% /run/media/liveuser/c3917e42-d97e-4b7b-97f8-23d0079eae38
└─sdb3      btrfs                     fedora_localhost-live f3f27a6b-6113-48d0-9e4e-26cf67f89b77   11.7G    97% /run/media/liveuser/fedora_localhost-live
sdc         iso9660  Joliet Extension Fedora-WS-Live-37-1-7 2022-11-05-10-15-31-00                              
├─sdc1      iso9660  Joliet Extension Fedora-WS-Live-37-1-7 2022-11-05-10-15-31-00                     0   100% /run/initramfs/live
├─sdc2      vfat     FAT16            ANACONDA              7268-1544                                           
└─sdc3                                                                                                          
sr0                                                                                                             
zram0                                                                                                           [SWAP]

$ sudo  btrfs subvolume list /run/media/liveuser/fedora_localhost-live

Because of large output, I would like to share in gist. sudo btrfs subvolume list /run/media/liveuser/fedora_localhost-live · GitHub

computersavvy · December 12, 2022, 1:39am

I suggested that you run sudo btrfs subvolume list / so you could see all on the system. You limited that output to only the subvolumes at /run/media/liveuser/fedora_localhost-live which is not the entire system by any means.

However, I can see that /dev/sda is btrfs (currently unmounted and data not shown_
/dev/sdb1 is efi
/dev/sdb2 is ext4 (probably /boot)
/dev/sdb3 is btrfs, probably the the main OS volume, but the way it was mounted is not accessible.

The subvolume command shows that you have 438 subvolumes on /dev/sda3, almost all are at root/var/lib/docker/btrfs/subvolumes/ so you have ~435 docker subvolumes and from fdisk we can see that /dev/sda3 is 97% full of those docker subvolumes.

For recovery you will need to mount the main file system as follows in prep for a chroot into the installed OS.

Again boot to the live media
use the su command to gain root access
Unmount all you have mounted at /run/media/liveuser/
mount -t btrfs -o subvol=root,compress=zstd:1 UUID=f3f27a6b-6113-48d0-9e4e-26cf67f89b77 /mnt
for i in "/proc" "/sys" "/run" "/dev" ; do mount -o bind "$i" /mnt"$i" ; done
chroot /mnt
mount -a
now run the mount command to verify that everything is mounted properly. mount and look at the mount points /, /home, /boot, /boot/efi, /dev, /run, /proc, /sys. All should be listed and all should be as just done.

At this point you should be in the main installed OS as root and can do the recovery from having docker completely fill the root subvolume with docker subvolumes.

I do not work with docker so cannot help with that, but I am 100% certain that the reason you cannot log in regularly is the fact that the main OS has no free space in the file system and this gets you to a point where you can begin to clean it up.

I have no idea who may have been following this thread to this point, but there may be someone who is experienced with docker that can assist with a proper clean up of the runaway docker config and cleaning out all its subvolumes.

If you get no additional response in the next day then please open a new thread for assistance in cleaning up the docker processes and removing the hundreds of docker subvolumes. Be sure to reference this thread if a new one is started.

There is one thing that may recover some space for you.
dnf clean all to clean out all the normal dnf cache data.
dnf system-upgrade clean to clean out the cache data from the system upgrade.
I do not know how much room will be freed up, but if the output of df shows that / and /home have dropped below 90% then it is likely that you could reboot and log in normally (depending upon where in the upgrade it may have halted)

I just ran dnf clean all and freed up almost 20 GB of space from cached cruft of long term updates without manually cleaning it out.

Just as an FYI, whenever in a chroot environment the way to get back to the live media environment is the exit command.

nguyenvinhlinh1 · December 12, 2022, 4:29am

Thank you so much for it. I must be honest that did re-install the OS because of my project deadline.
But I really appriciate your precious information, I will try your practice eventhough on my cleaned OS installed, it’s a really good practice to debug when users failed to login even with root.

Can you please gimme your btc address, I would like to send you, it’s not much but all my honest work.

computersavvy · December 12, 2022, 4:03pm

I will repeat my earlier comment that it appears your system ran out of space because of an out-of-control docker config that filled the drive with docker subvolumes.

Please make sure that is fixed in the future so it does not happen again.

Yes, it is known, and has been that way for eons, that the system will prevent all logins if the file system fills up. It is up to the user to monitor the system and prevent this situation.

No rewards or recompense is expected. Please keep the btc for your own use.