Unable to boot after an update

I updated my fedora 37 using “sudo dnf update”.

Everything seemed to go well, it finished updating and said completed, but i didn’t get the prompt to type a new command in the terminal which usually happens after updating.
I also wasn’t able to open any application after the update, and there was only a logout option in the power off menu.
The logout option didn’t do anything. so I had to perform a hard shutdown.
Now the system doesn’t boot from any of the available kernels.

They all get stuck after this message

[ OK ] Reached target initrd-switch-root.target - Switch Root.
Starting initrd-switch-root.service - Switch Root…
[ 5.606573] systemd-Journald[301]: Received SIGTERM from PID 1 (systemd).

Except on the latest kernel I get this .

I tried to chroot from a live environment, but I get the following error

[liveuser@localhost-live ~]$ sudo mount /dev/sda6 /mnt
[liveuser@localhost-live ~]$ sudo chroot /mnt
chroot: failed to run command ‘/bin/bash’: No such file or directory

I tried some solutions for this error, but nothing was successful.

Any help would be appreciated.

Just for future reference.
Never interrupt an update since it probably is doing something in the background and a forced power off may break the system totally if you interrupt a write process. I have had systems take 10 minutes or more to do the full update before control is returned to the user. The fact that the prompt did not return should have been your first clue the system was still processing the last command.

Your situation is a perfect example of what not to do, and why.

Fedora uses btrfs for the / file system and mounting is not as simple as mount /dev/sda6 /mnt

In the live image system run lsblk -f to see all the partitions and their UUIDs.
The partition that shows as btrfs on the system drive should contain both / and /home as btrfs subvolumes and you will need to make note of the UUID for that partition.
Then to mount the / file system the command would be
mount -t btrfs -o subvol=root,compress=zstd:1 UUID=<uuid seen above> /mnt which should then give a properly mounted root file system at /mnt.

You also cannot chroot into that file system before also mounting /proc, /run, /dev, and /sys as appropriate in the file system you now have at /mnt.
This link give the full process to properly enter a chroot environment to attempt recovery.

Note that the UUID on the mount command in step 4 should be the one you have on your system and found with the lsblk command above.

Thanks for the replay!

I don’t like updating with gnome-software because it’s really slow for me, sometimes it takes like 15min just to fetch updates even though I have decent internet connection speed.
But it seems safer so I’m gonna stick to it in the future.

My approach might be wrong here, I’m trying to reinstall the kernel.
I followed the steps you provided to chroot into the file system and got the following error:

[liveuser@localhost-live ~]$ su 
[root@localhost-live liveuser]# lsblk -f 
NAME        FSTYPE   FSVER LABEL                 UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0       squashfs 4.0                                                                             
loop1       ext4     1.0   Anaconda              8fe0120d-7a70-4e73-8185-f8521c2529e0                
├─live-rw   ext4     1.0   Anaconda              8fe0120d-7a70-4e73-8185-f8521c2529e0    1.4G    81% /
└─live-base ext4     1.0   Anaconda              8fe0120d-7a70-4e73-8185-f8521c2529e0                
└─live-rw   ext4     1.0   Anaconda              8fe0120d-7a70-4e73-8185-f8521c2529e0    1.4G    81% /
├─sda1      vfat     FAT32                       54BA-EB04                                           
├─sda3      ntfs                                 8ECEC03BCEC01CF9                                    
├─sda4      ntfs                                 62F23C26F23BFCBB                                    
├─sda5      ext4     1.0                         6370c3c1-d199-487f-b31d-1934752e89d7                
└─sda6      btrfs          fedora_localhost-live e445a94a-f877-4da7-b5e2-655dd8deb0c5                
├─sdb1      ntfs           Data                  7A7099F27099B4FB                                    
└─sdb2      ext4     1.0   FedoraHDD             8d6ed568-3490-4a7c-8c55-cb59b8abeaac                
├─sdc1      exfat    1.0   SK                    4E21-0000                                           
└─sdc2      vfat     FAT16 VTOYEFI               B228-8EFB                                           
zram0                                                                                                [SWAP]
[root@localhost-live liveuser]# mount -t btrfs -o subvol=root,compress=zstd:1 UUID=e445a94a-f877-4da7-b5e2-655dd8deb0c5 /mnt
[root@localhost-live liveuser]# for dir in sys proc run dev ; do mount --bind /$dir /mnt/$dir ; done
[root@localhost-live liveuser]# chroot /mnt
flatpak: error while loading shared libraries: /lib64/libzstd.so.1: file too short
[root@localhost-live /]# mount -a
[root@localhost-live /]# ls 
afs  bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  nix  opt  path  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@localhost-live /]# dnf reinstall kernel\*
Bus error (core dumped)

I’m not really sure where to go from here.

I just tried to check coredumpctl for core dumps and I got this error

[root@localhost-live /]# coredumpctl list
coredumpctl: error while loading shared libraries: /usr/lib/systemd/libsystemd-shared-251.11-2.fc37.so: file too short
[root@localhost-live /]# ls -l /usr/lib/systemd/libsystemd-shared-251.11-2.fc37.so
-rwxr-xr-x. 1 root root 0 Feb 7 18:05 /usr/lib/systemd/libsystemd-shared-251.11-2.fc37.so

I tried to replace the file but I got this error

[root@localhost-live systemd]# cp libsystemd-shared-251.11-2.fc37.so /mnt/usr/lib/systemd/
cp: error writing ‘/mnt/usr/lib/systemd/libsystemd-shared-251.11-2.fc37.so’: No space left on device

But I do have some free space

I think powering off your machine corrupted the file system.

In the screenshot above, look at

1.123966 VFS Cannot open root device UUID=…
1.133991 Unable to mount root file system or unknown block. 

The fastest and safest would be to reinstall Fedora.

In future, when upgrading with dnf, run it in a screen or tmux session. That way, when Gnome Shell crashes, it won’t take down the upgrade process. When running the dnf upgrade in gnome-terminal and that window crashes it interrupts the upgrade process and leaves the system in an undefined state.

Thanks for the replay!
Is it possible to reinstall fedora without losing any data?

Also worth mentioning, this error appears only when booting from the latest kernel.

1.123966 VFS Cannot open root device UUID=…
1.133991 Unable to mount root file system or unknown block.

When booting from older kernels I just get stuck here.

It is worth mentioning that a corrupted file system can do that. Your btrfs file system seems to be full even though it should not be. Powering off while the system is writing during an update can make the file system appear full with unlinked btree pointers.

I would concur with @augenauf in that a full reinstall into partitions sda5 & sda6 would be best.

You might be able to recover your home data by mounting that subvolume then copying all the content to external storage before the reinstall.

You could, while booted to the live install media (and not in the chroot environment), mount the /home volume with

[root@localhost-live liveuser]# mount -t btrfs -o subvol=home,compress=zstd:1 UUID=e445a94a-f877-4da7-b5e2-655dd8deb0c5 /mnt

then /mnt should contain the same as /home on your running system and the content could be backed up (read but not written to). Once that was done then using fdisk to delete /dev/sda5 and /dev/sda6 would free up the space to allow a reinstall.

1 Like

Alright I’ll do that.