Silverblue won't boot after forced shutdown, searching for ways to recover it

Hello,

Today, after my Silverblue install was suddenly unable to launch any flatpak, I tried rebooting it, but it hang. Being in a hurry because I had to join a meeting, I forced the shutdown by keeping the power button down.

That was an extremely bad idea, as after typing my passphrase in order to unlock my LUKS-encrypted drive, I was met with an emergency shell.

/run/initramfs/rdsosreport.txt indicates that the LUKS partition was successfully decrypted, but then the OSTree could not be found (file edited to only keep the relevant parts)

...
[   10.874461] bluefin systemd-cryptsetup[712]: Set cipher aes, mode xts-plain64, key size 512 bits for device /dev/disk/by-uuid/6d7d0a53-3c12-46e3-a03d-e877a29d586d.
[   12.825407] bluefin systemd[1]: Found device dev-disk-by\x2duuid-5eb69e8f\x2d333f\x2d4119\x2d877e\x2dbb86bba3a213.device - /dev/disk/by-uuid/5eb69e8f-333f-4119-877e-bb86bba3a213.
[   12.825811] bluefin systemd[1]: Reached target initrd-root-device.target - Initrd Root Device.
[   12.826958] bluefin systemd[1]: Finished systemd-cryptsetup@luks\x2d6d7d0a53\x2d3c12\x2d46e3\x2da03d\x2de877a29d586d.service - Cryptography Setup for luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d.
[   12.828143] bluefin systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes.
[   12.828177] bluefin systemd[1]: Reached target sysinit.target - System Initialization.
[   12.828200] bluefin systemd[1]: Reached target basic.target - Basic System.
[   12.832444] bluefin systemd[1]: Finished dracut-initqueue.service - dracut initqueue hook.
[   12.833296] bluefin systemd[1]: Reached target remote-fs-pre.target - Preparation for Remote File Systems.
[   12.833336] bluefin systemd[1]: Reached target remote-cryptsetup.target - Remote Encrypted Volumes.
[   12.833360] bluefin systemd[1]: Reached target remote-fs.target - Remote File Systems.
[   12.834085] bluefin systemd[1]: Starting dracut-pre-mount.service - dracut pre-mount hook...
[   12.858135] bluefin systemd[1]: Finished dracut-pre-mount.service - dracut pre-mount hook.
[   12.859250] bluefin systemd[1]: Starting systemd-fsck-root.service - File System Check on /dev/disk/by-uuid/5eb69e8f-333f-4119-877e-bb86bba3a213...
[   12.877072] bluefin systemd[1]: Finished systemd-fsck-root.service - File System Check on /dev/disk/by-uuid/5eb69e8f-333f-4119-877e-bb86bba3a213.
[   12.878397] bluefin systemd[1]: Mounting sysroot.mount - /sysroot...
[   12.890036] bluefin kernel: BTRFS: device label bluefin_fedora devid 1 transid 91961 /dev/mapper/luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d (252:0) scanned by mount (874)
[   12.890082] bluefin kernel: BTRFS info (device dm-0): first mount of filesystem 5eb69e8f-333f-4119-877e-bb86bba3a213
[   12.890102] bluefin kernel: BTRFS info (device dm-0): using crc32c (crc32c-x86_64) checksum algorithm
[   12.890117] bluefin kernel: BTRFS info (device dm-0): using free-space-tree
[   12.910877] bluefin systemd[1]: Mounted sysroot.mount - /sysroot.
[   12.911831] bluefin systemd[1]: Starting ostree-prepare-root.service - OSTree Prepare OS/...
[   12.933121] bluefin ostree-prepare-root[891]: ostree-prepare-root: Couldn't find specified OSTree root '/sysroot//ostree/boot.0/default/dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/0': No such file or directory
[   12.933952] bluefin systemd[1]: ostree-prepare-root.service: Main process exited, code=exited, status=1/FAILURE
[   12.934071] bluefin systemd[1]: ostree-prepare-root.service: Failed with result 'exit-code'.
[   12.934251] bluefin systemd[1]: Failed to start ostree-prepare-root.service - OSTree Prepare OS/.
[   12.934333] bluefin systemd[1]: ostree-prepare-root.service: Triggering OnFailure= dependencies.
[   12.935086] bluefin systemd[1]: Stopped target remote-cryptsetup.target - Remote Encrypted Volumes.
...

I first tried to decrypt the partition again from the both emergency and recovery shells but failed, so I booted a Silverblue LiveUSB in Troubleshoot mode, where it automatically decrypted the partition but could not find a Linux system there (No Linux systems found).

With some fiddling I was able to manually decrypt and mount the partition

$ cryptsetup open --type luks /dev/nvm0n1p3 luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d
...

$ cryptsetup status luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d
/dev/mapper/luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d is active.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device:  /dev/nvme0n1p3
  sector size:  512
  offset:  32768 sectors
  size:    496756736 sectors
  mode:    read/write

$ btrfs check /dev/mapper/luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d
UUID: 5eb69e8f-333f-4119-877e-bb86bba3a213
found 103380017152 bytes used, no error found
total csum bytes: 82616628
total tree bytes: 3147022336
total fs tree bytes: 2889121792
total extent tree bytes: 146046976
btree space waste bytes: 632915339
file data blocks allocated: 177383219200
 referenced 135478329344

$ mkdir -p /tmp/mnt

$ mount /dev/mapper/luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d /tmp/mnt

$ ls /tmp/mnt
home root var

From there, I don’t know how to proceed to recover my system. Listing /root/ostree (mounted to /tmp/mnt/root/ostree) lists the following folders boot.1 boot.1.1 deploy lock repo while btrfs subvolume list -o /tmp/mnt does not list any snapshot.

$ btrfs subvolume list -o /tmp/mnt/
ID 256 gen 91963 top level 5 path var
ID 257 gen 91958 top level 5 path home
ID 258 gen 91964 top level 5 path root

$ btrfs subvolume list -os /tmp/mnt/
[empty output]

What steps should I take now to keep diagnosing the issue and possibly fix it?

Thanks in advance,
Axel

(Note: I already tried booting the ostree entry 1 in GRUB with the same result)

You can try creating a symlink that points boot.0 to boot.1.

Hello,

Unfortunately doing so leads to the same outcome:

[    9.941348] bluefin systemd[1]: Starting ostree-prepare-root.service - OSTree Prepare OS/...
[    9.954783] bluefin ostree-prepare-root[884]: ostree-prepare-root: Couldn't find specified OSTree root '/sysroot//ostree/boot.0/default/dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/0': No such file or directory
[    9.955850] bluefin systemd[1]: ostree-prepare-root.service: Main process exited, code=exited, status=1/FAILURE
[    9.955983] bluefin systemd[1]: ostree-prepare-root.service: Failed with result 'exit-code'.
[    9.956176] bluefin systemd[1]: Failed to start ostree-prepare-root.service - OSTree Prepare OS/.
[    9.956257] bluefin systemd[1]: ostree-prepare-root.service: Triggering OnFailure= dependencies.

I checked after booting into recovery mode that the symlink is there (turns out the LUKS partition is left open in recovery mode, all I had to do was to mount it somewhere):

$ ls -l /tmp/mnt/root/ostree
total 8
lrwxrwxrwx. 1 root root  6 Jun 10 19:04 boot.0 -> boot.1
lrwxrwxrwx. 1 root root  8 Jun  2 17:08 boot.1 -> boot.1.1
drwxr-xr-x. 1 root root 26 Jun 10 19:02 boot.1.1
drwxr-xr-x. 1 root root 14 Jun 10 07:44 deploy
-rw-------. 1 root root  0 Jun 10 12:02 lock
drwxr-xr-x. 1 root root 80 Jun 10 07:44 repo

Actually, it fails because ostree-prepare-root is looking for /sysroot//ostree/boot.0/default/dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/0 where the ostree/boot.1.1 containes other trees

$ ls -l /tmp/mnt/root/ostree/boot.1.1/default
total 0
drwxr-xr-x. 1 root root 2 Jun  2 17:08 10c83f4b746fb3e785540edbfd4c85c350246f3b40f2d00424b0125c5168c1ac
drwxr-xr-x. 1 root root 2 Jun  2 17:08 87261a5fd4130582614ab002893ffba7ca3bfadba54b784ba65ec3c0dfbebfd6

I was about to write that I should focus on getting boot.1 to boot when I restarted and tried to boot it again just out of sanity… and to my surprise it booted!

Now I have to fix OSTree, because rpm-ostree status errors out.

$ rpm-ostree status
error: Loading sysroot: Failed to invoke RegisterClient: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Could not activate remote peer 'org.projectatomic.rpmostree1': startup job failed

I checked rpm-ostreed.service’s logs and found the following:

-- Boot 715a5f1ad56745a49501ba17ac74ae75 --
juin 10 22:19:24 axel-EliteBook-845 systemd[1]: Starting rpm-ostreed.service - rpm-ostree System Management Daemon...
juin 10 22:19:24 axel-EliteBook-845 rpm-ostree[1912]: Reading config file '/etc/rpm-ostreed.conf'
juin 10 22:19:24 axel-EliteBook-845 rpm-ostree[1912]: error: Couldn't start daemon: Error setting up sysroot: loading sysroot: Parsing deployment /ostree/boot.0/default/dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/0 in stateroot 'default': readlinkat: Aucun fichier ou dossier de ce nom
juin 10 22:19:24 axel-EliteBook-845 systemd[1]: rpm-ostreed.service: Main process exited, code=exited, status=1/FAILURE
juin 10 22:19:24 axel-EliteBook-845 systemd[1]: rpm-ostreed.service: Failed with result 'exit-code'.
juin 10 22:19:24 axel-EliteBook-845 systemd[1]: Failed to start rpm-ostreed.service - rpm-ostree System Management Daemon.

I tried cleaning things up by booting back into recovery mode and removing the symlink, but after that boot.1 would no longer boot until I re-create the symlink. So it definitely helped.

I’ll take a break for now as I have a semi-functional system, but I’ll definitely need to cleanup the mess in order to get rpm-ostreed.service to start.

So, we would need the output of:

$ ls /boot/ostree
$ cat /boot/loader/entries/*
$ ls /sysroot/ostree/boot.0/fedora/
$ ls /sysroot/ostree/deploy/fedora/deploy/

and then we’ll try to re-align things manually.

It is really weird that this happened.

Thanks for taking your time to help me!

$ ls /boot/ostree
default-10c83f4b746fb3e785540edbfd4c85c350246f3b40f2d00424b0125c5168c1ac
default-87261a5fd4130582614ab002893ffba7ca3bfadba54b784ba65ec3c0dfbebfd6
default-dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b
initramfs-overlays

$ cat /boot/loader/entries/*
title Bluefin (Version: 42.20250601.1 / FROM Fedora Silverblue 42) (ostree:1)
version 1
options rd.luks.uuid=luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d rhgb quiet root=UUID=5eb69e8f-333f-4119-877e-bb86bba3a213 rootflags=subvol=root rw ostree=/ostree/boot.0/default/87261a5fd4130582614ab002893ffba7ca3bfadba54b784ba65ec3c0dfbebfd6/0
linux /ostree/default-87261a5fd4130582614ab002893ffba7ca3bfadba54b784ba65ec3c0dfbebfd6/vmlinuz-6.14.5-300.fc42.x86_64
initrd /ostree/default-87261a5fd4130582614ab002893ffba7ca3bfadba54b784ba65ec3c0dfbebfd6/initramfs-6.14.5-300.fc42.x86_64.img
initrd /ostree/initramfs-overlays/30a9147f6fb5e2cd4fcaa3855a7362a82b3c3ed97141c352b35de019edd66058.img
aboot /ostree/deploy/default/deploy/869d176689a447c659e24b538876f528e495d9354acaeb96188b3e97311cfb7b.0/usr/lib/ostree-boot/aboot.img
abootcfg /ostree/deploy/default/deploy/869d176689a447c659e24b538876f528e495d9354acaeb96188b3e97311cfb7b.0/usr/lib/ostree-boot/aboot.cfg
title Bluefin (Version: 42.20250608 / FROM Fedora Silverblue 42) (ostree:0)
version 2
options rd.luks.uuid=luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d rhgb quiet root=UUID=5eb69e8f-333f-4119-877e-bb86bba3a213 rootflags=subvol=root rw ostree=/ostree/boot.0/default/dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/0
linux /ostree/default-dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/vmlinuz-6.14.5-300.fc42.x86_64
initrd /ostree/default-dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/initramfs-6.14.5-300.fc42.x86_64.img
initrd /ostree/initramfs-overlays/30a9147f6fb5e2cd4fcaa3855a7362a82b3c3ed97141c352b35de019edd66058.img
aboot /ostree/deploy/default/deploy/55626fb992f8da1a1072e3e2705a2ceea79fc0bda44bef2a3dc8b1cbceb18cb4.0/usr/lib/ostree-boot/aboot.img
abootcfg /ostree/deploy/default/deploy/55626fb992f8da1a1072e3e2705a2ceea79fc0bda44bef2a3dc8b1cbceb18cb4.0/usr/lib/ostree-boot/aboot.cfg

$ ls /sysroot/ostree/boot.0/fedora/
ls: impossible d'accéder à '/sysroot/ostree/boot.0/fedora/': Aucun fichier ou dossier de ce nom

$ ls /sysroot/ostree/deploy/fedora/deploy/
ls: impossible d'accéder à '/sysroot/ostree/deploy/fedora/deploy/': Aucun fichier ou dossier de ce nom

Ah sorry, I hadn’t realized that you were on Bluefin. You need to replace fedora by default in the last two commands.

No problem, I probably should have mentioned it.

$ ls /sysroot/ostree/boot.0/default/
10c83f4b746fb3e785540edbfd4c85c350246f3b40f2d00424b0125c5168c1ac
87261a5fd4130582614ab002893ffba7ca3bfadba54b784ba65ec3c0dfbebfd6

$ ls /sysroot/ostree/deploy/default/deploy/
10768cb15aea0ec37de7ab09fce7fb48fce7d13d94b43a7ed50f34a0b1511ba8.0
10768cb15aea0ec37de7ab09fce7fb48fce7d13d94b43a7ed50f34a0b1511ba8.0.origin
55626fb992f8da1a1072e3e2705a2ceea79fc0bda44bef2a3dc8b1cbceb18cb4.0
55626fb992f8da1a1072e3e2705a2ceea79fc0bda44bef2a3dc8b1cbceb18cb4.0.origin
869d176689a447c659e24b538876f528e495d9354acaeb96188b3e97311cfb7b.0
869d176689a447c659e24b538876f528e495d9354acaeb96188b3e97311cfb7b.0.origin

And I missed:

$ ls -l /sysroot/ostree/boot.0/*/*
/sysroot/ostree/boot.0/default/10c83f4b746fb3e785540edbfd4c85c350246f3b40f2d00424b0125c5168c1ac:
total 4
lrwxrwxrwx. 1 root root 97  2 juin  19:08 0 -> ../../../deploy/default/deploy/10768cb15aea0ec37de7ab09fce7fb48fce7d13d94b43a7ed50f34a0b1511ba8.0

/sysroot/ostree/boot.0/default/87261a5fd4130582614ab002893ffba7ca3bfadba54b784ba65ec3c0dfbebfd6:
total 4
lrwxrwxrwx. 1 root root 97  2 juin  19:08 0 -> ../../../deploy/default/deploy/869d176689a447c659e24b538876f528e495d9354acaeb96188b3e97311cfb7b.0

Sorry for the delay. I don’t have the time to reproduce that right now so you will have to try things out, moving the outlier (the folder that is not referenced anywhere) to another place and see if that fixes things.

Hello,

No problem, I’m already glad you gave some of your free time to help.

By the outlier, do you mean /sysroot/ostree/boot.0? To recall, it was missing and it is actually a symlink to /sysroot/ostree/boot.1, and without it the second deployment would fail to boot.

I tried removing the symlink and

  1. Creating an empty folder at /sysroot/ostree/boot.0/default/dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/0 where rpm-ostreed expects to find a sysroot.
  2. Symlinking /sysroot/ostree/boot.1/default/10c83f4b746fb3e785540edbfd4c85c350246f3b40f2d00424b0125c5168c1ac/0 to the aforementioned folder

But both result in the following error when attempting to start rpm-ostreed.service

juin 16 18:08:24 axel-EliteBook-845 systemd[1]: Starting rpm-ostreed.service - rpm-ostree System Management Daemon...
juin 16 18:08:25 axel-EliteBook-845 rpm-ostree[331376]: Reading config file '/etc/rpm-ostreed.conf'
juin 16 18:08:25 axel-EliteBook-845 rpm-ostree[331376]: error: Couldn't start daemon: Error setting up sysroot: loading sysroot: Reading current subbootversion: Reading ostree/boot.0: readlinkat: Argument invalide
juin 16 18:08:25 axel-EliteBook-845 systemd[1]: rpm-ostreed.service: Main process exited, code=exited, status=1/FAILURE
juin 16 18:08:25 axel-EliteBook-845 systemd[1]: rpm-ostreed.service: Failed with result 'exit-code'.
juin 16 18:08:25 axel-EliteBook-845 systemd[1]: Failed to start rpm-ostreed.service - rpm-ostree System Management Daemon.

I have yet to reboot my system in this state, I’ll edit this comment to let you know if either deployment boot.

Edit: ostree:0 won’t boot (not surprising TBH) and neither will ostree:1. I had to revert back to /sysroot/ostree/boot.0 being a symlink to /sysroot/ostree/boot.1 for ostree:1 to boot again.

The outlier is likely the one in /sysroot/ostree/deploy/default/deploy/, on which rpm-ostreed is failling when starting up:

[    9.954783] bluefin ostree-prepare-root[884]: ostree-prepare-root: Couldn't find specified OSTree root '/sysroot//ostree/boot.0/default/dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/0': No such file or directory

You should start from the ostree/boot/default folder, and make sure they point to something valid. If they don’t, remove them from there.

I’m not sure to follow, should I symlink /sysroot/ostree/boot.0/default/dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/0 to /sysroot/ostree/deploy/default/deploy/55626fb992f8da1a1072e3e2705a2ceea79fc0bda44bef2a3dc8b1cbceb18cb4.0 (as it seem to be the only folder not mentioned when running ls -l /sysroot/ostree/boot.0/*/*) or should I delete /sysroot/ostree/deploy/default/deploy/55626fb992f8da1a1072e3e2705a2ceea79fc0bda44bef2a3dc8b1cbceb18cb4.0 and /sysroot/ostree/deploy/default/deploy/55626fb992f8da1a1072e3e2705a2ceea79fc0bda44bef2a3dc8b1cbceb18cb4.0.origin instead ?

Well, I tried the symlink option, and now boot.0 boots! Unfortunately rpm-ostreed.service still exits upon startup, so I can’t update my system

juin 18 17:31:25 axel-EliteBook-845 systemd[1]: Starting rpm-ostreed.service - rpm-ostree System Management Daemon...
juin 18 17:31:25 axel-EliteBook-845 rpm-ostree[5749]: Reading config file '/etc/rpm-ostreed.conf'
juin 18 17:31:25 axel-EliteBook-845 rpm-ostree[5749]: error: Couldn't start daemon: Error setting up sysroot: loading sysroot: Reading current subbootversion: Reading ostree/boot.0: readlinkat: Argument invalide
juin 18 17:31:25 axel-EliteBook-845 systemd[1]: rpm-ostreed.service: Main process exited, code=exited, status=1/FAILURE
juin 18 17:31:25 axel-EliteBook-845 systemd[1]: rpm-ostreed.service: Failed with result 'exit-code'.
juin 18 17:31:25 axel-EliteBook-845 systemd[1]: Failed to start rpm-ostreed.service - rpm-ostree System Management Daemon.

Can you try cleaning things up with ostree commands? Maybe removing the other deployments (other than the one you have booted right now).

Can you try rebasing to Fedora Silverblue or Kinoite? And check if it works?

I’m not familiar with ostree and looking at its man page, it is not obvious how I can do that.

Unfortunately rpm-ostree is busted and won’t work (see #4) and I don’t know if it can be done using ostree.