Hello,
Today, after my Silverblue install was suddenly unable to launch any flatpak, I tried rebooting it, but it hang. Being in a hurry because I had to join a meeting, I forced the shutdown by keeping the power button down.
That was an extremely bad idea, as after typing my passphrase in order to unlock my LUKS-encrypted drive, I was met with an emergency shell.
/run/initramfs/rdsosreport.txt indicates that the LUKS partition was successfully decrypted, but then the OSTree could not be found (file edited to only keep the relevant parts)
...
[ 10.874461] bluefin systemd-cryptsetup[712]: Set cipher aes, mode xts-plain64, key size 512 bits for device /dev/disk/by-uuid/6d7d0a53-3c12-46e3-a03d-e877a29d586d.
[ 12.825407] bluefin systemd[1]: Found device dev-disk-by\x2duuid-5eb69e8f\x2d333f\x2d4119\x2d877e\x2dbb86bba3a213.device - /dev/disk/by-uuid/5eb69e8f-333f-4119-877e-bb86bba3a213.
[ 12.825811] bluefin systemd[1]: Reached target initrd-root-device.target - Initrd Root Device.
[ 12.826958] bluefin systemd[1]: Finished systemd-cryptsetup@luks\x2d6d7d0a53\x2d3c12\x2d46e3\x2da03d\x2de877a29d586d.service - Cryptography Setup for luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d.
[ 12.828143] bluefin systemd[1]: Reached target cryptsetup.target - Local Encrypted Volumes.
[ 12.828177] bluefin systemd[1]: Reached target sysinit.target - System Initialization.
[ 12.828200] bluefin systemd[1]: Reached target basic.target - Basic System.
[ 12.832444] bluefin systemd[1]: Finished dracut-initqueue.service - dracut initqueue hook.
[ 12.833296] bluefin systemd[1]: Reached target remote-fs-pre.target - Preparation for Remote File Systems.
[ 12.833336] bluefin systemd[1]: Reached target remote-cryptsetup.target - Remote Encrypted Volumes.
[ 12.833360] bluefin systemd[1]: Reached target remote-fs.target - Remote File Systems.
[ 12.834085] bluefin systemd[1]: Starting dracut-pre-mount.service - dracut pre-mount hook...
[ 12.858135] bluefin systemd[1]: Finished dracut-pre-mount.service - dracut pre-mount hook.
[ 12.859250] bluefin systemd[1]: Starting systemd-fsck-root.service - File System Check on /dev/disk/by-uuid/5eb69e8f-333f-4119-877e-bb86bba3a213...
[ 12.877072] bluefin systemd[1]: Finished systemd-fsck-root.service - File System Check on /dev/disk/by-uuid/5eb69e8f-333f-4119-877e-bb86bba3a213.
[ 12.878397] bluefin systemd[1]: Mounting sysroot.mount - /sysroot...
[ 12.890036] bluefin kernel: BTRFS: device label bluefin_fedora devid 1 transid 91961 /dev/mapper/luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d (252:0) scanned by mount (874)
[ 12.890082] bluefin kernel: BTRFS info (device dm-0): first mount of filesystem 5eb69e8f-333f-4119-877e-bb86bba3a213
[ 12.890102] bluefin kernel: BTRFS info (device dm-0): using crc32c (crc32c-x86_64) checksum algorithm
[ 12.890117] bluefin kernel: BTRFS info (device dm-0): using free-space-tree
[ 12.910877] bluefin systemd[1]: Mounted sysroot.mount - /sysroot.
[ 12.911831] bluefin systemd[1]: Starting ostree-prepare-root.service - OSTree Prepare OS/...
[ 12.933121] bluefin ostree-prepare-root[891]: ostree-prepare-root: Couldn't find specified OSTree root '/sysroot//ostree/boot.0/default/dd1e236e145f30318cc828a12f86eda9212bbf65f2bc98938451c97c05fdc02b/0': No such file or directory
[ 12.933952] bluefin systemd[1]: ostree-prepare-root.service: Main process exited, code=exited, status=1/FAILURE
[ 12.934071] bluefin systemd[1]: ostree-prepare-root.service: Failed with result 'exit-code'.
[ 12.934251] bluefin systemd[1]: Failed to start ostree-prepare-root.service - OSTree Prepare OS/.
[ 12.934333] bluefin systemd[1]: ostree-prepare-root.service: Triggering OnFailure= dependencies.
[ 12.935086] bluefin systemd[1]: Stopped target remote-cryptsetup.target - Remote Encrypted Volumes.
...
I first tried to decrypt the partition again from the both emergency and recovery shells but failed, so I booted a Silverblue LiveUSB in Troubleshoot mode, where it automatically decrypted the partition but could not find a Linux system there (No Linux systems found).
With some fiddling I was able to manually decrypt and mount the partition
$ cryptsetup open --type luks /dev/nvm0n1p3 luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d
...
$ cryptsetup status luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d
/dev/mapper/luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d is active.
type: LUKS2
cipher: aes-xts-plain64
keysize: 512 bits
key location: keyring
device: /dev/nvme0n1p3
sector size: 512
offset: 32768 sectors
size: 496756736 sectors
mode: read/write
$ btrfs check /dev/mapper/luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d
UUID: 5eb69e8f-333f-4119-877e-bb86bba3a213
found 103380017152 bytes used, no error found
total csum bytes: 82616628
total tree bytes: 3147022336
total fs tree bytes: 2889121792
total extent tree bytes: 146046976
btree space waste bytes: 632915339
file data blocks allocated: 177383219200
referenced 135478329344
$ mkdir -p /tmp/mnt
$ mount /dev/mapper/luks-6d7d0a53-3c12-46e3-a03d-e877a29d586d /tmp/mnt
$ ls /tmp/mnt
home root var
From there, I don’t know how to proceed to recover my system. Listing /root/ostree (mounted to /tmp/mnt/root/ostree) lists the following folders boot.1 boot.1.1 deploy lock repo while btrfs subvolume list -o /tmp/mnt does not list any snapshot.
$ btrfs subvolume list -o /tmp/mnt/
ID 256 gen 91963 top level 5 path var
ID 257 gen 91958 top level 5 path home
ID 258 gen 91964 top level 5 path root
$ btrfs subvolume list -os /tmp/mnt/
[empty output]
What steps should I take now to keep diagnosing the issue and possibly fix it?
Thanks in advance,
Axel
(Note: I already tried booting the ostree entry 1 in GRUB with the same result)