Silverblue won't boot after forced shutdown, searching for ways to recover it

Ask Fedora
, ,
21

can you use bootc to switch?

sudo bootc switch quay.io/fedora-ostree-desktops/silverblue:42
22

bootc is busted as well:

$ sudo bootc switch ghcr.io/ublue-os/bluefin-dx:stable
ERROR Switching: Initializing storage: Acquiring sysroot: loading sysroot: Reading current subbootversion: Reading ostree/boot.0: readlinkat: Invalid argument
23

Iā€™m facing a very similar issue. Fixed the initial booting issue the same way with a symlink but rpm-ostree is broken as well.

The root cause is also a missing version for me, which also shows up with:

# ostree admin status
error: loading sysroot: Parsing deployment /ostree/boot.1/fedora/edaba760dfa1563e751ade303a3caf7c041e19ce8d6a0db7f8c66909d2c03d08/0 in stateroot 'fedora': readlinkat: No such file or directory

Does anyone know where ostree is looking up the references?

1 Like
24

Found the solution!

You need to grep for the bad/missing commit in /boot/loader/entries/ and remove the offending ostree-?.conf file. It allows ostree and rpm-ostree to recover.

In my case:

$ rm /boot/loader/entries/ostree-3.conf
$ ostree admin status
* fedora e9a1a0dcc743abaf6294cbfc274f419748b7aab215773537ac446da656776deb.0
    Version: 42.20250619.0
    origin: <unknown origin type>
  fedora f52af93ac1c37adbba6de9fcc606e70d009126eaec3f172635fa72d9ea3ea928.0 (rollback)
    Version: 41.20250322.0
    Pinned: yes
    origin: <unknown origin type>
25

Update: no it doesnā€™t fix it. It did boot, but once I tried to rpm-ostree upgrade I ended up in a very bad state where all of /ostree/boot.* checkouts are wiped out. Similar to what happened to this person: OSTree boot links can get out of sync Ā· Issue #2283 Ā· ostreedev/ostree Ā· GitHub

By the way, this github issue is the upstream tracking bug. And it looks like itā€™s very hard to fix/recover from this error. I think I will move away from ostree as a result, since it breaks the confidence I had about safe atomic updates (the issue has been unresolved for 4 years now).

26

Thanks for your feedback, it seems to be near impossible to fix without a full reinstall.

For my part I will stick with ostree but I will configure automated BTRFS snapshots in the hope that I can recover my system the next time it happens (as for backups Iā€™m already covered by automated restic snapshots to my NAS)

27

Arg, this is really bad. Itā€™s weird that we havenā€™t been able to reproduce it yet.

We are working on moving off ostree into pure composefs native storage: composefs-rs integration in bootc Ā· Issue #1190 Ā· bootc-dev/bootc Ā· GitHub. Itā€™s not ready yet but should hopefully help in the future.

28

Iā€™m not an expert by any means and thatā€™s just a wild guess, but is it possible that these issues are somehow related to disk encryption?

See Fedora 42 Kinoite doesn't boot anymore (yesterday evening it did) - #20 by hricky and the following Fedora 42 Kinoite doesn't boot anymore (yesterday evening it did) - #21 by hricky.

29

I finally managed to fix things up completely without a reinstall. Write-up on what I did is here: OSTree boot links can get out of sync Ā· Issue #2283 Ā· ostreedev/ostree Ā· GitHub

4 Likes
30

I have similar issue from today.
The symlinking boot.0 to boot.1 allowed me to boot with GRUB option 1 too. I didnā€™t have time yet to try to fix broken thingsā€¦
I can say last time before the problem, my machine was running out of battery and shutdown. I got this issue on next bootā€¦

31

Out of curiosity, are you using disk encryption on this machine?

32

Indeed

1 Like
33

@hricky , did you notice by any chance that only users having encrypted root partitions are experiencing this issue?

34

According to journald, I have shutdown logs (around 6 seconds) so itā€™s not a ā€œforcedā€ shutdown like OP.

35

It seems to be the case, so Iā€™ll enable TPM, Secure Boot, or whatever in the BIOS on a bare metal machine and install Fedora Silverblue with disk encryption to try to reproduce these issues.

2 Likes
36

I donā€™t know if it can be related, but to give as much as information to help to understand : I have 42.20250713 entry in GRUB (entry 0, which does not boot). So I think it means I did upgrade the 13 July.
But journalctl -b -1 returns logs from 07-11 to 07-14 (when out-of-battery happened) so I think I never booted on the new entry 20250713

37

Tried tips in the github issue but rpm-ostree upgrade failed (didnā€™t copy the error) and back to emergency mode after reboot. I had to recreate boot.x folders and links.
I even made a mistake by reversing boot.1 and boot.0 and I ended up to boot in initial setup mode :sweat_smile:
Fortunaltely I was able to switch back and be able to get back a working system. Iā€™ll try to fix this weekend

1 Like
38

I installed Fedora Silverblue 42 with the default disk encryption and partitioning. I then pushed the reset button on the front panel during the shutdown and startup processes several times, but I still canā€™t reproduce the issue. Is there something related to disk encryption or another specific setting I can try?

39

My uninformed guess would be to try and force shutdown in the middle of an upgrade, for some definition of ā€œmiddleā€ (maybe after rpm-ostree has returned but before the machine goes through a clean reboot). Iā€™m also wondering what changed recently to make more people experience the issue.

40

I have already tried this using GNOME Software and will continue to try different rpm-ostree commands.