Updated Silverblue always boots to "You are in emergency mode" but no root user

Ask Fedora
1

Since shortly after 2021-09-06, every time I run “rpm-ostree update” on my Silverblue F33 system the resulting image won’t boot (therefore I’m still booting from my 2021-09-06 image).

The message on the screen says: “You are in emergency mode” and suggests some logging commands I might try, but then tells me it “Cannot open access to console. The root account is locked” so I can’t actually run those commands.

Google searches suggest file system checks on /home, but I assume if /home was damaged I wouldn’t be able to boot the 2021-09-06 image either. Other suggestions about using a live image and adding a root user are not Silverblue specific, so I’m not sure they even apply with the immutable boot images.

Any suggestions for a quick fix?

2

As a followup I found troubleshooting tips that suggest ostree fsck to look for corrupted commits.

However, running this command as root indicated no corrupt commits.

About to try rpm-ostree cleanup -p. Will report on results (as long as my system remains bootable).

3

You can try to get back to a more “pristine” state by doing rpm-ostree reset. First take note of your overlaid packages with rpm-ostree status first, because the reset will remove them.

Then reboot, do rpm-ostree cleanup -m to clean out the local metadata repos, then try to upgrade with rpm-ostree upgrade.

1 Like
4

Thanks for the suggestion - I tried those steps but still encounter the same error.

$ rpm-ostree status
State: busy
Transaction: refresh-md
  Initiator: client(id:gnome-software dbus:1.119 unit:app-gnome-gnome\x2dsoftware\x2dservice-2369.scope uid:1000)
Deployments:
  ostree://fedora:fedora/33/x86_64/silverblue
                   Version: 33.20211020.0 (2021-10-20T19:07:57Z)
                    Commit: b37a72dc497a2d4cc31525fd5a0e86e8214b6119998de47d89699bbcd2c5f1ca
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
                      Diff: 116 upgraded, 119 removed, 1 added

  ostree://fedora:fedora/33/x86_64/silverblue
                   Version: 33.20210906.0 (2021-09-06T22:48:35Z)
                    Commit: 2422b465e2e0cd01de01beb6eeb1d67d949a3f340c86c8fa515c9d7d635e6ba9
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31

● ostree://fedora:fedora/33/x86_64/silverblue
                   Version: 33.20210906.0 (2021-09-06T22:48:35Z)
                BaseCommit: 2422b465e2e0cd01de01beb6eeb1d67d949a3f340c86c8fa515c9d7d635e6ba9
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: akmod-nvidia conntrack encfs firewall-config genisoimage gnome-tweak-tool guvcview hdparm net-tools pavucontrol perl-Expect-Simple perl-Term-Prompt samba
                            screen ufw wodim xorg-x11-drv-nvidia xorg-x11-drv-nvidia-cuda
             LocalPackages: rpmfusion-nonfree-release-33-1.noarch rpmfusion-free-release-33-1.noarch
                    Pinned: yes
  • The pinned image is the “good” boot image I’ve been using
  • The other 33.20210906.0 image is the result of rpm-ostree reset and also boots successfully
  • The 33.20211020.0 image was created by running rpm-ostree cleanup -m / rpm-ostree upgrade and always boots into emergency mode
5

Can you post your output for rpm-ostree db diff 2422b465e2e0cd01de01beb6eeb1d67d949a3f340c86c8fa515c9d7d635e6ba9 b37a72dc497a2d4cc31525fd5a0e86e8214b6119998de47d89699bbcd2c5f1ca

6

Full disclosure - your steps gave me confidence to attempt an upgrade (rebase) to Fedora 35. If that works I will close this issue. Meanwhile, here’s the information you asked for.

$ rpm-ostree db diff 2422b465e2e0cd01de01beb6eeb1d67d949a3f340c86c8fa515c9d7d635e6ba9 b37a72dc497a2d4cc31525fd5a0e86e8214b6119998de47d89699bbcd2c5f1ca
ostree diff commit from: 2422b465e2e0cd01de01beb6eeb1d67d949a3f340c86c8fa515c9d7d635e6ba9
ostree diff commit to:   b37a72dc497a2d4cc31525fd5a0e86e8214b6119998de47d89699bbcd2c5f1ca
Upgraded:
  appstream-data 33-3.fc33 -> 33-4.fc33
  audit 3.0.5-1.fc33 -> 3.0.6-1.fc33
  audit-libs 3.0.5-1.fc33 -> 3.0.6-1.fc33
  btrfs-progs 5.13.1-1.fc33 -> 5.14.1-1.fc33
  buildah 1.22.3-1.fc33 -> 1.23.0-1.fc33
  cairomm 1.12.2-4.fc33 -> 1.12.2-11.fc33
  catatonit 0.1.5-3.fc33 -> 0.1.6-1.fc33
  cifs-utils 6.11-2.fc33 -> 6.13-3.fc33
  cifs-utils-info 6.11-2.fc33 -> 6.13-3.fc33
  containernetworking-plugins 1.0.0-1.fc33 -> 1.0.1-1.fc33
  cracklib 2.9.6-24.fc33 -> 2.9.6-27.fc33
  cracklib-dicts 2.9.6-24.fc33 -> 2.9.6-27.fc33
  cups-filters 1.28.9-2.fc33 -> 1.28.10-1.fc33
  cups-filters-libs 1.28.9-2.fc33 -> 1.28.10-1.fc33
  curl 7.71.1-10.fc33 -> 7.71.1-11.fc33
  ethtool 2:5.13-1.fc33 -> 2:5.14-1.fc33
  firefox 91.0.2-1.fc33 -> 92.0-2.fc33
  flatpak 1.10.2-3.fc33 -> 1.10.3-1.fc33
  flatpak-builder 1.0.12-1.fc33 -> 1.0.14-1.fc33
  flatpak-libs 1.10.2-3.fc33 -> 1.10.3-1.fc33
  flatpak-selinux 1.10.2-3.fc33 -> 1.10.3-1.fc33
  flatpak-session-helper 1.10.2-3.fc33 -> 1.10.3-1.fc33
  fuse-overlayfs 1.5.0-1.fc33 -> 1.7.1-2.fc33
  ghostscript 9.53.3-6.fc33 -> 9.55.0-1.fc33
  ghostscript-tools-fonts 9.53.3-6.fc33 -> 9.55.0-1.fc33
  ghostscript-tools-printing 9.53.3-6.fc33 -> 9.55.0-1.fc33
  gjs 1.66.2-8.fc33 -> 1.66.2-10.fc33
  gnome-software 3.38.2-2.fc33 -> 3.38.2-4.fc33
  gnome-software-rpm-ostree 3.38.2-2.fc33 -> 3.38.2-4.fc33
  hplip 3.21.2-10.fc33 -> 3.21.2-11.fc33
  hplip-common 3.21.2-10.fc33 -> 3.21.2-11.fc33
  hplip-libs 3.21.2-10.fc33 -> 3.21.2-11.fc33
  httpd 2.4.48-1.fc33 -> 2.4.51-1.fc33
  httpd-filesystem 2.4.48-1.fc33 -> 2.4.51-1.fc33
  httpd-tools 2.4.48-1.fc33 -> 2.4.51-1.fc33
  hwdata 0.350-1.fc33 -> 0.352-1.fc33
  ibus-typing-booster 2.14.4-1.fc33 -> 2.14.13-1.fc33
  jitterentropy 3.0.2-2.git.409828cf.fc33 -> 3.3.0-1.fc33
  kernel 5.13.12-100.fc33 -> 5.14.12-100.fc33
  kernel-core 5.13.12-100.fc33 -> 5.14.12-100.fc33
  kernel-devel 5.13.12-100.fc33 -> 5.14.12-100.fc33
  kernel-headers 5.13.3-100.fc33 -> 5.14.9-100.fc33
  kernel-modules 5.13.12-100.fc33 -> 5.14.12-100.fc33
  kernel-modules-extra 5.13.12-100.fc33 -> 5.14.12-100.fc33
  libcurl 7.71.1-10.fc33 -> 7.71.1-11.fc33
  libdnf 0.63.1-1.fc33 -> 0.64.0-1.fc33
  liberation-fonts-common 1:2.1.4-1.fc33 -> 1:2.1.5-1.fc33
  liberation-mono-fonts 1:2.1.4-1.fc33 -> 1:2.1.5-1.fc33
  liberation-sans-fonts 1:2.1.4-1.fc33 -> 1:2.1.5-1.fc33
  liberation-serif-fonts 1:2.1.4-1.fc33 -> 1:2.1.5-1.fc33
  libgs 9.53.3-6.fc33 -> 9.55.0-1.fc33
  libmodulemd 2.13.0-1.fc33 -> 2.13.0-2.fc33
  libnfsidmap 1:2.5.4-0.fc33 -> 1:2.5.4-2.rc3.fc33
  libopenmpt 0.4.23-1.fc33 -> 0.4.24-1.fc33
  libraqm 0.7.0-6.fc33 -> 0.7.2-1.fc33
  librepo 1.14.1-1.fc33 -> 1.14.2-1.fc33
  libsane-airscan 0.99.26-1.fc33 -> 0.99.27-1.fc33
  libsane-hpaio 3.21.2-10.fc33 -> 3.21.2-11.fc33
  libsmbclient 2:4.13.10-0.fc33 -> 2:4.13.12-0.fc33
  libssh 0.9.5-1.fc33 -> 0.9.6-1.fc33
  libssh-config 0.9.5-1.fc33 -> 0.9.6-1.fc33
  libwbclient 2:4.13.10-0.fc33 -> 2:4.13.12-0.fc33
  libxcrypt 4.4.25-1.fc33 -> 4.4.26-2.fc33
  libxcrypt-compat 4.4.25-1.fc33 -> 4.4.26-2.fc33
  libxcrypt-devel 4.4.25-1.fc33 -> 4.4.26-2.fc33
  mod_lua 2.4.48-1.fc33 -> 2.4.51-1.fc33
  mozjs78 78.13.0-1.fc33 -> 78.15.0-1.fc33
  nfs-utils 1:2.5.4-0.fc33 -> 1:2.5.4-2.rc3.fc33
  nss 3.69.0-1.fc33 -> 3.71.0-1.fc33
  nss-softokn 3.69.0-1.fc33 -> 3.71.0-1.fc33
  nss-softokn-freebl 3.69.0-1.fc33 -> 3.71.0-1.fc33
  nss-sysinit 3.69.0-1.fc33 -> 3.71.0-1.fc33
  nss-util 3.69.0-1.fc33 -> 3.71.0-1.fc33
  ntfsprogs 2:2017.3.23-14.fc33 -> 2:2021.8.22-2.fc33
  openssh 8.4p1-7.fc33 -> 8.4p1-8.fc33
  openssh-clients 8.4p1-7.fc33 -> 8.4p1-8.fc33
  openssh-server 8.4p1-7.fc33 -> 8.4p1-8.fc33
  openssl-devel 1:1.1.1k-1.fc33 -> 1:1.1.1l-2.fc33
  openssl-libs 1:1.1.1k-1.fc33 -> 1:1.1.1l-2.fc33
  pango 1.48.4-1.fc33 -> 1.48.4-2.fc33
  perl-Digest 1.19-1.fc33 -> 1.20-1.fc33
  podman 3:3.3.0-1.fc33 -> 3:3.3.1-1.fc33
  podman-gvproxy 3:3.3.0-1.fc33 -> 3:3.3.1-1.fc33
  podman-plugins 3:3.3.0-1.fc33 -> 3:3.3.1-1.fc33
  python-pip-wheel 20.2.2-2.fc33 -> 20.2.2-4.fc33
  python-unversioned-command 3.9.6-2.fc33 -> 3.9.7-1.fc33
  python3 3.9.6-2.fc33 -> 3.9.7-1.fc33
  python3-audit 3.0.5-1.fc33 -> 3.0.6-1.fc33
  python3-libs 3.9.6-2.fc33 -> 3.9.7-1.fc33
  python3-pillow 7.2.0-7.fc33 -> 7.2.0-8.fc33
  python3-pip 20.2.2-2.fc33 -> 20.2.2-4.fc33
  rng-tools 6.13-2.git.d207e0b6.fc33 -> 6.14-1.git.56626083.fc33
  samba-client 2:4.13.10-0.fc33 -> 2:4.13.12-0.fc33
  samba-client-libs 2:4.13.10-0.fc33 -> 2:4.13.12-0.fc33
  samba-common 2:4.13.10-0.fc33 -> 2:4.13.12-0.fc33
  samba-common-libs 2:4.13.10-0.fc33 -> 2:4.13.12-0.fc33
  samba-libs 2:4.13.10-0.fc33 -> 2:4.13.12-0.fc33
  sane-airscan 0.99.26-1.fc33 -> 0.99.27-1.fc33
  system-config-printer-libs 1.5.15-1.fc33 -> 1.5.15-5.fc33
  system-config-printer-udev 1.5.15-1.fc33 -> 1.5.15-5.fc33
  tzdata 2021a-1.fc33 -> 2021b-1.fc33
  urw-base35-bookman-fonts 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-c059-fonts 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-d050000l-fonts 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-fonts 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-fonts-common 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-gothic-fonts 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-nimbus-mono-ps-fonts 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-nimbus-roman-fonts 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-nimbus-sans-fonts 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-p052-fonts 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-standard-symbols-ps-fonts 20200910-5.fc33 -> 20200910-6.fc33
  urw-base35-z003-fonts 20200910-5.fc33 -> 20200910-6.fc33
  vim-minimal 2:8.2.3367-1.fc33 -> 2:8.2.3404-1.fc33
  webkit2gtk3 2.32.3-1.fc33 -> 2.32.4-1.fc33
  webkit2gtk3-jsc 2.32.3-1.fc33 -> 2.32.4-1.fc33
Removed:
  ntfs-3g-2:2017.3.23-14.fc33.x86_64
  ntfs-3g-system-compression-1.0-4.fc33.x86_64
Added:
  ntfs-3g-libs-2:2021.8.22-2.fc33.x86_64
7

And after rebasing on Fedora 35 my system boots successfully. I re-applied my layered packages and everything seems to be working (though now I have to get used to Gnome 41!).

It probably isn’t worth time trying to track down what went wrong on a discontinued Fedora release.

Thanks again for you help!

8

Unfortunately, I’m out of ideas of what could be the issue. Hopefully the rebase works for you.