Hard drive became read-only, now Silverblue fails to boot

Project Discussion
1

Hi all. Might any of you be able to help me with this problem?

I booted into an updated version of Silverblue 29. Suddenly I couldn’t run, update, install or uninstall any flatpak apps, where previously they’d been working. When I tried to run any flatpak app in Terminal, it said e.g.:

$ flatpak run org.gnome.Calculator
error: open(O_TMPFILE): Read-only file system

When I tried to uninstall an app, it said e.g.:

$ flatpak uninstall org.gnome.Geary
Uninstalling from system:
org.gnome.Geary/x86_64/stable
org.gnome.Geary.Locale/x86_64/stable
Is this ok [y/n]: y
Uninstalling: org.gnome.Geary/x86_64/stable
error: Failed to uninstall org.gnome.Geary/x86_64/stable: Read-only file system

Flatpak was working properly before I rebooted. rpm-ostree upgrade said:

$ rpm-ostree upgrade
note: automatic updates (stage) are enabled
error: Read-only file system

rpm-ostree status said:

$ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.service: last run failed
Deployments:
● ostree://fedora-workstation:fedora/29/x86_64/silverblue
Version: 29.20181102.0 (2018-11-02T22:24:07Z)
BaseCommit: c6b548c9aac739084fac49193dcf7267df89deb525d15dd71da727ce4d762893
GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
LayeredPackages: cronie gnome-tweaks nano offlineimap syncthing trash-cli unclutter youtube-dl

ostree://fedora-workstation:fedora/29/x86_64/silverblue
Version: 29.1.2 (2018-10-24T23:20:30Z)
BaseCommit: f17b670fa8cf69144be5ae0c968dc2ee7eb6999a5f7a54f1ee71eec7783e434a
GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
LayeredPackages: cronie gnome-tweaks nano offlineimap syncthing trash-cli unclutter youtube-dl

After copying those examples I rebooted, and now I can’t get past the emergency shell.

I installed Silverblue in the first place because a few months ago, Fedora Workstation suddenly started saying I had a read-only filesystem; I rebooted to see if that would fix it, and then couldn’t get past the emergency shell. It looks like the same has happened again.

Does anyone have any ideas what’s happened? Is there any way I can fix this, other than reinstalling and hoping it doesn’t happen again?

(The emergency shell’s help text isn’t helpful either: it tells me I may want to save the logs to a USB drive after mounting it, but doesn’t actually say how to mount it.)

2

Is it possible your hard drive is slowly failing? If it’s been happening on silverblue and fedora workstation and keeps happening it sounds like you’re filesystems are getting corrupted and remounted read/only. When you are at the emergency shell you can try running fsck on your disk(s) to see if any errors are reported.

3

@dustymabe is entirely right, though there’s one major thing I’d like to add: backup all important data IMMEDIATELY before attempted anything. You don’t want to risk losing anything if there’s just one push for your drive to over the edge!

badblocks is a very thorough disk checker that you might want to try. You might also want to try a RAM test; although rare, bad RAM can sometimes indirectly cause filesystem corruption.

4

I think I’ve fixed it!

In the emergency shell, I read through the output of journalctl (as the emergency shell prompt suggested). A few lines were helpfully highlighted in red.

I had already tried fsck -a because I’d seen near the bottom of the log “run fsck manually”; but fsck exited suspiciously quickly. Further up the log I noticed a message saying “run fsck manually, i.e. without -a”. Thereabouts it also said that the filesystem /dev/mapper/fedora_sedna-root was invalid; (sedna is my computer’s name).

So I ran fsck /dev/mapper/fedora_sedna-root and approved the suggested fixes. Then I ran reboot, and now I’m back in GNOME.

I also noticed a message in the log saying “failed to resume from hibernate”. I suspect what happened is this: when I shut the computer down before this began (which I rarely do, but I was going on holiday), while it was shutting down I think I closed the laptop lid. (This is a ThinkPad X230.)

I suspect the system tried to hibernate while shutting down, so it didn’t complete hibernating, and therefore couldn’t resume successfully. Last time this happened was also after the computer had been off for a while.

Does this seem plausible? While the system is shutting down (or while the shutdown dialog is displayed), it should never attempt to sleep or hibernate; instead it should just shut down — is this a known issue somewhere?

5

I think that’s managed by some mix of UPower, udev, and systemd. That being said, I’d still be on the lookout, since filesystem issues due to stuff like this is still usually pretty rare.

6

Had the same issue. After fsck of my ssd(sda5) and reboot.
Silverblue works perfect for upgrades, but grub problems or hibernate (all entries try to resume from hibernate, having resume=“” set) may break it. It may be related to gnome suspend-then-hibernate