I noticed two Fedora 39 installations, a laptop and desktop, booting with read-only filesystems. I first noticed Nheko complaining about unable to restore keys or something wanting to send me back to login screen, then I opened terminal and encountered errors about unable to write to ~/.zsh_history due to read-only filesystem.
Cause
I don’t know. I have a suspicion of it being ro in kernel boot parameters.
(A friend group tells me that kernel should remount partitions as rw during the boot progress which doesn’t seem to happen there and suggests checking fstab (where I see nothing special) and sudo /usr/lib/systemd/systemd-remount-fs which I didn’t try due to the workarounds I document below.)
Related Issues
I didn’t find anything related on bugzilla and am unsure of whether this is related to recent grub or kernel update as the laptop was fine for some weeks before this issue started happening. I think the desktop was also fine until today morning, I upgraded it to Fedora 39 yesterday.
Workarounds
During boot in grub menu press e to edit and on the kernel boot line change ro to rw.
When / is rw, I appended rw to the GRUB_CMDLINE_LINUX= line in /etc/default/grub and ran sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Hi, thanks for the write-up. In order to put it into Common Issues , it really has to be a widespread problem. We also need a Bugzilla ticket to get some traction in fixing it. I’ll move the to Ask Fedora for the moment. Let’s see if this hits a large number of people, or if we can discover some major issue once it is reported and debugged in Bugzilla.
As an update, I have checked smartctl on both devices and they report:
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
So this cannot be a result of two entirely different hardware failing at once or at least there is nothing to show it was other than ro-mounting possibly being a symptom of it.
A common cause of the root file system to remain mounted in read-only mode is that it needs to be fixed by running fsck. If you don’t run fsck on that file system, and if you force it to be remounted to read-write, then any inconsistencies in the file system will likely spread to such an extent that all your date will be lost. A cause if inconsistencies can be a sudden loss of power where the operation system did not have the time to properly unmount the file system.
I don’t think that is the case, because one of the systems uses btrfs which apparently doesn’t need fscking. I have however ran it on the other just in case, there haven’t been any power issues though.
When I execute the command specified by Aminda: sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
I get two error messages:
/usr/bin/grub2-editenv: error: cannot open `/boot/grub2/grubenv.new': No such file or directory.
/usr/sbin/grub2-mkconfig: line 268: /boot/efi/EFI/fedora/grub.cfg.new: No such file or directory
Everything I said about grub config files is wrong since Fedora 34, while I don’t know if this helps in your case, the wiki has instructions on reverting it.
I haven’t been experiencing this on either Fedora 39 or experimental 40 (which has a different issue of kernel not getting updated in either needrestart or grub, which I am not going to worry about past my bed time or when I have something else to do).
If it executed that is an invalid command since about F34.
The correct command would be sudo grub2-mkconfig -o /boot/grub2/grub.cfg as noted by Aminda just above. The fix is in the link also posted there. I will edit the first post with the incorrect command so others do not have the same error.
We need to know precisely what one of the failing servers is doing/not doing that causes the problem. Having it mount read only seems to indicate that the file system may have a problem and the server is trying to prevent further damage/corruption.
The first step is to find out what you have for a system and what the file
systems are. Please post the output of cat /etc/fedora-release and lsblk -f as well as mount. Please use the preformatted text tags with the </> button on the toolbar with text you copy & paste from your screen when posting the requested info (similar to what you did with the uname command).
I reinstalled Fedora Server from a USB to recover, and all is right now. I think it was an incorrect /etc/fstab entry for an NSF mount. I should have checked if it rebooted correctly on one machine before blindly copying it to all the others. A good lesson. Thank you for the advice. I appreciate the assitance.