Fedora-IoT Read Only Filesystem after upgrade

So I have Fedora IoT deployed to a Raspberry Pi 3. I run rpm-ostree upgrade and then reboot and when booting the new image, a bunch of services such as dbus-broker fail to start. I cannot ssh to the device either. When I reboot to the original image everything works fine. I looked at the journal and saw errors saying the services could not start do to a read only file system. I checked the mount info in both the “good” image and “bad” image and found that /sysroot is coming up as RO when I boot to the new image:

Good

$ mount | grep mmcblk0p3
/dev/mmcblk0p3 on /sysroot type ext4 (rw,relatime,seclabel)
/dev/mmcblk0p3 on / type ext4 (rw,relatime,seclabel)
/dev/mmcblk0p3 on /usr type ext4 (ro,relatime,seclabel)
/dev/mmcblk0p3 on /var type ext4 (rw,relatime,seclabel)
/dev/mmcblk0p3 on /var/lib/containers/storage/overlay type ext4 (rw,relatime,seclabel)

Bad

$ mount | grep mmcblk0p3
/dev/mmcblk0p3 on /sysroot type ext4 (ro,relatime,seclabel)
/dev/mmcblk0p3 on / type ext4 (rw,relatime,seclabel)
/dev/mmcblk0p3 on /usr type ext4 (ro,relatime,seclabel)
/dev/mmcblk0p3 on /var type ext4 (rw,relatime,seclabel)

Has anyone seen this before?

See https://bugzilla.redhat.com/show_bug.cgi?id=1862568

If you’re affected by this, probably the simplest is to reboot into the previous bootloader entry, then do an upgrade. Alternatively directly doing

mount -o remount,rw /sysroot
systemctl start <failed services>
rpm-ostree upgrade

might work.

1 Like

Cool. I’m glad I’m not the only one. I’ll just use the previous image for now and upgrade to the next released image. I don’t want to rely on remounting since I want to have this run headless and I can’t ssh to it when it’s broken.