Kinoite remounting /var (including home) read-only during a session?

Has anyone else had the following happen? At some point, I start getting pop-up error windows saying that some ~/.config/… file (for instance, .config/konsolerc) cannot be written and to contact the system administrator. Touching ~/test says that the file system is read-only. Mount shows that /var and /var/home are mounted ro. It doesn’t happen immediately after booting, so something later just decides to remount /var ro. This scenario has happened to me twice, while doing unrelated things. Most recently I was creating a vm using gnome boxes flatpak, while running btop in a toolbox to monitor how that was going. So the system was under some load, and then the ro remount happened.

Where would I look to figure out what is remounting /var ro? And WTF would there even be anything that would ever remount /var as ro?

Below is the journalctl output at the scene of the crime. Note the “g_mkstemp … Read-only file system” error, and perhaps the kernel output right before that. Spare me any “that’s nothing to write home about” puns. Anyone have an idea what this journalctl output means, and which group I should send it to?

BTW: this is in fedora kinoite 36.20221019.0

I was not expecting there to be anything directly about the failure in journalctl, because of course if /var is read-only, how would anything get logged? But fortunately looked anyway thinking there may be a clue before the remount happened. Maybe whatever happened first remounted /var/home ro, then /var later? I recall using mount and seeing /var mounted ro as well as /var/home.

Some explanation: the sdb disk is because I have kinoite booted from an external drive. Could that be why this happened? Is there some issue with mixing an ostree distro like kinoite with external drives? I’m using an external drive because what I read about attempts to dual-boot kinoite scared me away from trying that first.

Oct 21 22:28:46 lapdog systemd[1]: pcscd.service: Deactivated successfully.
Oct 21 22:28:46 lapdog audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pcscd comm="systemd" e
xe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 21 22:29:44 lapdog gnome-boxes[2188]: portals.vala:57: Failed to request to run in background: Timeout was reached
Oct 21 22:36:50 lapdog gnome-boxes[2188]: machine.vala:198: display Microsoft Windows 10 disconnected
Oct 21 22:36:50 lapdog virtqemud[2209]: libvirt version: 8.4.0
Oct 21 22:36:50 lapdog virtqemud[2209]: hostname: lapdog
Oct 21 22:36:50 lapdog virtqemud[2209]: cannot parse process status data
Oct 21 22:37:07 lapdog gnome-boxes[2188]: ../gobject/gsignal.c:2722: handler '33355' of instance '0x55768cdb9020' is not blocked
Oct 21 22:37:58 lapdog kernel: sd 6:0:0:0: [sdb] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=28s
Oct 21 22:37:59 lapdog kernel: sd 6:0:0:0: [sdb] tag#17 Sense Key : Illegal Request [current]
Oct 21 22:37:59 lapdog kernel: sd 6:0:0:0: [sdb] tag#17 Add. Sense: Invalid command operation code
Oct 21 22:37:59 lapdog kernel: sd 6:0:0:0: [sdb] tag#17 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
Oct 21 22:38:00 lapdog virtqemud[2209]: g_mkstemp("/home/user1/.var/app/org.gnome.Boxes/config/libvirt/qemu/lib/domain-2-win10/qemu.screendump.BWXAU1") fa
iled: Read-only file system
Oct 21 22:38:00 lapdog gnome-boxes[2188]: cannot finish stream: this function is not supported by the connection driver: virStreamFinish
-- Boot 48a5fd162aec48028bb530fd08f5b00d --
Oct 22 12:47:09 fedora kernel: microcode: microcode updated early to revision 0x2f, date = 2019-02-17
Oct 22 12:47:09 fedora kernel: Linux version 5.19.16-200.fc36.x86_64 (mockbuild@bkernel01.iad2.fedoraproject.org) (gcc (GCC) 12.2.1 20220819 (Red Hat 12.2
.1-2), GNU ld version 2.37-36.fc36) #1 SMP PREEMPT_DYNAMIC Sun Oct 16 22:50:04 UTC 2022

[yes - that was a windows 10 vm. Don’t judge.]

Oct 21 22:37:58 lapdog kernel: sd 6:0:0:0: [sdb] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=28s
Oct 21 22:37:59 lapdog kernel: sd 6:0:0:0: [sdb] tag#17 Sense Key : Illegal Request [current]
Oct 21 22:37:59 lapdog kernel: sd 6:0:0:0: [sdb] tag#17 Add. Sense: Invalid command operation code
Oct 21 22:37:59 lapdog kernel: sd 6:0:0:0: [sdb] tag#17 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00

Looks like you disk is failing, thus returning bad results and then the kernel remounts the partition RO.

smartctl shows 3 UDMA_CRC errors and 3 ATA errors. Disk is brand new: 23 power-on hours. The times logged for the 3 ATA errors are all 0 days + 0 hours - which probably means that smart on this disk doesn’t log times properly. They may have occurred during this incident.
Tried smartctl -t long, and it passes.
But, 3 ATA errors in 23 hours is not good.