I run Fedora 38. I rebooted the system today, it worked fine, then all of a sudden it went read-only.
I don’t want to reboot it in case I cannot bring it up again. I would run fsck but worry it might make things worse. Any ideas of how I might diagnose the problem, and what is best to do? Thank you for your help.
PS: I tried smartctl (as suggested in another thread) but it’s not installed.
It maybe a hardware issue.
Run dmesg and look for messages about you disk.
Thank you for the reply.
In the meantime I had to reboot, as nothing was working any more. Now all works as usual, but not sure for how long.
smartctl --health /dev/sda3
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.5.5-200.fc38.x86_64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
I’ve tried fsck with no options, it says ‘dirty bit’ is set, but no other information.
dmesg don’t seem to show anything in particular about the disk.
Can I check the disk in more detail?
If you want to check the disk, then you should boot from a live system and run the check utilities while the filesystem is not mounted.
We need to know what your file system is, and we need logs. Use journalctl to gather relevant logs
Thank you for the replies.
A.Ismail: I don’t think I know how to do that.
Flo: I’ve looked at the logs between the previous boot, and the moment log messages could no longer be saved because the system was read-only. At first sight, I cannot spot anything explicitly worrying. I think (but am not sure) that the system was installing the latest updates, and at the same time I was trying to save some .pdf files and evince was reaching time-out. Might have been that there was an incompatibility between these processes? Perhaps I should have waited for the updates to install, then reboot, and only then write more files?
I can post the logs, but it’s a long list. Perhaps I have missed something that explains the problem.
And your root file system is btrfs?
Yes, please post the logs as “Preformatted Text”
In the case that the file system is read-only no journal logs will be saved.
You would need to look at the journal logs while the system is read-only as the logs are in memory at that time.
Flo: Thank you. Yes, file system is btrfs.
Sep 30 04:51:06 fedora.fritz.box systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Sep 30 04:51:06 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:51:06 fedora.fritz.box audit: BPF prog-id=133 op=UNLOAD
Sep 30 04:51:06 fedora.fritz.box audit: BPF prog-id=132 op=UNLOAD
Sep 30 04:51:06 fedora.fritz.box audit: BPF prog-id=131 op=UNLOAD
Sep 30 04:51:32 fedora.fritz.box evince[8479]: failed to commit changes to dconf: Timeout was reached
Sep 30 04:51:45 fedora.fritz.box systemd[1]: Starting dnf-makecache.service - dnf makecache...
Sep 30 04:51:54 fedora.fritz.box audit: BPF prog-id=134 op=LOAD
Sep 30 04:51:54 fedora.fritz.box audit: BPF prog-id=135 op=LOAD
Sep 30 04:51:54 fedora.fritz.box audit: BPF prog-id=136 op=LOAD
Sep 30 04:51:54 fedora.fritz.box systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Sep 30 04:51:54 fedora.fritz.box systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Sep 30 04:51:54 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:51:58 fedora.fritz.box dnf[19306]: Waiting for process with pid 10623 to finish.
Sep 30 04:53:23 fedora.fritz.box dnf[19306]: Fedora 38 - x86_64 84 kB/s | 19 kB 00:00
Sep 30 04:53:24 fedora.fritz.box dnf[19306]: Fedora 38 openh264 (From Cisco) - x86_64 9.8 kB/s | 989 B 00:00
Sep 30 04:53:24 fedora.fritz.box dnf[19306]: Fedora Modular 38 - x86_64 140 kB/s | 19 kB 00:00
Sep 30 04:53:24 fedora.fritz.box dnf[19306]: Fedora 38 - x86_64 - Updates 130 kB/s | 17 kB 00:00
Sep 30 04:53:24 fedora.fritz.box dnf[19306]: Fedora Modular 38 - x86_64 - Updates 143 kB/s | 18 kB 00:00
Sep 30 04:53:24 fedora.fritz.box dnf[19306]: RPM Fusion for Fedora 38 - Free 54 kB/s | 6.6 kB 00:00
Sep 30 04:53:25 fedora.fritz.box dnf[19306]: RPM Fusion for Fedora 38 - Free - Updates 59 kB/s | 6.2 kB 00:00
Sep 30 04:53:25 fedora.fritz.box dnf[19306]: teams 7.3 kB/s | 1.5 kB 00:00
Sep 30 04:53:25 fedora.fritz.box dnf[19306]: Metadata cache created.
Sep 30 04:53:25 fedora.fritz.box systemd[1]: dnf-makecache.service: Deactivated successfully.
Sep 30 04:53:25 fedora.fritz.box systemd[1]: Finished dnf-makecache.service - dnf makecache.
Sep 30 04:53:25 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-makecache comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:53:25 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dnf-makecache comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:53:25 fedora.fritz.box systemd[1]: dnf-makecache.service: Consumed 2.574s CPU time.
Sep 30 04:53:31 fedora.fritz.box PackageKit[4795]: resolve transaction /19709_ceebddbc from uid 1000 finished with success after 1913317ms
Sep 30 04:53:32 fedora.fritz.box PackageKit[4795]: refresh-cache transaction /19708_aeaadaae from uid 1000 finished with success after 621ms
Sep 30 04:53:33 fedora.fritz.box gnome-software[5874]: Only 1 apps for recent list, hiding
Sep 30 04:53:33 fedora.fritz.box PackageKit[4795]: resolve transaction /19710_adbcabcd from uid 1000 finished with success after 30ms
Sep 30 04:53:33 fedora.fritz.box PackageKit[4795]: resolve transaction /19711_acddeecc from uid 1000 finished with success after 30ms
Sep 30 04:53:54 fedora.fritz.box gnome-shell[5502]: Window manager warning: Ping serial 2316524 was reused for window W244, previous use was for window W243.
Sep 30 04:54:06 fedora.fritz.box evince[8425]: failed to commit changes to dconf: Timeout was reached
Sep 30 04:54:20 fedora.fritz.box systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Sep 30 04:54:20 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:54:20 fedora.fritz.box audit: BPF prog-id=136 op=UNLOAD
Sep 30 04:54:20 fedora.fritz.box audit: BPF prog-id=135 op=UNLOAD
Sep 30 04:54:20 fedora.fritz.box audit: BPF prog-id=134 op=UNLOAD
Sep 30 04:54:21 fedora.fritz.box PackageKit[4795]: resolve transaction /19712_aedacdeb from uid 1000 finished with success after 6772ms
Sep 30 04:54:22 fedora.fritz.box PackageKit[4795]: search-file transaction /19713_ceecacdc from uid 1000 finished with success after 730ms
Sep 30 04:54:45 fedora.fritz.box audit: BPF prog-id=137 op=LOAD
Sep 30 04:54:45 fedora.fritz.box audit: BPF prog-id=138 op=LOAD
Sep 30 04:54:45 fedora.fritz.box audit: BPF prog-id=139 op=LOAD
Sep 30 04:54:45 fedora.fritz.box systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Sep 30 04:54:45 fedora.fritz.box systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Sep 30 04:54:45 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:55:06 fedora.fritz.box gnome-shell[5502]: Window manager warning: Ping serial 2388815 was reused for window W253, previous use was for window W252.
Sep 30 04:55:10 fedora.fritz.box gnome-shell[5502]: Window manager warning: Ping serial 2393230 was reused for window W28, previous use was for window W31.
Sep 30 04:55:18 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmlogger_farm_check comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:55:18 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmlogger_check comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:55:18 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmlogger_farm_check comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:55:19 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmlogger_check comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:55:18 fedora.fritz.box systemd[1]: Starting pmlogger_check.service - Check pmlogger instances are running...
Sep 30 04:55:18 fedora.fritz.box systemd[1]: Starting pmlogger_farm_check.service - Check and migrate non-primary pmlogger farm instances...
Sep 30 04:55:18 fedora.fritz.box systemd[1]: Started pmlogger_farm_check.service - Check and migrate non-primary pmlogger farm instances.
Sep 30 04:55:18 fedora.fritz.box systemd[1]: Started pmlogger_check.service - Check pmlogger instances are running.
Sep 30 04:55:18 fedora.fritz.box systemd[1]: pmlogger_farm_check.service: Deactivated successfully.
Sep 30 04:55:19 fedora.fritz.box systemd[1]: pmlogger_check.service: Deactivated successfully.
Sep 30 04:55:19 fedora.fritz.box evince[8026]: failed to commit changes to dconf: Timeout was reached
Sep 30 04:55:33 fedora.fritz.box evince[8787]: failed to commit changes to dconf: Timeout was reached
Sep 30 04:56:31 fedora.fritz.box evince[8396]: failed to commit changes to dconf: Timeout was reached
Sep 30 04:56:31 fedora.fritz.box evince[8396]: Error: Timeout was reached
Sep 30 04:56:40 fedora.fritz.box gnome-shell[5502]: Window manager warning: Ping serial 2482787 was reused for window W262, previous use was for window W260.
Sep 30 04:56:43 fedora.fritz.box evince[8900]: failed to commit changes to dconf: Timeout was reached
Sep 30 04:56:45 fedora.fritz.box evince[8094]: failed to commit changes to dconf: Timeout was reached
Sep 30 04:56:53 fedora.fritz.box evince[8900]: Error: Timeout was reached
Sep 30 04:57:28 fedora.fritz.box systemd[1]: fwupd.service: Deactivated successfully.
Sep 30 04:57:28 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=fwupd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:57:58 fedora.fritz.box systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Sep 30 04:57:58 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:57:58 fedora.fritz.box audit: BPF prog-id=139 op=UNLOAD
Sep 30 04:57:58 fedora.fritz.box audit: BPF prog-id=138 op=UNLOAD
Sep 30 04:57:58 fedora.fritz.box audit: BPF prog-id=137 op=UNLOAD
Sep 30 04:58:00 fedora.fritz.box systemd[1]: Starting pmie_check.service - Check PMIE instances are running...
Sep 30 04:58:00 fedora.fritz.box audit: BPF prog-id=140 op=LOAD
Sep 30 04:58:00 fedora.fritz.box audit: BPF prog-id=141 op=LOAD
Sep 30 04:58:00 fedora.fritz.box audit: BPF prog-id=142 op=LOAD
Sep 30 04:58:00 fedora.fritz.box systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Sep 30 04:58:02 fedora.fritz.box systemd[1]: Started pmie_check.service - Check PMIE instances are running.
Sep 30 04:58:02 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmie_check comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:58:03 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:58:03 fedora.fritz.box systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Sep 30 04:58:08 fedora.fritz.box systemd[1]: pmie_check.service: Deactivated successfully.
Sep 30 04:58:08 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmie_check comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:58:18 fedora.fritz.box systemd[1]: Starting pmie_farm_check.service - Check and migrate non-primary pmie farm instances...
Sep 30 04:58:18 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmie_farm_check comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:58:18 fedora.fritz.box systemd[1]: Started pmie_farm_check.service - Check and migrate non-primary pmie farm instances.
Sep 30 04:58:21 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmie_farm_check comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:58:21 fedora.fritz.box systemd[1]: pmie_farm_check.service: Deactivated successfully.
Sep 30 04:58:27 fedora.fritz.box pcp-pmie[2997]: Restart unresponsive PMDAs pmdaroot[4]@fedora.fritz.box pmdalinux[4]@fedora.fritz.box
Sep 30 04:58:34 fedora.fritz.box evince[7940]: Error: Timeout was reached
Sep 30 04:59:24 fedora.fritz.box PackageKit[4795]: daemon quit
Sep 30 04:59:24 fedora.fritz.box systemd[1]: packagekit.service: Deactivated successfully.
Sep 30 04:59:24 fedora.fritz.box systemd[1]: packagekit.service: Consumed 8.522s CPU time.
Sep 30 04:59:24 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=packagekit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:59:28 fedora.fritz.box evince[8278]: failed to commit changes to dconf: Timeout was reached
Sep 30 04:59:40 fedora.fritz.box systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Sep 30 04:59:40 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 04:59:40 fedora.fritz.box audit: BPF prog-id=142 op=UNLOAD
Sep 30 04:59:40 fedora.fritz.box audit: BPF prog-id=141 op=UNLOAD
Sep 30 04:59:40 fedora.fritz.box audit: BPF prog-id=140 op=UNLOAD
Sep 30 04:59:44 fedora.fritz.box audit: BPF prog-id=143 op=LOAD
Sep 30 04:59:44 fedora.fritz.box audit: BPF prog-id=144 op=LOAD
Sep 30 04:59:44 fedora.fritz.box audit: BPF prog-id=145 op=LOAD
Sep 30 04:59:44 fedora.fritz.box systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Sep 30 04:59:44 fedora.fritz.box systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Sep 30 04:59:44 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 05:00:28 fedora.fritz.box evince[8145]: failed to commit changes to dconf: Timeout was reached
Sep 30 05:00:43 fedora.fritz.box wpa_supplicant[1218]: wlp3s0: WPA: Group rekeying completed with 3c:a6:2f:92:12:84 [GTK=CCMP]
Sep 30 05:01:01 fedora.fritz.box CROND[23490]: (root) CMD (run-parts /etc/cron.hourly)
Sep 30 05:01:03 fedora.fritz.box run-parts[23515]: (/etc/cron.hourly) starting 0anacron
Sep 30 05:01:05 fedora.fritz.box anacron[23536]: Anacron started on 2023-09-30
Sep 30 05:01:05 fedora.fritz.box anacron[23536]: Will run job `cron.daily' in 36 min.
Sep 30 05:01:05 fedora.fritz.box anacron[23536]: Jobs will be executed sequentially
Sep 30 05:01:05 fedora.fritz.box run-parts[23538]: (/etc/cron.hourly) finished 0anacron
Sep 30 05:01:05 fedora.fritz.box CROND[23489]: (root) CMDEND (run-parts /etc/cron.hourly)
Sep 30 05:01:58 fedora.fritz.box systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Sep 30 05:01:58 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 05:01:58 fedora.fritz.box audit: BPF prog-id=145 op=UNLOAD
Sep 30 05:01:58 fedora.fritz.box audit: BPF prog-id=144 op=UNLOAD
Sep 30 05:01:58 fedora.fritz.box audit: BPF prog-id=143 op=UNLOAD
Sep 30 05:03:10 fedora.fritz.box audit: BPF prog-id=146 op=LOAD
Sep 30 05:03:10 fedora.fritz.box audit: BPF prog-id=147 op=LOAD
Sep 30 05:03:10 fedora.fritz.box audit: BPF prog-id=148 op=LOAD
Sep 30 05:03:10 fedora.fritz.box systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Sep 30 05:03:10 fedora.fritz.box systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Sep 30 05:03:10 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 05:03:42 fedora.fritz.box gnome-shell[5502]: Window manager warning: Ping serial 2904934 was reused for window W107, previous use was for window W64.
Sep 30 05:03:44 fedora.fritz.box evince[8878]: failed to commit changes to dconf: Timeout was reached
Sep 30 05:03:44 fedora.fritz.box evince[8878]: Error: Timeout was reached
Sep 30 05:03:52 fedora.fritz.box evince[7938]: failed to commit changes to dconf: Timeout was reached
Sep 30 05:03:53 fedora.fritz.box evince[7938]: Error: Timeout was reached
Sep 30 05:06:14 fedora.fritz.box systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Sep 30 05:06:14 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 05:06:14 fedora.fritz.box audit: BPF prog-id=148 op=UNLOAD
Sep 30 05:06:14 fedora.fritz.box audit: BPF prog-id=147 op=UNLOAD
Sep 30 05:06:14 fedora.fritz.box audit: BPF prog-id=146 op=UNLOAD
Sep 30 05:06:54 fedora.fritz.box audit: BPF prog-id=149 op=LOAD
Sep 30 05:06:54 fedora.fritz.box audit: BPF prog-id=150 op=LOAD
Sep 30 05:06:54 fedora.fritz.box audit: BPF prog-id=151 op=LOAD
Sep 30 05:06:54 fedora.fritz.box systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Sep 30 05:06:54 fedora.fritz.box systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Sep 30 05:06:54 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 05:07:26 fedora.fritz.box evince[9034]: failed to commit changes to dconf: Timeout was reached
Sep 30 05:07:38 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 05:07:40 fedora.fritz.box audit: BPF prog-id=152 op=LOAD
Sep 30 05:07:40 fedora.fritz.box audit: BPF prog-id=153 op=LOAD
Sep 30 05:07:40 fedora.fritz.box audit: BPF prog-id=154 op=LOAD
Sep 30 05:07:40 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 05:07:38 fedora.fritz.box systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Sep 30 05:08:27 fedora.fritz.box pcp-pmie[2997]: Restart unresponsive PMDAs pmdaroot[4]@fedora.fritz.box pmdalinux[4]@fedora.fritz.box
Sep 30 05:07:40 fedora.fritz.box systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Sep 30 05:07:40 fedora.fritz.box systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Sep 30 05:07:59 fedora.fritz.box evince[9034]: Error: Timeout was reached
Sep 30 05:08:13 fedora.fritz.box evince[8257]: failed to commit changes to dconf: Timeout was reached
Sep 30 05:08:27 fedora.fritz.box evince[8651]: failed to commit changes to dconf: Timeout was reached
Sep 30 05:08:37 fedora.fritz.box evince[8622]: failed to commit changes to dconf: Timeout was reached
Sep 30 05:08:38 fedora.fritz.box evince[8257]: failed to commit changes to dconf: Timeout was reached
Sep 30 05:08:43 fedora.fritz.box evince[8651]: Error: Timeout was reached
Sep 30 05:08:47 fedora.fritz.box evince[8577]: failed to commit changes to dconf: Timeout was reached
Sep 30 05:08:52 fedora.fritz.box evince[8651]: failed to commit changes to dconf: Timeout was reached
Sep 30 05:08:53 fedora.fritz.box evince[8577]: Error: Timeout was reached
Sep 30 05:08:56 fedora.fritz.box audit: BPF prog-id=151 op=UNLOAD
Sep 30 05:08:56 fedora.fritz.box audit: BPF prog-id=150 op=UNLOAD
Sep 30 05:08:56 fedora.fritz.box audit: BPF prog-id=149 op=UNLOAD
Sep 30 05:08:57 fedora.fritz.box evince[8622]: Error: Timeout was reached
Sep 30 05:10:43 fedora.fritz.box wpa_supplicant[1218]: wlp3s0: WPA: Group rekeying completed with 3c:a6:2f:92:12:84 [GTK=CCMP]
Sep 30 05:11:45 fedora.fritz.box systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Sep 30 05:11:45 fedora.fritz.box audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 05:11:45 fedora.fritz.box audit: BPF prog-id=154 op=UNLOAD
Sep 30 05:11:45 fedora.fritz.box audit: BPF prog-id=153 op=UNLOAD
Sep 30 05:11:45 fedora.fritz.box audit: BPF prog-id=152 op=UNLOAD
Sep 30 05:12:57 fedora.fritz.box audit: BPF prog-id=155 op=LOAD
Sep 30 05:12:57 fedora.fritz.box audit: BPF prog-id=156 op=LOAD
Sep 30 05:12:57 fedora.fritz.box audit: BPF prog-id=157 op=LOAD
Sep 30 05:12:57 fedora.fritz.box systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Sep 30 05:12:57 fedora.fritz.box systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Sep 30 05:12:57 fedora.fritz.box audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 30 05:15:18 fedora.fritz.box evince[8211]: failed to commit changes to dconf: GDBus.Error:org.gtk.GDBus.UnmappedGError.Quark._g_2dfile_2derror_2dquark.Code8: Failed to create file “/home/xxx/.config/dconf/user.5EWXB2”: Read-only file system
Sep 30 05:15:18 fedora.fritz.box evince[8211]: Attempting to store changes into '/home/xxx/.local/share/recently-used.xbel', but failed: Failed to create file “/home/xxx/.local/share/recently-used.xbel.KRYXB2”: Read-only file system
Sep 30 05:15:18 fedora.fritz.box evince[8211]: Attempting to set the permissions of '/home/xxx/.local/share/recently-used.xbel', but failed: Read-only file system
Barry A Scott: I can chmod u-w / then journalctl and chmod u+w /
though I am a bit uneasy, in case something goes wrong.
One thing I haven’t mentioned: while my system was misbehaving, dmesg showed something about ‘impossible to rotate, journal corrupted’ - I didn’t copy the exact message and now it doesn’t show any more. Perhaps I should boot in read-only mode, and retrieve that, as Barry suggested, but I am uneasy to do that (I don’t think the chmod would be enough, and even that, I would rather not mess with).
A live system is like the one used to install Fedora, basically a Linux distribution running off a USB drive (or DVD if you still use these). You can use the one you used to install Fedora if you still have it.
While in the live system, try to mount the BTRFS partition and check what dmesg reports.
You are confusing the symptoms with the cause.
What i was getting at is that when linux boots the journal records are written to RAM. Once the root file system is mounted writable then the journal records are saved to the DISK.
But when the file system fails to mount writable, as is your situation, it is not possible to save the journal records. They are lost when you reboot.
When there is a problem like this with the filesystem, you should avoid anything that writes to the filesystem with a problem. You should boot from different media, such as a USB with a Live Linux Installer, make sure you have copies of important files (e.g., by
mounting the partition as read-only) and the checking that the drive is working properly.
Gnome Disks provides S.M.A.R.T status, but some drives report “healthy status” when the drive is actually failed. Some drive vendors provide tools (as images to make bootable USB drives) to check their drives as well as drive firmware updates/fixes. You can also check other forums for reports of problems with your drive and firmware version.
While running the “Live” system you have access to journalctl and dmesg until you reboot. You want to try mounting the problem partition read-only and check for error messages. Post them here.
Fedora’s btrfs has many recent improvements, so be careful about “easy btrfs repair” posts that use older versions. It is best to report the errors here so people familiar with the current btrfs versions can provide reliable advice.
Understood, thank you.
I’ll try to make a USB installer (I have upgraded online for a couple of years) and check this. When I do, I’ll post the output here.
I did run fsck -A, and it seemed to have made some repairs, things look stable now, but clearly I’ll have to keep checking.
My suggestion is to focus on diagnosis, not repair. This might be harder than fixing it, mainly because it takes more patience. But it’s also safe. Repairs aren’t always safe.
Corrupt systemd journals are semi-common, and journalctl skips over the corrupt entries (the whole journal file isn’t corrupt, just one or two entries - resulting in log rotation to avoid further problems with that file).
We really need to see kernel messages related to mounting the file system. All file system mount related problems are kernel messages, so yeah we need dmesg.
How to get it?
You could add rd.break=pre-mount as a kernel parameter, which will get you a prompt while still in the initramfs environment. It’s super limited and directories aren’t quite what you’d expect. Once at the prompt you can do: blkid and identify the Btrfs partition. And simply try to mount it, mount /dev/sdXY /sysroot (the mount point /sysroot is unique to the initramfs environment, other more familiar locations don’t yet exist). If the problem is with Btrfs, this mount will fail, and now you can get dmesg and take a cell phone photo and post it here.
Still another option is USB install media (any) and doing something similar. Boot, head to a terminal program, identify the partitions, mount the btrfs partition (or try to) and then dmesg and see what the kernel says the issue is.
OK re(read) the thread and see it was a one time read-only. So in that case, you can just reboot and it should ordinarily recover on its own. But you’re right to be suspicious.
I would use journalctl to look at previous kernel messages that have been recorded. However, because the file system went read only, the log might not have these messages because they couldn’t be written. We’d need dmesg at the time the error occurred in this case. My guess is the file system became confused and went read only to avoid the confusion being written to disk, which is the correct behavior.
Btrfs has a read time and write time tree checker to verify correctness of certain very important metadata prior to writing them to disk and if something is wrong (including even bit flips) the tree checker will trigger and flip the file system read only to prevent the confusion from being written to disk. It could be something like that. But what’s next depends on the actual error rather than speculation.
It’s safe to do drive tests, but also a good idea to do memory tests.
There’s some tradeoffs between pre-boot environment testers like memtest86 (and descendants) which test more memory but can be difficult to use. And linux user space testers like memtester (in Fedora repo) which are easier to use but test less memory because linux has to be booted first and any memory being used prior to initiating the test is not tested. You can limit this side effect somewhat by booting with kernel parameter 3 (same as systemd.unit=multi-user.target) which will boot to a prompt and leave out the entire graphical target stack, saving quite a lot of memory for testing.
@Chris Murphy
Thank you so much for the thorough answer.
My guess is the file system became confused and went read only to avoid the confusion being written to disk, which is the correct behavior.
This is what I was thinking, too (I didn’t know the mechanisms, but assumed there was an incompatibility and the system went read-only to avoid problems).
The problem with checking the disk, as everyone who has replied taught me, is that each step takes me a lot of time (because I don’t really know what I’m doing, I’m trying to be cautious - for instance I got an external hard-drive for extra backup, and now I have problems with that too, which I need to sort out before I do anything potentially dangerous).
I would like to test the disk - I will, I hope, as soon as things become slightly less hectic.
In the meantime: I think you’re right to mention memory. Another problem I’ve had is that some of the times when I close the lid, the system reboots (on its own). Someone said this may be a memory allocation problem, and guided me to look for it, but I couldn’t find the sort of error they were expecting to see. Last time this happened was this morning - would it be useful to get ‘dmsg’ or ‘journalctl’ for around the time when the system rebooted (when I closed the lid), and post the output here? Might this give you any hints about why the read-only problem happened - or do these two problems definitely not have the same cause?
This likely means an unclean shutdown. That 1) may confuse the filesystem if data “on the fly” did not make to the mass storage device and 2) means the last error messages are not preserved. There may be messages related to the power management configuration early in the boot process, and your system may need some kernel command-line option to handle lid closure properly.
You might find the tlp package useful, and the TLP Troubleshooting document is very thorough.
George N White III: I usually do a proper power down after the computer shuts itself down when I close the lid, in the hope it will restart in a ‘cleaner’ mode.
I’ve looked into this, including power management, a while ago, but nothing I tried has made a change. I am not sure if it’s related to the one-off problem with the system going read-only. I’ll give tlp another try.
Also, Chris Murphy: am still hoping to the memory test as you taught me. Apologies for going silent, things are very hectic right now.
I’ve installed memtester and trying to figure out how to use it. I read:
Note that the memory region will be overwritten during testing, so it is not
safe to specify memory which is allocated for the system or for other
applications; doing so will cause them to crash. If you absolutely must
test a particular region of actual physical memory, arrange to have that
memory allocated by your test software, and hold it in this allocated
state, then run memtester on it with this option.
So - I am not sure what to do.
Memtest86+ boots from a minimal system (USB key or using ventoy), so can test more of the memory than memtester. Memory tests should be run for a long time (more than a day). The latter is useful if you suspect memory problems after adding ram to system because you can use the system while it is testing the added RAM. You could start with
memtester set to leave enough RAM for basic use of the system, and avoid doing things that need more RAM.