BTRFS root disk forced readonly after regular reboot

Ask Fedora
, ,
1

Hello.

I had a quite scary BTRFS incident yesterday after doing a cold restart (safe shutdown, power on) on my Fedora machine. I was able to log in through GDM to GNOME just fine however various applications would not start and running flatpak update reported Error: Read-only file system. Some investigation later by checking dmesg and BTRFS had dumped a very scary looking list of logs and forced my root mount into read only, as verified by looking at /proc/mounts

[   20.254522] RPC: Registered named UNIX socket transport module.
[   20.254527] RPC: Registered udp transport module.
[   20.254529] RPC: Registered tcp transport module.
[   20.254531] RPC: Registered tcp-with-tls transport module.
[   20.254533] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   20.481422] nvme nvme0: using unchecked data buffer
[   20.580035] block nvme0n1: No UUID available providing old NGUID
[   20.676579] NET: Registered PF_QIPCRTR protocol family
[   22.023018] Generic FE-GE Realtek PHY r8169-0-500:00: attached PHY driver (mii_bus:phy_addr=r8169-0-500:00, irq=MAC)
[   22.180319] r8169 0000:05:00.0 enp5s0: Link is Down
[   25.137435] r8169 0000:05:00.0 enp5s0: Link is Up - 1Gbps/Full - flow control rx/tx
[   27.816555] rfkill: input handler disabled
[   34.681426] rfkill: input handler enabled
[   36.548789] rfkill: input handler disabled
[   80.415069] page: refcount:2 mapcount:0 mapping:00000000e18d003d index:0x18166940 pfn:0x368fd9
[   80.415080] memcg:ffff8a2840979180
[   80.415084] aops:btree_aops ino:1
[   80.415094] flags: 0x17ffffce00422e(referenced|uptodate|lru|workingset|private|writeback|node=0|zone=2|lastcpupid=0x1fffff)
[   80.415103] raw: 0017ffffce00422e fffff8224db2e788 fffff8224da3f688 ffff8a284ffbdb00
[   80.415108] raw: 0000000018166940 ffff8a2a72795d10 00000002ffffffff ffff8a2840979180
[   80.415111] page dumped because: eb page dump
[   80.415115] BTRFS critical (device dm-0): corrupt leaf: root=7 block=1655283384320 slot=216, bad key order, prev (18446744073709551606 128 1431407063040) current (18446603336221196278 128 1431407067136)
[   80.415127] BTRFS info (device dm-0): leaf 1655283384320 gen 3973095 total ptrs 369 free space 722 owner 7
[   80.415133] 	item 0 key (18446744073709551606 EXTENT_CSUM 1431396388864) itemoff 16275 itemsize 8
[   80.415138] 		range start 1431396388864 end 1431396397056 length 8192
[   80.415141] 	item 1 key (18446744073709551606 EXTENT_CSUM 1431396397056) itemoff 16271 itemsize 4
[   80.415145] 		range start 1431396397056 end 1431396401152 length 4096
[   80.415148] 	item 2 key (18446744073709551606 EXTENT_CSUM 1431396401152) itemoff 16263 itemsize 8
[   80.415152] 		range start 1431396401152 end 1431396409344 length 8192
[   80.415155] 	item 3 key (18446744073709551606 EXTENT_CSUM 1431396409344) itemoff 16255 itemsize 8
[   80.415159] 		range start 1431396409344 end 1431396417536 length 8192

(37 similar lines omitted to decrease post size)

[   80.417672] 	item 366 key (18446744073709551606 EXTENT_CSUM 1431412924416) itemoff 10055 itemsize 16
[   80.417676] 		range start 1431412924416 end 1431412940800 length 16384
[   80.417679] 	item 367 key (18446744073709551606 EXTENT_CSUM 1431413071872) itemoff 9959 itemsize 96
[   80.417684] 		range start 1431413071872 end 1431413170176 length 98304
[   80.417687] 	item 368 key (18446744073709551606 EXTENT_CSUM 1431413170176) itemoff 9947 itemsize 12
[   80.417691] 		range start 1431413170176 end 1431413182464 length 12288
[   80.417694] BTRFS error (device dm-0): block=1655283384320 write time tree block corruption detected
[   88.423983] BTRFS: error (device dm-0) in btrfs_commit_transaction:2536: errno=-5 IO failure (Error while writing out transaction)
[   88.423994] BTRFS info (device dm-0 state E): forced readonly
[   88.424000] BTRFS warning (device dm-0 state E): Skipping commit of aborted transaction.
[   88.424003] BTRFS error (device dm-0 state EA): Transaction aborted (error -5)
[   88.424007] BTRFS: error (device dm-0 state EA) in cleanup_transaction:2021: errno=-5 IO failure
[   88.424014] BTRFS: error (device dm-0 state EA) in btrfs_sync_log:3564: errno=-5 IO failure
[  113.681719] BTRFS error (device dm-0 state EA): failed to run delalloc range, root=256 ino=79648483 folio=0 submit_bitmap=0 start=0 len=4096: -30
[  180.488513] BTRFS error (device dm-0 state EMA): remounting read-write after error is not allowed

Since BTRFS had (correctly from what I’ve read) forced the mount into read only after getting confused, I performed another cold restart. After that, my system mounted fine and as rw by checking /proc/mounts. Checking dmesg showed systemd complaining about a corrupted journal file.

[   43.930943] BTRFS info (device dm-0 state M): use zstd compression, level 1
[   43.933662] audit: type=1130 audit(1773681397.929:11): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   43.954309] systemd-journald[1554]: Received client request to flush runtime journal.
[   43.979483] systemd-journald[1554]: /var/log/journal/3a21d69ee10444cf971d7eee12b27344/system.journal: Journal file corrupted, rotating.
[   43.979492] systemd-journald[1554]: Rotating system journal.

Using BTRFS assistant I ran a scrub to see the damage. The scrub report a single uncorrectable error and in dmesg complained about a single file with a bad checksum.

UUID:             92d1675b-93d8-4ffb-a278-be015ccf7957
Scrub started:    Mon Mar 16 17:46:58 2026
Status:           finished
Duration:         0:02:27
Total to scrub:   367.41GiB
Rate:             2.50GiB/s
Error summary:    csum=1
  Corrected:      0
  Uncorrectable:  1
  Unverified:     0

[ 1118.215995] BTRFS info (device dm-0): scrub: started on devid 1
[ 1130.738636] BTRFS error (device dm-0): scrub: unable to fixup (regular) error at logical 1758288281600 on dev /dev/mapper/luks-d52b5a13-5291-49be-ab07-4fc34873056b physical 26116227072
[ 1130.739867] BTRFS warning (device dm-0): scrub: checksum error at logical 1758288281600 on dev /dev/mapper/luks-d52b5a13-5291-49be-ab07-4fc34873056b, physical 26116227072 root 256 inode 76156717 offset 4096 length 4096 links 1 (path: user/.var/app/com.usebottles.bottles/data/bottles/bottles/SCP---Containment-Breach/drive_c/users/steamuser/Documents/SCP - Containment Breach/1.3.11/SFX/Radio/Buzz.ogg)
[ 1130.739880] BTRFS error (device dm-0): bdev /dev/mapper/luks-d52b5a13-5291-49be-ab07-4fc34873056b errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[ 1269.691367] BTRFS info (device dm-0): scrub: finished on devid 1 with status: 0

The corrupt file had its permissions intact however couldn’t be opened.

I don’t see anything immediately concerning reported by smartctl. The only things that stood out was a Error Information Log Entries: 1, which appears to be mundane from looking online, and a high Data Units Written, which from tracking down my SSD’s datasheet is well under the manufacturers specified write endurance.

My system seems to be functional but I’m still cautious as to if things are ok. Are there any additional steps I could take to verify that my drive isn’t silently dying? For BTRFS, are there any additional actions I should take regarding the uncorrectable error?

Do note that I am using BTRFS under LUKS, as is indicated by the mapped luks device.

2

slot=216 means item 216 has the problem, however …

(37 similar lines omitted to decrease post size)
[ 80.417672] item 366 key (18446744073709551606 EXTENT_CSUM 1431412924416) itemoff 10055 itemsize 16

Unfortunately one of those lines omitted is what I need to see what went wrong. However you did include this:

[ 80.417694] BTRFS error (device dm-0): block=1655283384320 write time tree block corruption detected

The write time tree checker is a kind of limited on-line fsck. It’s checking for the correctness of certain critical metadata before they get written to disk, and to prevent wrong things from making it to disk.

This is actually a good error because there’s a pretty good chance that some memory corruption made the metadata wrong, and the write time tree checker probably prevented it from making it to disk, which would have permanently written the confusion to persistent media.

So yeah, the correct action is to just reboot. And it’s a good idea you did scrub afterward.

[ 1130.739867] BTRFS warning (device dm-0): scrub: checksum error at logical 1758288281600 on dev /dev/mapper/luks-d52b5a13-5291-49be-ab07-4fc34873056b, physical 26116227072 root 256 inode 76156717 offset 4096 length 4096 links 1 (path: …

This means a particular 4KiB data block is corrupt. And there is no way to fix it because there’s only one copy of the data.

What this tells me is you should do some aggressive memory tests. I think you’re seeing bitflips and it’s sometimes hitting the file system because Btrfs is checksumming and verifying everything including data, which is an enormously large target for corruption compared to just metadata. Btrfs is not designed to be a memory tester, so while it exposes memory issues it doesn’t tell us for sure it’s bad RAM. We kinda have to infer this.

There’s a bunch of different memory testers out there. I have read that some more nuanced problems with RAM can take a long time (days of testing) for memtest86 and variants to find. The good thing is that it’s pretty straightforward to use and most any RAM manufacturer will know about it and honor their warranty when memory fails such a test.

memtester is a user space program, it’s in the Fedora repo. This reportedly is much more aggressive, finds memory errors pretty quickly. However the draw back is it tests less memory because quite a large chunk (up to 2G) is used by kernel and user space. You can limit the untested portion by booting in non-graphical mode and running memtester from a shell, rather than the graphical Terminal program.

File systm is fine, just one corrupt block in a file. But I recommend testing memory and replacing it or issues will continue to pop up.