BTRFS corruption reports on Fedora 44 yet?

Ask Fedora
, ,
1

To clarify, this didn’t happen to me but are there any reports of kernel 6.19.14-300.fc44.x86_64 causing btrfs corruption?

BTRFS error (device nvme0n1p3): parent transid verify failed on logical 12435308 mirror 1 wanted 95 found 16
BTRFS error (device nvme0n1p3): parent transid verify failed on logical 12435308 mirror 2 wanted 95 found 16

Just got a report of a clean installs of both F44 KDE Plasma and F44 Workstation showing BTRFS corruption after offline upgrades over on the Fedora Discord, same user on the same PC
Basic system specs:
CPU: Ryzen 5 4500
Motherboard: Gigabyte B550 Gaming X V2
GPU: RX 5500XT

Nothing helpful to diagnose so I’ll summarize

  • User states they installed F44 KDE without running the self-integrity check
    • 20 minutes in they used Plasma Discover to update the system.
    • Reboots to the offline updates before completing and rebooting again
    • System boots the new kernel, and the system is left on the spinner splashscreen for 10 minutes
    • The user hops into the discord for help
    • I had them press ESC to view the boot log which gave the gave the error as in the 2nd screenshot below
    • System was unresponsive so I had them force reset the system and try all three kernels, which gave the same errors as above.
  • Given it was a brand new install, I directed them to do a clean install of F44 Workstation
    • F44 Workstation Installation USB passed the self-integrity check and Erase Disk was chosen to wipe the previous system
    • They run an offline update with GNOME Software
    • Boom, it fails with the same error as with the F44 KDE install previously
2

In both cases the updates were done with the gui using packagekit.

Have they tried another new install then performing the update with cli dnf?

3

I am not sure if this could be related, it draw my attention that I had a similar problem with an AMD Ryzen 5 computer too, the Intel upgrade was fine. It could be a coincidence who knows.

4

@computersavvy I’m that person. I have not tried updating through dnf as I’m not very used to Fedora yet. I have installed Fedora 43, and am going to test updating through dnf. I should also mention I ran into an error after pulling out the install media once the install completed.

I am currently upgrading all packages through dnf and am about to restart to see if the issue persists through Fedora 43.

error
error3072×4096 951 KB

Edit: I have just upgraded all system packages through dnf and nothing has crashed yet. It seems to only be an issue in F44 as far as I have been able to tell.

5

Short explanation:
The most typical reason for this error is something went wrong in the storage stack, i.e. writes didn’t make it to persistent media. So at the next boot, Btrfs doesn’t see what it’s expecting.

Long explanation:
The portion of the messages mirror 1 and mirror 2 means there’s two copies of the file system metadata, and both copies have the same problem. Because dup metadata writes are concurrent they’re both equally susceptible to being lost if the device doesn’t consistently honor flush or FUA.

We see flash drives do this sometimes persistently and it causes no problem unless there’s a power failure or forced reboot; or more frequently it’s a transient firmware bug, but also it won’t manifest unless there’s a power failure or forced reboot.

The problem has already occurred in the prior boot, so we’d need to see that dmesg to get a clue what may have occurred. Since the journal file is written to / and it’s the / file system that’s damaged and can’t be mounted, it’s not obvious how to access /var/log/journal but you might be able to use a live usb to boot, and use rescue mount mount -o ro,rescue=all which often permits Btrfs to skip over broken parts of the file system and mount read only. Then use journalctl -D to redirect to the journal log file.

Anyway, I think it’s more likely a transient bug in the flash drive firmware than in the kernel - because if it were a bug in the kernel, we’d see a lot more reports. But that doesn’t mean it’s not a kernel bug. It’s just hard to track these things down without logs. Primarily we need dmesg for the prior boot and current boot, untrimmed and unedited. And also btrfs check (implies readonly option).

A huge problem for figuring out issues is any attempt to repair wipes away the history of the file system state and we have no way of figuring anything out. It’s just the nature of repair.

6

One strategy is to preserve the original storage device unaltered and attempt repairs on an image of the problem drive. This gives you the option of trying different recovery procedures and means history may be available.

7

It’s possible to create a snapshot target with dmsetup - use the Btrfs block device as the base, and the overlay/cow device can be a file on another file system or even use a zram block device. For extra safety use blockdev to set the btrfs block device ro first.

8

That would make an interesting boot recovery option on the boot menu. Label it something like “Attempt transient recovery” and let it attempt to recover the filesystem with the overlay in place. If the recovery works, give the user the option to commit the overlay to the base image and reboot. If it doesn’t work, then nothing is lost because the changes were only done to the overlay.