I fcked up my brtfs drive by updating, is there anything I can do to save it?

Ask Fedora
, ,
1

Hi all,

I installed updates via “dnf update”, but I didn’t check if I still had space leftover, I suspect I didn’t as during this update my Fedora 40 KDE crashed hard.

I restarted but after seeing the KDE login screen the screen remained black.

After some googling, I tried via cmdline to balance the btrfs drive as btrfs fileusage showed my metadata size to be quite full.

However balanceing didn’t work as there was no space, it reported.

Another internet suggestion was to add an USB device to btrfs temporarily for rebalancing such that I can remove some space later.

Adding the USB device did the trick & I could rebalance & remove some files.
However removing the device gave an error (can’t remember what)

Here I made the mistake I guess of restarting as Fedora would no longer boot & be stuck unable to mount my partition.

Even with the USB device still present I could no longer mount at all.

After many tries with btrfs check, repair & in the end restore I got most of my files back. & I can now mount my partition with:

mount -o degraded,ro,rescue=all /dev/nvme0n1p6 /mnt

but some files are corrupted. 1 specific 400 gb file I couldn’t restore

Is there any chance I can restore them?

output of

btrfs filesystem usage /mnt

WARNING: cannot read detailed chunk info, per-device usage will not be shown, run as root
Overall:
    Device size:                   1.91TiB
    Device allocated:              1.90TiB
    Device unallocated:           13.57GiB
    Device missing:                7.47GiB
    Device slack:                 16.00EiB
    Used:                          1.72TiB
    Free (estimated):            196.41GiB      (min: 189.62GiB)
    Free (statfs, df):           185.94GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:1.85TiB, Used:1.67TiB (90.33%)

Metadata,DUP: Size:28.00GiB, Used:27.50GiB (98.21%)

System,DUP: Size:8.00MiB, Used:240.00KiB (2.93%)

I know it’s a long shot, but thanks in advance!

4

Google isn’t your friend with Btrfs, sorry. There’s a lot of spaghetti throwing at walls.

If Btrfs mishaves, it’s important to change the file system as little as possible. It’s best to balance with filters, to limit the changes to resolve the immediate problem rather than using it as a big hammer.

Adding a device to give it more room is a neat trick but comes at very high risk which is that the file system is now spread across two devices and if the 2nd device is not properly removed from the Btrfs pool before being physically removed from the computer - it results in Btrfs corruption because parts of the file system have been physically removed. And then there’s the reboot - and now mounting it degraded - I’d say too many major changes were made to fix it.

If you use -o degraded,ro,rescue=usebackuproot,ibadroots I wonder if it will let you copy out that 400G file using ddrescue pointed at the file?

Btrfs, with datacsum checking enabled, checks each 4K block for integrity. If there’s a csum mismatch, btrfs issues an EIO error to user space for that block. It doesn’t turn over corrupt data. Whether you get a hole or zeros, is up to the application. Most applications just fail at that point, they don’t try to read subsequent blocks. What you want is to get all the good blocks, and an error message for the bad blocks - the error message is found in dmesg during the copy.

So you can at least get most of the 400GB file, most likely, and be informed which blocks are bad. It might help knowing what portion of the file is good vs not.

2 Likes
5

Thank you for your reply, it’s very heartening to hear that my google-fu skills weren’t the problem here and that btrfs is just a bit of pain.

I’m sadly unable to try your suggestion, after obtaining part of my 400 gb file through “btrfs restore”, I gave up and wiped my drive to start fresh. Losing only one file in a complete system wipe without proper backups is still a win in my book.

So now that I’ve started fresh, are there any precautions I can take to prevent this issue from happening again? Besides having proper backups of everything important & less important (working on it)

Thanks!

6

Btrfs still has edge cases, however rare, that cause it to become confused. The problem is always panic, and trying to fix the problem without understanding the problem. But then the problem with patience is you have better things to do than deal with a confused file system. That isn’t supposed to happen in the first place.

I’m not sure what happened in your particular case - there are many out of space tests being done on Btrfs prior to each release. But tests aren’t perfect. File systems are the most non-deterministic thing we have in computing. As they age, no two file systems are alike. Therefore I can’t tell you exactly how to prevent it other than, don’t panic.

You can ask for help in https://matrix.to/#/#fedora:fedoraproject.org - that’s a good first step, the more complex cases I can help with, just ping cmurf. The discussion forum I may not check for weeks.

If the problem is pretty urgent and you haven’t gotten a response soon enough, you can ask on btrfs channel in libera.chat - in fact you can go there first if you want. But Fedora is a community and the community fully backs Btrfs. We take in bugs, report them upstream, we have Btrfs developers who can help. So for a random question about Btrfs, maybe btrfs, for help with a specific problem, it’s a coin toss. I can more easily escalate in Fedora backchannel than btrfs can - usually. But developers and experts lurk there so it’s a good resource.

You might consider installing btrfs-maintenance package and enabling the btrfs-balance.timer

dnf install btrfsmaintenance
systemctl enable btrfs-balance.timer

This periodically runs a minimalist filtered balance of the file system. The minimalist approach actually works best, and it’s entirely automated.

Extra technical details:
Btrfs dynamically allocates space for metadata and data into separate block groups. So long as the ratio for data and metadata is mostly the same as the file system ages, this works OK. But if there’s large changes in the ratio, in particular when the file system is nearly full, it’s possible the file system runs out of space in one type of block group before the other. So it really is out of space even if some space remains. Normally the file system can handle this situation, including having reserve free space to successfully delete files. The catch is that Btrfs needs free space to delete files. Deleting files actually consumes space, it doesn’t immediately free it. That’s because it’s copy on write. It must write the fact a file is deleted, before it can free the blocks that file is using.

The btrfs-balance.timer/service runs a filtered balance that relocates the extents in the least used block groups, freeing their space, making it possible to allocate that space for either metadata or data. This aids the dynamical allocation of block group, making confusion less likely when the file system is nearly full.

btrfsmaintenance is maintained by upstream Btrfs developers, it’s safe.

I don’t use it myself, mainly because if I run into a problem like this, I’m likely to consider it a bug, and report the bug upstream. Any balance so dramatically changes the file system layout that it makes a bug report less useful. So I tend to not fix things at all - gather a bunch of relevant dianostic information, file a bug report, and then fix the problem. But not everyone is interested in filing bug reports - you have better things to do, so btrfsmaintenance is helpful - although I’m not certain if it would have prevented the problem you ran into.

2 Likes
7

I also asked in the Fedora Btrfs Matrix room. In kernel 6.11, we have a new “dynamic block group reclaim” feature, which might be helpful instead of enabling a system timer.

  • dynamic block group reclaim:
    • tunable framework to avoid situations where eager data allocations
      prevent creating new metadata chunks due to lack of unallocated
      space
    • reuse sysfs knob bg_reclaim_threshold (otherwise used only in zoned
      mode) for a fixed value threshold
    • new on/off sysfs knob “dynamic_reclaim” calculating the value based
      on heuristics, aiming to keep spare working space for relocating
      chunks but not to needlessly relocate partially utilized block
      groups or reclaim newly allocated ones
    • stats are exported in sysfs per block group type, files “reclaim_*”
    • this may increase IO load at unexpected times but the corner case of
      no allocatable block groups is known to be worse
8

It’ll take some time to better understand if that will really obviate the filtered balance.

9

There is a tradeoff between the effort needed to do backups and routine maintenance on btrfs that reduce the chances of encountering an edge case and the effort needed to repair or restore a damaged filesystem – potentially when you are facing a tight deadline.