Possible Change proposal: automate BTRFS maintenance actions

Project Discussion
, ,
1

Problem

Fedora uses BTRFS by default. This is a more complex filesystem than EXT4 or F2FS.

Supposedly, the performance decreases a lot if maintenance actions like scrub and balance are not ran for a long time.

Most users will never run these actions are there is nothing indicating the need. People dont read docs or “set calendar events to do a filesystem balance”.

Goal

BTRFS is the “click default install and go” option. Users should just install it and it should work.

This means that these technical actions should not need to be exposed to the users, but ran automatically.

Idea

I would like to create a change proposal, to run common BTRFS maintenance tasks on all Fedora variants using BTRFS.

These tasks take way less time and resources if they are ran every month.

Also, using scrub and balance is important to maintain performance.

Using systemd timers, priority setting etc, these can be fine tuned to run every month, with low priority.

The BTRFS task balance runs in the background automatically, so it just needs to be initiated.

sudo btrfs balance start / --full-balance &
sudo btrfs scrub start / &

These actions should not be ran at once but with some offset.

defragment may speed up performance on magnetical media (HDD, tape), but cause significant wear on flash drives (SDD, NVME, eMMC, SD). OR it may be beneficial, I am not sure what the correct info here is.

Prerequisites

The commands should run on / fine, on Atomic and Traditional variants.

They should only run if BTRFS is used, if it is not used, the timers should be disabled.

Topics

Who would like to join making this possible for implementation in F41?

Updates

  • BTRFS is better than EXT4 as it shows errors, misinterpreted as “having more errors”. It can automatically solve them, which this topic tries to do
  • the balance and scrub tasks work on Fedora Atomic Desktops, while others may not
  • btrfsmaintenance (link) is used on OpenSUSE, implements a lot of tasks and has configurable options
  • a majority of voters is:
    • pro automatic instead of with a message
    • pro monthly interval, instead of more frequently
    • contra including btrfs filesystem defragment but pro btrfs scrub and btrfs balance
5 Likes
2

I think this is a great idea especially since most users do not know that this isn’t already done. BTRFS has issues. Easing the issues by automating these tools would be a great help.

I think the jury is still out on BTRFS as a whole, Gamers notoriously have issues eventually with it, and making analogies from a Data Center, Cloud experiences do not translate to common users workloads.

I would go the extra step and add that Snapshot support needs to be added, but there are other issues to allow for that mechanism.

1 Like
3

Snapshot support in dnf absolutely needs to be added. But this is more complex and probably suitable for F42.

4

Would help to know where these issues are documented, bugzilla etc.
Do you have references for the proposal?

5

We already have btrfsmaintenance.noarch, but nothing suggests to users that it might be important for their use case, and guidance may be needed for cases where the default settings aren’t appropriate. Some users may prefer a mechanism that announces “time to perform btrfs maintenance” rather than a fully automated mechanism.

The installer could have a public service message:

You are configuring Fedora to run on btrfs. This filesystem requires periodic maintenance, which can be automated using btrfs-maintenance. Do you want to add btrfs-maintenance to the list packages that will be installed?

1 Like
6

Regarding scrub from the BTRFS documentation …

Scrub is not a filesystem checker (fsck) and does not verify nor repair structural damage in the filesystem. It really only checks checksums of data and tree blocks, it doesn’t ensure the content of tree blocks is valid and consistent. There’s some validation performed when metadata blocks are read from disk (Tree checker) but it’s not extensive and cannot substitute full btrfs-check(8) run

And it’s recommended usage …

The user is supposed to run it manually or via a periodic system service. The recommended period is a month but it could be less. The estimated device bandwidth utilization is about 80% on an idle filesystem.

Interestingly btrfs assistant defaults to one month cycle for automatic scrub of selected devices under btrfs maintanence.

On the topic of balance, it is more complex. There can be performance issues during use …

Balancing operations are very IO intensive and can also be quite CPU intensive, impacting other ongoing filesystem operations. Typically large amounts of data are copied from one location to another, with corresponding metadata updates.

I would recommend, if balance is to be considered for this auto maintenance to look at setting up some sane profiles to have filtering for automatic balance maintenance operations. Usually, I have found I use it only when removing or adding storage devices/partitions. Say a usage percentage filter, since this operation requires available space on the device where the balance operation is being performed, there is a risk of hitting the ENOSPC error as per …

The balance operation needs enough work space, i.e. space that is completely unused in the filesystem, otherwise this may lead to ENOSPC reports. See the section ENOSPC for more details.

Also this warning …

Running balance without filters will take a lot of time as it basically move data/metadata from the whole filesystem and needs to update all block pointers

I also think @chrismurphy might want to enlighten this conversation on what is currently the state of already configured and setup maintenance on btrfs in Fedora.

2 Likes
7

Thanks for the helpful comments

I have not tried it, but will this package prompt for actions?

@jakfrost I cannot add something useful here. Usage filters means that only often used data is balanced?

A GUI popup dialog allowing to run a balance before a shutdown may be useful. Maybe add it to the shutdown menu “do a filesystem cleanup and shutdown”.

This will confuse users, but it would be a good way to integrate when the device is not used.

Having a GUI popup could use Zenity and KDialog on the respective systems. Due to “UI Toolkit Cleanliness” it would likely need to be 2 different implementations for KDE, LXQt and GNOME, Mate, Cinnamon etc.

8

I think the information in the first post is based on outdated perception of BTRFS.
For my homeserver, I do run scrub and balance once a month, but the effect is minimal. Balance is really only slightly useful after you have created very large amounts of data or moved around very large amounts of data. And that may be respective to the actual total (free storage).
Even on my homeserver, btrfs balance running monthly doesn’t move around that much data at all.

Running it for ALL Fedora users, makes no sense to me. Especially without filters.

Not sure how people use Fedora for the past few years? Has anyone performed these actions on, say, a home laptop? I haven’t on any home laptop and minipcs.

On my homeserver, they are run automatically, first scrub, when finished, balance, once a month. But as said, my server runs totally fine not running it for 8 months (due to a typo in my maintenance script that I discovered yesterday :innocent:). This is how I run it for all my drives in my homeserver/NAS only:

btrfs scrub start -Bd -c 2 -n 4 /dev/nvme0n1p2
btrfs scrub start -Bd -c 2 -n 4 /dev/sda1
btrfs scrub start -Bd -c 2 -n 4 /dev/sdb1
btrfs scrub start -Bd -c 2 -n 4 /dev/sdc1

btrfs balance start -dusage=85 /
btrfs balance start -dusage=85 /mnt/drives/data0
btrfs balance start -dusage=85 /mnt/drives/data1
btrfs balance start -dusage=85 /mnt/drives/data2
2 Likes
9

Recently, on Btrfs, they introduced a knob for automatic balancing.
I created a post here

bg_reclaim_threshold
(RW, since: 5.19)

Used space percentage of total device space to start auto block group claim. Mostly for zoned devices.

https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#sysfs-interface

For further details here

Winning use cases:

It seems that Meta uses it with some benefits:
This can be worked around by:

  • enabling automatic reclaim
  • periodically running balance

The latter is widely deployed via btrfsmaintenance
(GitHub - kdave/btrfsmaintenance: Scripts for btrfs maintenance tasks like periodic scrub, balance, trim or defrag on selected mountpoints or directories.) and the former is used at scale at Meta with good results.

I think this patch aims to solve it automatically in the kernel: [PATCH v2 0/6] btrfs: dynamic and periodic block_group reclaim - Boris Burkov

For scrub, it does not involve any performance regression.

4 Likes
10

this sounds great! thanks

11

For those who want to enable “automatic balancing,” here is an excellent guide: Balancing a Btrfs filesystem | Forza's Ramblings

12

But 90% of it is outdated, people should only look at the bottom part and ignore the top :slight_smile: I wish he would move that part to the top and call the rest ‘archived’ or something like that.

1 Like
13

The underlying problem here is that btrfs seems to have a reputation for being a high-maintenance filesystem. No one is suggesting periodic maintenance for ext4 or xfs. Why is that?

14

We don’t have to go too far to see issues with BTRFS, Here on the forums, there’s threads on these issues. I don’t want to come off as someone with an axe to grind on BTRFS, but there’s clearly issues. We don’t even need to bring up RAID. . .

As a user of LVM+XFS. . . We’re good.

15

All filesystems require maintenance. Those legacy filesystems are like driving a car without doing periodic maintenance and only fixing it when something breaks. If your filesystem doesn’t detect bitrot, maintenance is deferred until a corrupt file causes a problem that can’t be ignored.

With btrfs in wider use we see reports of corrupt files on btrfs filesystems. Btrfs doesn’t increase the rate at which bits get corrupted, just makes them visible.

My work used XFS for many years on SGI IRIX64 to process remote sensing data from NASA and our own ground station. The files came with checksums and were stored on a robotic tape library and processed using striped XFS filesystems. Updates to algorithms often required reprocessing all the files. Bitrot meant that some files with a valid checksum at the last processing had changed (valid files were retrieved from another site). Btrfs automates the use of checksums and applies it to all files. Btrfs scrub verifies files and filesystem metadata using the checksums, and if a valid copy is available, repairs the damage, so it is not that difference from the manual maintenance we did using xfs.

2 Likes
16

So the forum is the only source of issues? None of which resulted in bugzilla?

1 Like
17

Comparing issues on Bugzilla and the Forums, with Bugzilla’s barrier to entry vs. Forums/Reddit and the like, when we understand the desktop user could potentially care less for one or the other? :thinking: Apples to Oranges IMO. . .

Note : I do understand the prioritzation of Bugzilla over the other. I’m merely adding the variability of a Desktop/Small Business user over Data Center/Cloud/Enterprise.

18

Many forum posts report btrfs checksum errors that ext4 or xfs would not detect, and that might have been repaired with btrfs scrub.

2 Likes
19

Many of which were hardware issues as I recall, that as you right say ext4 etc would not have detected.

20

I think running scrub and balance operations on btrfs filesystems would improve reliability. Especially for people who have a high rate of change and take/remove lots of snapshots. It is easy to have metadata challenges in those situations.

That being said, I have a few opinions on some of the above points:

  • btrfsmaintenance is a nice set of helpers but it isn’t required. Fedora could implement automation without this.
  • I am not a fan of automating btrfs balance with --full-balance. That is very intrusive. A more targeted balance would be much less impactful on the system and would solve most of the actual issues people experience.
  • Has anyone done any testing on the risks of any of these tasks? For example, if you hard power off a machine running a balance, does that increase the risk of the filesystem becoming corrupt?
2 Likes