BTRFS boot failure and easy GUI methods for system recovery




A few short how-to's with screenshots, showing easy ways to boot up off a USB live ISO and return back to previous points in time with BTRFS

Read More
2 Likes

You may find mention on forums about using Timeshift on Fedora Linux BTRFS with Ubuntu style @ sub-volumes. This now no longer works unless you are using something like Linux Mint. It was never probably a good idea to start with but it does illustrate the extents to which people have gone to, and just to get some kind of easy GUI recovery setup going.

Would you mind clarifying what you mean by β€œwas never probably a good idea to start with”? Is there something inherently flawed with Ubuntu style @ sub-volumes and/or Timeshift?

I use both Fedora with snapper/BTRFS Assistant and Debian with Timeshift and neither have failed me.

I used BTRFS on my work computer (fedora linux) some years ago and was generally happy. This is a great article describing some nice new tools but the issue I had back then was cleaning up a full file system. On a non BTRFS system this is simple - delete no longer needed files and get some space for new ones. But on BTRFS with snapshots you get no space back because the blocks are still held by snapshots so you would need to find and delete those same files in sometimes multiple snapshots before space is recovered. Hopefully by now there is a better way.

Timeshift on ext4 shouldn’t β€˜fail’ you, whether that’s on Fedora Linux or with any other standard system that supports it. It’s only when you start trying to apply Timeshift to BTRFS that the snags and issues start to happen.

I don’t think that there is anything inherently wrong or β€˜flawed’ with prefixing BTRFS sub-volumes with an β€˜@’ in itself. It certainly does indicate to the user that there’s something non-standard about a folder or section of their file system. But on the other hand, it can make things look all a bit extrapolated and disconnected to standard notation, so I personally prefer the Fedora Linux approach.

The systems are the systems that we have though, whatever the decisions of senior managers or steering committees.

The main point of what I was trying to say, is that it is not generally a good idea to try forcing two basically different systems into configurations that neither set of system developers are expecting to see. If one team is going in direction X and the other team in direction Y, then eventually, at some point, one of the sides will release code that breaks what you are trying to do …

We need to remember that Timeshift is maintained by Linux Mint. Their repo on Github explains the system’s limitations.

An interesting avenue might be spinning up a Mint Debian Edition as a BTRFS virtual machine and seeing how the Mint setup might apply to a standard Debian one, perhaps ?

But having several years of core β€˜long term support’ on a desktop that will usually have extra non-updated software added to it over time is something that should also be taken into account …

It can be quite tricky to actually see how much space is really being taken up on a CoW system in general. Sometimes it’s best not to pay too much attention to the figures.

You can get a lot of double counting and false reporting that can make the system seem to be taking up more space than it is.

It’s interesting to see all the general file system add-ons that are getting added to the Linux kernel every month. There have been a lot of recent improvements.

1 Like

Balance is for balancing RAID devices

Note: A periodic balance on non-raid devices is useful to prevent premature ENOSPC situations which happen when the filesystem cannot allocate new data or metadata chunks, even if raw disk space appears available.

Yes and No. But also trying to keep things simple and within the scope of easy GUI methods.

Balance is a bit of questionable item when running on single disk. Everyone runs out of disk space at some point > Error No Space (enospc)

Yes, possibly in hind-sight, the article should have expanded a little more on this. Possibly.

The Btrfs docs state on line one that the β€œprimary purpose of the balance feature is to spread block groups across all devices”

They also go on to say that the process can be very intensive and, somewhat ironically, won’t run if there isn’t enough space there to start with.

What they don’t say, at least directly, is that running a balance occasionally may help with adjusting the way that space reported. That’s discussed on a separate a troubleshooting page.

Heavy deletions of files and snapshots can cause the Data reporting to go slightly out of sync with the Meta-Data logs.

The BTRFS-ASSISTANT wiki gives no discussion on the matter, in the same way that it also omits suggestions on snapper numbers.

However, following on from your comment, I did do some looking at the btrfsmaintenance code that installs along with the assistant. These are quite surprisingly old SUSE scripts that have been packaged together. The btfrs balance commands are up-to-date but I am not sure that all the script calls are linking up properly on their journey from the assistant.

If you run the btrfs-assistant balance command from the front menu panel it seems to by-pass the SUSE config script that adds filters to reduce the running intensities.

What I would currently recommend is to run the balance command at the terminal. Adding =50 filters will speed things up considerably.

Try sudo btrfs balance start -dusage=50 -musage=50 -v /
with sudo btrfs fi usage -b /
before and after to see the difference.

2 Likes

There’s also a nifty utility called btrfs-balance-least-used in the python3-btrfs rpm package, that makes balances way faster and smarter by moving less data, in stages.
It is most impressive when used on big, active volumes that where never balanced before, where an unfiltered full balance would take days. In this scenario btrfs-balance-least-used, easily does an unfiltered full balance in just minutes.

1 Like

Additional notes

The mechanism of system recovery was the main focus of the original article, and I wanted to write things down while my experiences and learning were still fresh and new in my mind.

With my need to get the article finished, I was constrained to place a certain amount of reliance on the official documents.

From extensive further research, there are now some additional observations and clarifications to make on maintenance and configuration.

Then there are the things that we just do automatically. It can be all too easy to forget to mention these important snippets. Power-save disabling is precisely one of those things.


Backup and partition adjustment

  • Make sure to DISABLE POWER-SAVE and screen lock features before attemping any of these procedures.

Idle time detectors won’t register what is happening. Discard any .img files that have had power supply or idle time interruptions. Even if they appear okay, they will usually contain hidden errors. Repeat any partition restores that have been interrupted too.


On the positive side, something to add is that Gparted is BTRFS aware. Remembering of course that any partition work is never 100% guaranteed, this does mean that Gparted should handle all the sub-volume adjustments. Tests that I have carried out on resizing show this to be the case. This works in both directions.

Snapper

I have now revised my auto-snapshot settings and discovered that man snapper-configs as well as excessive with its defaults, is not as clear as it could be either.

Working out that numbers only refers to the number of boot snapshots and reducing this figure from 50 to 5 saved an estimated 7 GiB of data space and reduced metadata usage by nearly 50%

BEFORE

@fedora:\~$ sudo btrfs fi us -b /

    Device size:                      153771573248

    Device allocated:                  96703873024

    Device unallocated:                57067700224

    Device missing:                              0

    Device slack:                       1073741824

    Used:                              84112801792

    Free (estimated):                  66286321664      (min: 37752471552)

    Free (statfs, df):                 66285273088

    Data ratio:                               1.00

    Metadata ratio:                           2.00

    Global reserve:                      319062016      (used: 0)

    Multiple profiles:                          no


Data,single: Size:83751862272, Used:74533240832 (88.99%)

   /dev/nvme1n1p5       83751862272


Metadata,DUP: Size:6442450944, Used:4789764096 (74.35%)

   /dev/nvme1n1p5       12884901888


@fedora:\~$ sudo btrfs fi du -s /.snapshots

     Total   Exclusive  Set shared  Filename

   1.31TiB    15.53GiB    39.51GiB  /.snapshots


@fedora:\~$ sudo ls /.snapshots | grep -e ^ -c

71

AFTER

@fedora:\~$ sudo snapper -v cleanup all

@fedora:\~$ sudo btrfs fi us -h /

    Device size:                 143.21GiB

    Device allocated:             90.06GiB

    Device unallocated:           53.15GiB

    Device missing:                  0.00B

    Device slack:                  1.00GiB

    Used:                         64.50GiB

    Free (estimated):             73.06GiB      (min: 46.49GiB)

    Free (statfs, df):            73.06GiB

    Data ratio:                       1.00

    Metadata ratio:                   2.00

    Global reserve:              302.62MiB      (used: 0.00B)

    Multiple profiles:                  no


Data,single: Size:78.00GiB, Used:58.09GiB (74.47%)

   /dev/nvme0n1p5         78.00GiB


Metadata,DUP: Size:6.00GiB, Used:3.21GiB (53.44%)

   /dev/nvme0n1p5         12.00GiB


@fedora:\~$ sudo btrfs fi du -s /.snapshots

     Total   Exclusive  Set shared  Filename

 485.31GiB    21.81GiB    23.52GiB  /.snapshots

@fedora:\~$ sudo snapper list -a

Config: home, subvolume: /home

  \# β”‚ Type   β”‚ Pre # β”‚ Date                          β”‚ User β”‚ Cleanup  β”‚ Description β”‚ Userdata

────┼────────┼───────┼───────────────────────────────┼──────┼──────────┼─────────────┼─────────

  0 β”‚ single β”‚       β”‚                               β”‚ root β”‚          β”‚ current     β”‚

153 β”‚ single β”‚       β”‚ Sat 01 Aug 2026 12:00:05 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

282 β”‚ single β”‚       β”‚ Mon 31 Aug 2026 06:00:14 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

291 β”‚ single β”‚       β”‚ Tue 01 Sep 2026 08:00:01 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

314 β”‚ single β”‚       β”‚ Tue 08 Sep 2026 10:00:14 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

330 β”‚ single β”‚       β”‚ Mon 14 Sep 2026 08:00:30 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

351 β”‚ single β”‚       β”‚ Sun 20 Sep 2026 11:00:44 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

356 β”‚ single β”‚       β”‚ Mon 21 Sep 2026 09:00:03 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

360 β”‚ single β”‚       β”‚ Tue 22 Sep 2026 07:00:00 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

365 β”‚ single β”‚       β”‚ Tue 22 Sep 2026 13:00:07 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

366 β”‚ single β”‚       β”‚ Tue 22 Sep 2026 15:00:04 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

367 β”‚ single β”‚       β”‚ Wed 23 Sep 2026 09:00:17 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

368 β”‚ single β”‚       β”‚ Wed 23 Sep 2026 10:00:02 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

369 β”‚ single β”‚       β”‚ Wed 23 Sep 2026 11:00:52 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

370 β”‚ single β”‚       β”‚ Wed 23 Sep 2026 12:00:21 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

371 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 10:00:10 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

372 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 11:00:10 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

373 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 12:00:04 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

374 β”‚ single β”‚       β”‚ Fri 25 Sep 2026 11:00:01 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

375 β”‚ single β”‚       β”‚ Fri 25 Sep 2026 12:00:08 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

376 β”‚ single β”‚       β”‚ Fri 25 Sep 2026 13:00:08 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚


Config: root, subvolume: /

  \# β”‚ Type   β”‚ Pre # β”‚ Date                          β”‚ User β”‚ Cleanup  β”‚ Description β”‚ Userdata

────┼────────┼───────┼───────────────────────────────┼──────┼──────────┼─────────────┼─────────

  0 β”‚ single β”‚       β”‚                               β”‚ root β”‚          β”‚ current     β”‚

242 β”‚ single β”‚       β”‚ Sat 01 Aug 2026 12:00:05 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

460 β”‚ single β”‚       β”‚ Mon 31 Aug 2026 06:00:14 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

472 β”‚ single β”‚       β”‚ Tue 01 Sep 2026 08:00:01 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

512 β”‚ single β”‚       β”‚ Tue 08 Sep 2026 10:00:14 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

542 β”‚ single β”‚       β”‚ Mon 14 Sep 2026 08:00:30 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

581 β”‚ single β”‚       β”‚ Sun 20 Sep 2026 11:00:44 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

589 β”‚ single β”‚       β”‚ Mon 21 Sep 2026 09:00:03 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

597 β”‚ single β”‚       β”‚ Tue 22 Sep 2026 07:00:00 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

605 β”‚ single β”‚       β”‚ Tue 22 Sep 2026 13:00:07 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

607 β”‚ single β”‚       β”‚ Tue 22 Sep 2026 15:00:04 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

610 β”‚ single β”‚       β”‚ Wed 23 Sep 2026 09:00:17 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

612 β”‚ single β”‚       β”‚ Wed 23 Sep 2026 10:00:02 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

613 β”‚ single β”‚       β”‚ Wed 23 Sep 2026 11:00:52 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

614 β”‚ single β”‚       β”‚ Wed 23 Sep 2026 12:00:21 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

615 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 09:03:46 CEST β”‚ root β”‚ number   β”‚ boot        β”‚

616 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 09:07:44 CEST β”‚ root β”‚ number   β”‚ boot        β”‚

617 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 10:00:11 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

618 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 11:00:10 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

619 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 11:48:05 CEST β”‚ root β”‚ number   β”‚ boot        β”‚

620 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 12:00:04 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

621 β”‚ single β”‚       β”‚ Fri 25 Sep 2026 10:57:16 CEST β”‚ root β”‚ number   β”‚ boot        β”‚

622 β”‚ single β”‚       β”‚ Fri 25 Sep 2026 10:59:58 CEST β”‚ root β”‚ number   β”‚ boot        β”‚

623 β”‚ single β”‚       β”‚ Fri 25 Sep 2026 11:00:01 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

624 β”‚ single β”‚       β”‚ Fri 25 Sep 2026 12:00:08 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚

625 β”‚ single β”‚       β”‚ Fri 25 Sep 2026 13:00:08 CEST β”‚ root β”‚ timeline β”‚ timeline    β”‚
@fedora:\~$ sudo snapper list -a | grep number

615 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 09:03:46 CEST β”‚ root β”‚ number   β”‚ boot        β”‚

616 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 09:07:44 CEST β”‚ root β”‚ number   β”‚ boot        β”‚

619 β”‚ single β”‚       β”‚ Thu 24 Sep 2026 11:48:05 CEST β”‚ root β”‚ number   β”‚ boot        β”‚

621 β”‚ single β”‚       β”‚ Fri 25 Sep 2026 10:57:16 CEST β”‚ root β”‚ number   β”‚ boot        β”‚

622 β”‚ single β”‚       β”‚ Fri 25 Sep 2026 10:59:58 CEST β”‚ root β”‚ number   β”‚ boot        β”‚

Note also that boot snaps are only on the root sub-volume and that records of /home are only timeline or manual

Btrfs balance

Looking at, and on reflecting on, the btrfs docs my feeling is that the term balance must have been coined at a time when RAID HDD’s needed to be striped to get faster parallel output of the same file from two drives.

We now want SSD’s in duplicate to cover against drive failure. RAID 1 is now the default.

Although I intially referenced the official documents, my feeling is now that the balance notes are out-of-date and have become somewhat misleading.

The term balance has to now be seen in the framework of general system equilbrium rather than as a method of getting an equal and balanced amount of data onto a parallel pair of drives. Either that or the term should be deprecated and a new one given, like repack, consolidate or reallocate.


I now RECOMMEND putting the btrfs assistant balance setting at weekly

On an SSD with a 150 GiB main partition, the automated -dusage=10 routine takes just over 1 whole second …

For a manual space-defragmentation use sudo btrfs -v balance start -dusage=40 -musage=5 /

No more than -dusage=50 -musage=10


In simple terms, try to imagine that the file system divides up and pre-allocates data blocks on the drive. The data gets written to a convenient block and its position and percentage occupancy of the block is recorded by the metadata. A bit like stacking things in boxes and then labeling them.

The metadata blocks need plenty of free space in them to be unimpeded when recording data changes. Compacting these beyond =10 will have negative effects when the disk is getting full. The number of metadata blocks will auto-adjust, depending on drive size and usage, and on the number of snapshots.

Some blocks are full, others only partly. When we balance the drive’s equilibrium, we look for blocks that are somewhat empty and try to fill them up with bits and bobs from somewhere else. This makes the free space less fragmented, easier to write to and easier to measure.

We don’t need to set the data usage filter to any greater than =50 as we can’t fit two blocks with greater than 50% usage in together with each other.

On tests carried out on repeated-roll-back identical state virtual machines, btrfs balance start -dusage=42 -musage=8 / and btrfs-balance-least-used -u 42 / were almost identical.

Repeat START POINT:

sudo btrfs fi df /

Data, single: total=26.01GiB, used=22.02GiB

Metadata, DUP: total=3.00GiB, used=1.68GiB

sudo time -f %E  btrfs balance start -dusage=42 -musage=8 /

Done, had to relocate 3 out of 31 chunks

0:01.31


sudo btrfs fi df /

Data, single: total=24.01GiB, used=22.03GiB

Metadata, DUP: total=3.00GiB, used=1.68GiB

sudo time -f %E btrfs-balance-least-used -u 42 /

Loading block group objects with used_pct <= 42 ... found 2

Balance block group vaddr 32477544448 used_pct 15 ... duration 0 sec total 31

Balance block group vaddr 28182577152 used_pct 30 ... duration 0 sec total 30

0:01.04


btrfs fi df /

Data, single: total=24.01GiB, used=22.23GiB

Metadata, DUP: total=3.00GiB, used=1.68GiB

Running as high as btrfs-balance-least-used -u 80 / achieved no more space compaction.


On the Snapper 50 to 5 number reduction:

BEFORE

sudo btrfs fi us -h /

    Device size:                 143.21GiB

    Device allocated:             90.06GiB

    Device unallocated:           53.15GiB

    Device missing:                  0.00B

    Device slack:                  1.00GiB

    Used:                         64.50GiB

    Free (estimated):             73.06GiB      (min: 46.49GiB)

    Free (statfs, df):            73.06GiB

    Data ratio:                       1.00

    Metadata ratio:                   2.00

    Global reserve:              302.62MiB      (used: 0.00B)

    Multiple profiles:                  no


Data,single: Size:78.00GiB, Used:58.09GiB (74.47%)

   /dev/nvme0n1p5         78.00GiB


Metadata,DUP: Size:6.00GiB, Used:3.21GiB (53.44%)

   /dev/nvme0n1p5         12.00GiB


AFTER

sudo btrfs balance start -dusage=50 -musage=5 /

Done, had to relocate 4 out of 79 chunks


sudo btrfs fi us -h /

    Device size:                 143.21GiB

    Device allocated:             81.06GiB

    Device unallocated:           62.15GiB

    Device missing:                  0.00B

    Device slack:                  1.00GiB

    Used:                         64.50GiB

    Free (estimated):             73.06GiB      (min: 41.98GiB)

    Free (statfs, df):            73.06GiB

    Data ratio:                       1.00

    Metadata ratio:                   2.00

    Global reserve:              251.22MiB      (used: 0.00B)


Data,single: Size:69.00GiB, Used:58.09GiB (84.19%)

   /dev/nvme0n1p5         69.00GiB


Metadata,DUP: Size:6.00GiB, Used:3.21GiB (53.44%)

   /dev/nvme0n1p5         12.00GiB

Note the Free (estimated): 73.06GiB and Used: 64.50GiB remain identical. The defragmented space gives a higher MIN free space estimate but the actual free space remains the same.

Resize

Something to note here on the last before/after is the Device slack: 1.00GiB

All SSD’s should be setup with spare unallocated space around the partions. This allows the firmware to trim bad sectors and keep the drive healthy.

I decided to use the command sudo btrfs filesystem resize -1G / to add a bit of breathing room on the actual partition itself. I am not sure if BTRFS will use this automatically. The docs say that it will use unallocated space in a tight squeeze but whether that means device or volume unallocated is not clear. Irrespectively, I thought at least it could help with an enospc situation.

HDD’s

Users that are still running on rotational devices may need to consider using manual only settings for SCRUB

RAID

The Fedora installer will automatically use mkfs.btrfs defaults when two drives are given as targets and will create RAID 1 which creates duplicate (mirrored) disks.

1 Like

When everyone’s done fighting with Snapper and ready to give up on periodic snapshots, try btrbk for a breath of fresh air.

Also, I recommend not setting nodatacow on anything that matters. This sacrifices btrfs data resiliency features; the most prominent being checksums (nodatacow implies nodatasum). Might as well run FAT32 at that point (like FAT32, btrfs does not journal).

A β€œdaily-driver” VM used for mainstream business applications is likely to perform well even with the underlying virtual disk file highly fragmented – between the VM’s OS disk caching and the host’s OS disk caching the user experience is likely to be just fine.

My personal experience is with Windows guests on btrfs-hosted virtual disks approaching a half million fragments. After one puts away benchmarking toys and starts using the VM for actual work they’re likely to find nothing amiss. Something something perfect enemy of the good and all…

Never saw that before, thanks for pointing that out. I don’t find snapper difficult to use though, especially with BTRFS Assistant.

I’ll try it on a VM, I’m interested to see if I can hook it to grub-btrfs and the automatic snapshotting with dnf operations.