Thank you very much to everyone for your help. For now, I’ve worked around the issue by creating a new system image on a new drive and migrating /home and /(root) with rsync. This let me get back to work quickly, although I did have to reconfigure some programs. Still, it’s just a workaround rather than a fix for the underlying problem, so when I find the time, I’ll return to the original disk and, with a fresh head, try to tackle it again using the steps you suggested.
Thank you again, and I look forward to talking to you soon.
Sorry about the giant walls of text and homework assignments. I’m genuinely curious if there’s a bug here, therefore I’m trying to collect information in case a bug report is needed. It should be possible to just back the file system out with a large file delete or delete a snapshot - if not, I’m inclined to think there’s a bug.
On another note, I would like to understand how the recurring issue of “full disk kills btrfs partition” could be mitigated. It’s something that I’ve seen sort of frequently in this forum and it often seems unrecoverable. would it make sense to reserve some space on btrfs partitions as spare that can be added when needed?
SGI IRIX64 wouldn’t let users exceed 90% of an XFS filesystem capacity. There was also a very noticeable drop in performance with a change in algorithm, I think at 85% of capacity. There are several conditions that should provide a highly visible warning to users: high temperature, low battery, filesystem/storage devices needing attention. Maybe these should have an identifier /tag that would make them easy to find in the journal. GUI environments could decide how to present such warnings to users.
It’s not a common problem. It is usually recoverable.
Ideally, if anyone runs into it, don’t try to fix it. Gather all available information, so we can treat it as a bug. Then fix it. I can understand if some folks don’t want to experience it and don’t want to report bugs. But there’s really only one way bugs get fixed which is solid problem reporting.
But for those who want to ensure they don’t experience such an issue, it’s fine to do this:
The btrfsmaintenance project is maintained by the upstream btrfs kernel and btrfs-progs maintainer. The balance timer runs on a schedule, not on demand, but it is a filtered balance so it’s fairly lightweight and shouldn’t take long or use significant system resources. This has been enabled on (open)SUSE for a long time so its behavior is well understood.
Soon, periodic and dynamic reclaim for data block groups will be enabled in the kernel, nothing to configure or enable, everyone will just get it. This works both on-demand and on a schedule. It won’t be necessary to also use the btrfs-balance.timer, but they also won’t conflict.
The patch is submitted upstream, but it might be another kernel release or two before it lands in a mainline kernel, and typically 4-6 weeks after upstream release it will be in Fedora stable updates.
I’ll do a separate write up on how to opt into this method using /etc/tmpfiles.d
GNOME Shell does provide low disk space warning. I’m able to trigger it well before Btrfs has any issues. But I think it should trigger sooner. I’ve filed a bug.
I’m genuinely curious if there’s a bug here, therefore I’m trying to collect information in case a bug report is needed. It should be possible to just back the file system out with a large file delete or delete a snapshot - if not, I’m inclined to think there’s a bug.