"No space left on device" But I have free space left

Ask Fedora
, , ,
25

Unallocated went to zero. Therefore no way to add more metadata block groups. Therefore eventually the existing ones got filled up.

The workload does matter, and how it ages matters too.

I do not manually balance at all in years, but I have recently enabled dynamic reclaim. If you are manually rebalancing (not really recommended for most use cases) do a filtered balance to limit it to data block groups. The btrfsmaintenance package has systemd service and timer units that uses filtered balance, so that’s OK too if you’re not interested in testing and reporting problems.

I personally would rather run into problems, and then get them reported upstream for broader improvement. But I realize this is not everyone’s cup of tea!

26

Does this mean that btrfs cannot allocate new metadata blocks on the second device, only data blocks, or is it unable to allocate metadata as this would also require free blocks for metadata on device 1 to make the allocation on device 2?

similar to xfs. I encountered something like this in a middle size environment ~150 - 200 systems. At some point, their NFS filesystems would be unable to create new files / inodes. FS had to be extended to allow xfs to allocate new blocks for inodes.
It turned out that the xfs filesystems were all heavily fragmented, and at some point all free data blocks did consist of single, max three consecutive blocks only! But XFS needs at least 4 consecutive blocks = 16KiB for inodes!
For some reason they had the nfs mount option noac enabled on all nfs clients, which had overridden virtually every optimization xfs can do to prevent/limit fragmentation. And ofc. the write performance was horrible. :scream:

The python script reminds me of xfs_db -r -c 'freesp -s' <device> or xfs_db -r -c 'frag ' <device>.

27

If starting out with a single device btrfs, and adding a 2nd device, it should be able to allocate any block group type.

Adding the device implies a kind of mkfs - superblocks will be written, and then because btrfs must update the device btree but can’t write to device 1, therefore must write to device 2, it must first create a metadata chunk on device 2, and that means updating the chunk btree which is stored in system bg. So a totally full single device btrfs, adding a 2nd device means creating at least a system (chunk btree) and metadata (all other btrees) chunk on that 2nd device. Right away. So that’ll either succeed or fail.

it’s potentially trickier with more devices.

28

Thanks for this post and to all the helpers. I was certainly very confused when I was getting the “No space left on device” errors when I was seeing 77G of free space on the drive.

Symptoms were being unable to update, write files, couldn’t open certain apps (like browsers, disk usage analyzer), frequently getting kicked to the user login screen (following a short black screen), couldn’t use the GNOME system menu Power button (it would just do the black screen → login screen thing). Thankfully poweroff still worked.

Due to being kicked to login screen frequently, I booted into a Fedora 44 live USB and mounted the BTRFS volume for the rest.

$ sudo btrfs filesystem usage /run/media/liveuser/fedora
Overall:
   Device size:                    236.89GiB
   Device allocated:               236.89GiB
   Device unallocated:               1.00MiB
   Device missing:                     0.00B
   Device slack:                       0.00B
   Used:                           159.13GiB
   Free (estimated):                76.92GiB     (min: 76.92GiB)
   Free (statfs, df):               76.92GiB
   Data ratio:                          1.00
   Metadata ratio:                      2.00
   Global reserve:                 432.16MiB     (used: 0.00B)
   Multiple profiles:                     no

Data,single: Size:220.87GiB, Used:143.95GiB (65.18%)
  /dev/nvme0n1p3        220.87GiB

Metadata,DUP: Size:8.00GiB, Used:7.59GiB (94.86%)
  /dev/nvme0n1p3         16.00GiB

System,DUP: Size:8.00MiB, Used:48.00KiB (0.59%)
  /dev/nvme0n1p3         16.00MiB

Looking at the above, Device allocated + unallocated (and maybe the Metadata) should be what stand out, yes?

First balance attempt failed with “No space left on device”, unsurprisingly.

$ sudo btrfs balance start -dusage=50 /run/media/liveuser/fedora

Followed by sudo dmesg | tail, which mentioned ‘77 enospc errors during balance’

Unlike the original poster, I could afford to delete some files. I deleted one of my Steam games that was >1GiB, ran the balance command with -dusage=50 again and that fixed the issue for me- ‘Device unallocated’ now showed as 80GiB, and I was able to run some updates normally. Yay!

I’ll look into enabling dynamic reclaim now. Is it still recommended? From what I can tell, it was being tested last year, but I’m not seeing it mentioned in more recent discussions, and it doesn’t appear to have been enabled by default yet.