Disk space does not grow despite large files having been deleted

Ask Fedora
, , ,
1

Hey all,

despite using Linux for some time now, there’s something new here I have no idea how to solve.
I’m using a 2TB SSD in my Framework Laptop 13 with Fedora 38 Workstation on a large 1.8TB partition with LUKS encryption.
Running VMWare for some Windows machines, however, the computer was very sluggish yesterday and I found out that the SSD ran almost completely full. It seems typical for SSDs to freeze up for some time when the space usage is near their designed capacity.
So I moved some of the lesser-used VMs to an external drive, in total around 700 GB, expecting this space to become free sooner or later. However, since this operation the machine has been running for hours now with no change in free capacity. I would expect at least 1TB to be available again, but it keeps telling me 53.5GB in Nautilus and the following in df (focus on /dev/dm-0):

Filesystem      1K-blocks       Used Available Use% Mounted on
devtmpfs             4096          0      4096   0% /dev
tmpfs            32815124       1052  32814072   1% /dev/shm
tmpfs            13126052       2488  13123564   1% /run
/dev/dm-0      1951834112 1892027356  52203172  98% /
/dev/dm-0      1951834112 1892027356  52203172  98% /home
             <snipped several irrelevant snap loop file systems>
tmpfs            32815124        212  32814912   1% /tmp
/dev/nvme0n1p2     996780     282076    645892  31% /boot
/dev/nvme0n1p1     613184      87580    525604  15% /boot/efi
tmpfs             6563024        192   6562832   1% /run/user/1000

lsblk:

root|~ ▷ lsblk /dev/nvme0n1
NAME                                          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
nvme0n1                                       259:0    0  1,8T  0 disk  
├─nvme0n1p1                                   259:1    0  600M  0 part  /boot/efi
├─nvme0n1p2                                   259:2    0    1G  0 part  /boot
└─nvme0n1p3                                   259:3    0  1,8T  0 part  
  └─luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 253:0    0  1,8T  0 crypt /home
                                                                        /var/lib/snapd/snap
                                                                        /

df:

root|~ ▷ df /
Filesystem                                             1K-blocks       Used Available Use% Mounted on
/dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 1951834112 1892027472  52203088  98% /

du:

root|/ ▷ sudo du -h -d 1 -c
361M	./boot
1,1M	./dev
1,1T	./home
0	./proc
2,7M	./run
0	./sys
12K	./tmp
64M	./root
0	./afs
51M	./etc
0	./lost+found
0	./media
0	./mnt
3,4G	./opt
0	./srv
24G	./usr
29G	./var
1,1T	.
1,1T	total

So how does this compute? It’s a 1.8TB capacity partition with 1.1TB actually allocated as du proves. Shouldn’t that leave 0.7TB free, now that I have deleted the respective amount of space and files? How come the volume is still considered 98% full?
The main problem now is that I couldn’t move the VMs back that I moved off to the external drive when needed. Eventually the disk is considered so full that none of them would be allowed to be copied back to the SSD. It’s almost like moving the VM files away caused the partition to shrink accordingly but it’s all still allocated like it always has been.

I have tried to run fstrim but it brought back only around 3GB of more free space, far from the 700GB expected. This might have to do with the fact that the partition in question is LUKS-encrypted but there must be a concept of freeing up unused space, right?
I’m too inexperienced to try around with fsck and its warnings really drove me off. Data loss is not an option. Backups do exist but restoring an entire OS along with all the data would take me days.
Found some other web sites where, among other circumstances, open files would not actually be deleted and their space freed up until the program having handles on the file(s) in question closes the handles. If that were the case, a reboot should sort it out, which it didn’t.
And of course I have checked the trashcan and the Lost+Found folders, both containing nothing.
So I’m really lost. What has become of these 700GB?

Does anybody know what is going on here?

Thank you for any input!

Cheers,
Joe

2

You seem to have run into a problem similar to others when a btrfs file system becomes nearly full. It does not readily release the space, and there are some btrfs commands which may assist in recovering the unused space.

I do not use btrfs on a regular basis so am not familiar enough with the appropriate commands to make suggestions but others here should be able to provide the needed help.

These threads provide some info but I have no first hand experience and those are only 2 of many that may assist.

3

Please copy/paste the output from sudo fpaste --btrfsinfo --printonly.

4

Also useful: sudo btrfs subvolume list / or just let us know if you’re making use of many snapshots, e.g. btrbk, timeshift, snapper.

And podman/docker/moby with the btrfs graph driver enabled will also make prolific use of subvolumes, snapshots, and may not clean up after itself aggressively enough. There’s an obscure concept called bookened extents that could also be a factor.

5

Okay, here are the results from the commands suggested:

johannes|~ ▷ sudo btrfs subvolume list /
Place your right index finger on the fingerprint reader
ID 256 gen 690040 top level 5 path home
ID 257 gen 690040 top level 5 path root
ID 258 gen 690014 top level 257 path var/lib/machines
ID 303 gen 689946 top level 5 path .veeam_snapshots/{fb96a14a-421a-4e98-b461-be290fc516ef}/256_home
ID 304 gen 689946 top level 5 path .veeam_snapshots/{fb96a14a-421a-4e98-b461-be290fc516ef}/257_root

johannes|~ ▷ sudo fpaste --btrfsinfo --printonly
=== fpaste 0.4.4.0 System Information ===
* Kernel (uname -r ; cat /proc/cmdline):
     6.1.12-200.fc37.x86_64
     BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.1.12-200.fc37.x86_64 root=UUID=38073a1e-e0c4-46e7-9f5e-d7175b820147 ro rootflags=subvol=root rd.luks.uuid=luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 rhgb quiet
     
* btrfs-progs (rpm -q btrfs-progs):
     btrfs-progs-6.5.1-1.fc38.x86_64
     
* btrfs mounts (grep btrfs /proc/mounts):
     /dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 / btrfs rw,seclabel,relatime,compress=zstd:1,ssd,space_cache=v2,subvolid=257,subvol=/root 0 0
     /dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 /var/lib/snapd/snap btrfs rw,seclabel,relatime,compress=zstd:1,ssd,space_cache=v2,subvolid=257,subvol=/root 0 0
     /dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 /home btrfs rw,seclabel,relatime,compress=zstd:1,ssd,space_cache=v2,subvolid=256,subvol=/home 0 0
     
* block devices (lsblk -o NAME,FSTYPE,SIZE,FSUSE%,MOUNTPOINT,UUID,MIN-IO,SCHED,DISC-GRAN,MODEL):
     NAME                                          FSTYPE        SIZE FSUSE% MOUNTPOINT                                 UUID                                 MIN-IO SCHED DISC-GRAN MODEL
     loop0                                         squashfs     80,7M   100% /var/lib/snapd/snap/authy/20                                                       512 none         4K 
     loop1                                         squashfs     70,5M   100% /var/lib/snapd/snap/authy/21                                                       512 none         4K 
     loop2                                         squashfs        4K   100% /var/lib/snapd/snap/bare/5                                                         512 none         4K 
     loop3                                         squashfs    105,8M   100% /var/lib/snapd/snap/core/16091                                                     512 none         4K 
     loop4                                         squashfs    105,8M   100% /var/lib/snapd/snap/core/16202                                                     512 none         4K 
     loop5                                         squashfs     55,7M   100% /var/lib/snapd/snap/core18/2785                                                    512 none         4K 
     loop6                                         squashfs     55,7M   100% /var/lib/snapd/snap/core18/2790                                                    512 none         4K 
     loop7                                         squashfs     73,9M   100% /var/lib/snapd/snap/core22/858                                                     512 none         4K 
     loop8                                         squashfs     73,9M   100% /var/lib/snapd/snap/core22/864                                                     512 none         4K 
     loop9                                         squashfs    164,8M   100% /var/lib/snapd/snap/gnome-3-28-1804/194                                            512 none         4K 
     loop10                                        squashfs    164,8M   100% /var/lib/snapd/snap/gnome-3-28-1804/198                                            512 none         4K 
     loop11                                        squashfs    496,9M   100% /var/lib/snapd/snap/gnome-42-2204/132                                              512 none         4K 
     loop12                                        squashfs      497M   100% /var/lib/snapd/snap/gnome-42-2204/141                                              512 none         4K 
     loop13                                        squashfs     81,3M   100% /var/lib/snapd/snap/gtk-common-themes/1534                                         512 none         4K 
     loop14                                        squashfs     91,7M   100% /var/lib/snapd/snap/gtk-common-themes/1535                                         512 none         4K 
     loop15                                        squashfs    115,5M   100% /var/lib/snapd/snap/insomnia/231                                                   512 none         4K 
     loop16                                        squashfs    115,5M   100% /var/lib/snapd/snap/insomnia/233                                                   512 none         4K 
     loop17                                        squashfs      5,4M   100% /var/lib/snapd/snap/magnus/393                                                     512 none         4K 
     loop18                                        squashfs      5,4M   100% /var/lib/snapd/snap/magnus/430                                                     512 none         4K 
     loop19                                        squashfs      3,6M   100% /var/lib/snapd/snap/protracker/502                                                 512 none         4K 
     loop20                                        squashfs      173M   100% /var/lib/snapd/snap/signal-desktop/525                                             512 none         4K 
     loop21                                        squashfs      173M   100% /var/lib/snapd/snap/signal-desktop/532                                             512 none         4K 
     loop22                                        squashfs     40,8M   100% /var/lib/snapd/snap/snapd/20092                                                    512 none         4K 
     loop23                                        squashfs     40,9M   100% /var/lib/snapd/snap/snapd/20290                                                    512 none         4K 
     zram0                                                         8G        [SWAP]                                                                            4096              4K 
     nvme0n1                                                     1,8T                                                                                           512 none       512B T-FORCE TM8FPZ002T
     ├─nvme0n1p1                                   vfat          600M    14% /boot/efi                                  CE16-4217                               512 none       512B 
     ├─nvme0n1p2                                   ext4            1G    28% /boot                                      a5734783-873c-4c36-a8b5-fe89f7c3bc86    512 none       512B 
     └─nvme0n1p3                                   crypto_LUKS   1,8T                                                   e19affbf-1032-448b-9f84-7c877cbfb2a4    512 none       512B 
       └─luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 btrfs         1,8T    97% /home                                      38073a1e-e0c4-46e7-9f5e-d7175b820147    512            512B 
     
* Kernel messages (journalctl -k -o short-monotonic --no-hostname | grep "Linux version\| ata\|Btrfs\|BTRFS\|] hd\| scsi\| sd\| sdhci\| mmc\| nvme\| usb\| vd"):
     [    0.000000] kernel: Linux version 6.1.12-200.fc37.x86_64 (mockbuild@bkernel01.iad2.fedoraproject.org) (gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4), GNU ld version 2.38-25.fc37) #1 SMP PREEMPT_DYNAMIC Wed Feb 15 04:35:34 UTC 2023
     [    0.996536] kernel: usbcore: registered new interface driver usbfs
     [    0.996536] kernel: usbcore: registered new interface driver hub
     [    0.996536] kernel: usbcore: registered new device driver usb
     [    1.474228] kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01
     [    1.474230] kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
     [    1.474231] kernel: usb usb1: Product: xHCI Host Controller
     [    1.474231] kernel: usb usb1: Manufacturer: Linux 6.1.12-200.fc37.x86_64 xhci-hcd
     [    1.474232] kernel: usb usb1: SerialNumber: 0000:00:0d.0
     [    1.474368] kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.01
     [    1.474369] kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
     [    1.474370] kernel: usb usb2: Product: xHCI Host Controller
     [    1.474371] kernel: usb usb2: Manufacturer: Linux 6.1.12-200.fc37.x86_64 xhci-hcd
     [    1.474371] kernel: usb usb2: SerialNumber: 0000:00:0d.0
     [    1.477133] kernel: usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01
     [    1.477134] kernel: usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
     [    1.477134] kernel: usb usb3: Product: xHCI Host Controller
     [    1.477135] kernel: usb usb3: Manufacturer: Linux 6.1.12-200.fc37.x86_64 xhci-hcd
     [    1.477136] kernel: usb usb3: SerialNumber: 0000:00:14.0
     [    1.478730] kernel: usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.01
     [    1.478731] kernel: usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
     [    1.478731] kernel: usb usb4: Product: xHCI Host Controller
     [    1.478732] kernel: usb usb4: Manufacturer: Linux 6.1.12-200.fc37.x86_64 xhci-hcd
     [    1.478733] kernel: usb usb4: SerialNumber: 0000:00:14.0
     [    1.478960] kernel: usb: port power management may be unreliable
     [    1.479190] kernel: usbcore: registered new interface driver usbserial_generic
     [    1.479193] kernel: usbserial: USB Serial support registered for generic
     [    1.490763] kernel: usbcore: registered new interface driver usbhid
     [    1.490764] kernel: usbhid: USB HID core driver
     [    1.512942] kernel: Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity=yes
     [    1.927779] kernel: usb 2-4: new SuperSpeed Plus Gen 2x1 USB device number 2 using xhci_hcd
     [    1.942226] kernel: usb 2-4: New USB device found, idVendor=17ef, idProduct=a391, bcdDevice= d.24
     [    1.942233] kernel: usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
     [    1.942235] kernel: usb 2-4: Product: USB3.1 Hub
     [    1.942237] kernel: usb 2-4: Manufacturer: VIA Labs, Inc.
     [    1.985642] kernel: usb 3-6: new high-speed USB device number 2 using xhci_hcd
     [    2.113342] kernel: usb 3-6: New USB device found, idVendor=17ef, idProduct=a392, bcdDevice= d.24
     [    2.113345] kernel: usb 3-6: New USB device strings: Mfr=1, Product=2, SerialNumber=0
     [    2.113346] kernel: usb 3-6: Product: USB2.0 Hub
     [    2.113347] kernel: usb 3-6: Manufacturer: VIA Labs, Inc.
     [    2.351497] kernel: usb 3-7: new high-speed USB device number 3 using xhci_hcd
     [    2.471599] kernel: usb 2-4.1: new SuperSpeed USB device number 3 using xhci_hcd
     [    2.484229] kernel: usb 2-4.1: New USB device found, idVendor=17ef, idProduct=a387, bcdDevice=31.03
     [    2.484235] kernel: usb 2-4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=6
     [    2.484238] kernel: usb 2-4.1: Product: USB-C Dock Ethernet
     [    2.484240] kernel: usb 2-4.1: Manufacturer: Realtek
     [    2.484242] kernel: usb 2-4.1: SerialNumber: 301000001
     [    2.505064] kernel: usb 3-7: New USB device found, idVendor=0bda, idProduct=5634, bcdDevice= 0.21
     [    2.505071] kernel: usb 3-7: New USB device strings: Mfr=3, Product=1, SerialNumber=2
     [    2.505073] kernel: usb 3-7: Product: Laptop Camera
     [    2.505076] kernel: usb 3-7: Manufacturer: Generic
     [    2.505078] kernel: usb 3-7: SerialNumber: 200901010001
     [    2.535714] kernel: usbcore: registered new interface driver r8152
     [    2.549610] kernel: usb 2-4.3: new SuperSpeed USB device number 4 using xhci_hcd
     [    2.564111] kernel: usb 2-4.3: New USB device found, idVendor=17ef, idProduct=a393, bcdDevice= d.23
     [    2.564114] kernel: usb 2-4.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
     [    2.564115] kernel: usb 2-4.3: Product: USB3.1 Hub
     [    2.564116] kernel: usb 2-4.3: Manufacturer: VIA Labs, Inc.
     [    2.621488] kernel: usb 3-9: new full-speed USB device number 4 using xhci_hcd
     [    2.627716] kernel: usb 2-4.1: reset SuperSpeed USB device number 3 using xhci_hcd
     [    2.749404] kernel: usb 3-9: New USB device found, idVendor=27c6, idProduct=609c, bcdDevice= 1.00
     [    2.749406] kernel: usb 3-9: New USB device strings: Mfr=1, Product=2, SerialNumber=3
     [    2.749407] kernel: usb 3-9: Product: Goodix USB2.0 MISC
     [    2.749408] kernel: usb 3-9: Manufacturer: Goodix Technology Co., Ltd.
     [    2.749409] kernel: usb 3-9: SerialNumber: UID806AC057_XXXX_MOC_B0
     [    2.788125] kernel: nvme 0000:01:00.0: platform quirk: setting simple suspend
     [    2.788156] kernel: nvme nvme0: pci function 0000:01:00.0
     [    2.892655] kernel: usb 3-6.3: new high-speed USB device number 5 using xhci_hcd
     [    3.033528] kernel: usb 3-6.3: New USB device found, idVendor=17ef, idProduct=a394, bcdDevice= d.23
     [    3.033532] kernel: usb 3-6.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
     [    3.033534] kernel: usb 3-6.3: Product: USB2.0 Hub
     [    3.033535] kernel: usb 3-6.3: Manufacturer: VIA Labs, Inc.
     [    3.033622] kernel: nvme nvme0: missing or invalid SUBNQN field.
     [    3.033657] kernel: nvme nvme0: Shutdown timeout set to 10 seconds
     [    3.040527] kernel: nvme nvme0: 8/0/0 default/read/poll queues
     [    3.050395] kernel:  nvme0n1: p1 p2 p3
     [    3.146603] kernel: usb 3-10: new full-speed USB device number 6 using xhci_hcd
     [    3.275823] kernel: usb 3-10: New USB device found, idVendor=8087, idProduct=0032, bcdDevice= 0.00
     [    3.275833] kernel: usb 3-10: New USB device strings: Mfr=0, Product=0, SerialNumber=0
     [    3.740588] kernel: usb 3-6.3.3: new high-speed USB device number 7 using xhci_hcd
     [    3.897709] kernel: usb 3-6.3.3: New USB device found, idVendor=17ef, idProduct=a395, bcdDevice=60.90
     [    3.897718] kernel: usb 3-6.3.3: New USB device strings: Mfr=10, Product=11, SerialNumber=0
     [    3.897722] kernel: usb 3-6.3.3: Product: USB2.0 Hub
     [    3.897724] kernel: usb 3-6.3.3: Manufacturer: Lenovo
     [    4.175509] kernel: usb 3-6.3.3.1: new full-speed USB device number 8 using xhci_hcd
     [    4.255335] kernel: usb 3-6.3.3.1: New USB device found, idVendor=04b4, idProduct=521a, bcdDevice= 0.00
     [    4.255344] kernel: usb 3-6.3.3.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
     [    4.255347] kernel: usb 3-6.3.3.1: Product: USB-I2C Bridge
     [    4.255349] kernel: usb 3-6.3.3.1: Manufacturer: Cypress Semiconductor
     [    4.329486] kernel: usb 3-6.3.3.2: new full-speed USB device number 9 using xhci_hcd
     [    4.426308] kernel: usb 3-6.3.3.2: New USB device found, idVendor=17ef, idProduct=30d1, bcdDevice= 0.41
     [    4.426312] kernel: usb 3-6.3.3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
     [    4.426313] kernel: usb 3-6.3.3.2: Product: ThinkPad USB-C Dock Gen2 USB Audio
     [    4.426314] kernel: usb 3-6.3.3.2: Manufacturer: Lenovo
     [    4.426315] kernel: usb 3-6.3.3.2: SerialNumber: 000000000000
     [    4.436577] kernel: hid-generic 0003:17EF:30D1.0004: hiddev96,hidraw2: USB HID v1.11 Device [Lenovo ThinkPad USB-C Dock Gen2 USB Audio] on usb-0000:00:14.0-6.3.3.2/input3
     [    4.502646] kernel: usb 3-6.3.3.3: new high-speed USB device number 10 using xhci_hcd
     [    4.600830] kernel: usb 3-6.3.3.3: New USB device found, idVendor=0bda, idProduct=5411, bcdDevice= 1.17
     [    4.600842] kernel: usb 3-6.3.3.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
     [    4.600846] kernel: usb 3-6.3.3.3: Product: 4-Port USB 2.0 Hub
     [    4.600849] kernel: usb 3-6.3.3.3: Manufacturer: Generic
     [    4.672567] kernel: usb 3-6.3.3.4: new low-speed USB device number 11 using xhci_hcd
     [    4.764719] kernel: usb 3-6.3.3.4: New USB device found, idVendor=046a, idProduct=0023, bcdDevice= 2.20
     [    4.764731] kernel: usb 3-6.3.3.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0
     [    4.847170] kernel: cherry 0003:046A:0023.0005: input,hidraw3: USB HID v1.11 Keyboard [HID 046a:0023] on usb-0000:00:14.0-6.3.3.4/input0
     [    4.890590] kernel: usb 3-6.3.3.3.1: new high-speed USB device number 12 using xhci_hcd
     [    4.899734] kernel: cherry 0003:046A:0023.0006: input,hidraw4: USB HID v1.11 Device [HID 046a:0023] on usb-0000:00:14.0-6.3.3.4/input1
     [    5.000402] kernel: usb 3-6.3.3.3.1: New USB device found, idVendor=0bda, idProduct=5411, bcdDevice= 1.17
     [    5.000412] kernel: usb 3-6.3.3.3.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
     [    5.000415] kernel: usb 3-6.3.3.3.1: Product: 4-Port USB 2.0 Hub
     [    5.000417] kernel: usb 3-6.3.3.3.1: Manufacturer: Generic
     [    5.086639] kernel: usb 3-6.3.3.3.4: new full-speed USB device number 13 using xhci_hcd
     [    5.186279] kernel: usb 3-6.3.3.3.4: New USB device found, idVendor=0b0e, idProduct=030b, bcdDevice= 2.97
     [    5.186290] kernel: usb 3-6.3.3.3.4: New USB device strings: Mfr=0, Product=2, SerialNumber=3
     [    5.186293] kernel: usb 3-6.3.3.3.4: Product: Jabra Evolve 65
     [    5.186296] kernel: usb 3-6.3.3.3.4: SerialNumber: 50C27509313F
     [    5.257304] kernel: jabra 0003:0B0E:030B.0007: input,hiddev97,hidraw5: USB HID v1.11 Device [Jabra Evolve 65] on usb-0000:00:14.0-6.3.3.3.4/input3
     [   10.270683] kernel: BTRFS: device label fedora_localhost-live devid 1 transid 690013 /dev/dm-0 scanned by systemd-udevd (816)
     [   11.027995] kernel: BTRFS info (device dm-0): using crc32c (crc32c-intel) checksum algorithm
     [   11.028009] kernel: BTRFS info (device dm-0): using free space tree
     [   11.343209] kernel: BTRFS info (device dm-0): bdev /dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
     [   15.609101] kernel: BTRFS info (device dm-0): enabling ssd optimizations
     [   21.083603] kernel: BTRFS info (device dm-0: state M): use zstd compression, level 1
     [   23.094382] kernel: BTRFS info: devid 1 device path /dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 changed to /dev/dm-0 scanned by (udev-worker) (1029)
     [   23.096477] kernel: BTRFS info: devid 1 device path /dev/dm-0 changed to /dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 scanned by (udev-worker) (1029)
     [   27.214059] kernel: usbcore: registered new interface driver btusb
     [   27.230166] kernel: usb 3-7: Found UVC 1.00 device Laptop Camera (0bda:5634)
     [   27.235136] kernel: usbcore: registered new interface driver uvcvideo
     [   27.810497] kernel: usb 3-6.3.3.3.4: 1:1: cannot set freq 48000 to ep 0x3
     [   27.851573] kernel: usbcore: registered new interface driver snd-usb-audio
     [   57.822884] kernel: usb 3-9: reset full-speed USB device number 4 using xhci_hcd
     [   60.061876] kernel: usb 3-9: reset full-speed USB device number 4 using xhci_hcd
     [   77.105690] kernel: hid-generic 0003:17EF:30D1.0009: hiddev96,hidraw2: USB HID v1.11 Device [Lenovo ThinkPad USB-C Dock Gen2 USB Audio] on usb-0000:00:14.0-6.3.3.2/input3
     [  640.900767] kernel: usb 3-9: reset full-speed USB device number 4 using xhci_hcd
     [  641.157738] kernel: usb 3-9: reset full-speed USB device number 4 using xhci_hcd
     [  698.116128] kernel: usb 3-9: reset full-speed USB device number 4 using xhci_hcd
     [  698.363100] kernel: usb 3-9: reset full-speed USB device number 4 using xhci_hcd
     
* btrfs usage (btrfs filesystem usage -T /):
     Overall:
         Device size:		   1.82TiB
         Device allocated:		   1.82TiB
         Device unallocated:		  13.00MiB
         Device missing:		     0.00B
         Device slack:		     0.00B
         Used:			   1.76TiB
         Free (estimated):		  49.72GiB	(min: 49.72GiB)
         Free (statfs, df):		  49.73GiB
         Data ratio:			      1.00
         Metadata ratio:		      2.00
         Global reserve:		 512.00MiB	(used: 0.00B)
         Multiple profiles:		        no
     
                                                              Data    Metadata System                             
     Id Path                                                  single  DUP      DUP       Unallocated Total   Slack
     -- ----------------------------------------------------- ------- -------- --------- ----------- ------- -----
      1 /dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 1.80TiB 22.00GiB  16.00MiB    13.00MiB 1.82TiB     -
     -- ----------------------------------------------------- ------- -------- --------- ----------- ------- -----
        Total                                                 1.80TiB 11.00GiB   8.00MiB    13.00MiB 1.82TiB 0.00B
        Used                                                  1.75TiB  7.13GiB 224.00KiB                          
     
* FSUUID (findmnt -n -o UUID $(stat -c '%m' "/")):
     38073a1e-e0c4-46e7-9f5e-d7175b820147
     
* btrfs allocations (FSUUID=$(findmnt -n -o UUID $(stat -c '%m' "/")); grep -R . /sys/fs/btrfs/"$FSUUID"/allocation/ | sed "s/^.*allocation//"):
     /metadata/disk_used:15317434368
     /metadata/bytes_pinned:2375680
     /metadata/chunk_size:1073741824
     /metadata/bytes_used:7658717184
     /metadata/bg_reclaim_threshold:0
     /metadata/dup/used_bytes:7658717184
     /metadata/dup/total_bytes:11811160064
     /metadata/disk_total:23622320128
     /metadata/total_bytes:11811160064
     /metadata/bytes_reserved:11288576
     /metadata/bytes_readonly:65536
     /metadata/bytes_zone_unusable:0
     /metadata/bytes_may_use:1841414144
     /metadata/flags:4
     /system/disk_used:458752
     /system/bytes_pinned:0
     /system/chunk_size:33554432
     /system/bytes_used:229376
     /system/bg_reclaim_threshold:0
     /system/dup/used_bytes:229376
     /system/dup/total_bytes:8388608
     /system/disk_total:16777216
     /system/total_bytes:8388608
     /system/bytes_reserved:0
     /system/bytes_readonly:0
     /system/bytes_zone_unusable:0
     /system/bytes_may_use:0
     /system/flags:2
     /global_rsv_reserved:536870912
     /data/disk_used:1921636462592
     /data/bytes_pinned:0
     /data/chunk_size:10737418240
     /data/bytes_used:1921636462592
     /data/bg_reclaim_threshold:0
     /data/single/used_bytes:1921636462592
     /data/single/total_bytes:1975025401856
     /data/disk_total:1975025401856
     /data/total_bytes:1975025401856
     /data/bytes_reserved:19243008
     /data/bytes_readonly:65536
     /data/bytes_zone_unusable:0
     /data/bytes_may_use:3211264
     /data/flags:1
     /global_rsv_size:536870912
     
* btrfs features (FSUUID=$(findmnt -n -o UUID $(stat -c '%m' "/")); grep -R . /sys/fs/btrfs/"$FSUUID"/features/ | sed "s/^.*features//"):
     /free_space_tree:1
     /no_holes:1
     /compress_zstd:1
     /skinny_metadata:1
     /extended_iref:1
     
* btrfs checksum (FSUUID=$(findmnt -n -o UUID $(stat -c '%m' "/")); cat /sys/fs/btrfs/"$FSUUID"/checksum):
     crc32c (crc32c-intel)
     

Gathering system info ..........johannes|~ ▷

Here’s a raw dump of my console when I tried some of the hints from the threads @computersavvy mentioned:

johannes|~ ▷ sudo btrfs balance start -dusage=30 /
Done, had to relocate 0 out of 1853 chunks
johannes|~ ▷ dmesg |grep -i btrfs
[    1.512942] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity=yes
[   10.270683] BTRFS: device label fedora_localhost-live devid 1 transid 690013 /dev/dm-0 scanned by systemd-udevd (816)
[   11.027995] BTRFS info (device dm-0): using crc32c (crc32c-intel) checksum algorithm
[   11.028009] BTRFS info (device dm-0): using free space tree
[   11.343209] BTRFS info (device dm-0): bdev /dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[   15.609101] BTRFS info (device dm-0): enabling ssd optimizations
[   21.083603] BTRFS info (device dm-0: state M): use zstd compression, level 1
[   23.094382] BTRFS info: devid 1 device path /dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 changed to /dev/dm-0 scanned by (udev-worker) (1029)
[   23.096477] BTRFS info: devid 1 device path /dev/dm-0 changed to /dev/mapper/luks-e19affbf-1032-448b-9f84-7c877cbfb2a4 scanned by (udev-worker) (1029)
[ 1059.245703] BTRFS info (device dm-0): balance: start -dusage=30
[ 1059.247459] BTRFS info (device dm-0): balance: ended with status: 0
johannes|~ ▷ btrfs balance start -dusage=0
btrfs balance start: exactly 1 argument expected, 0 given
johannes|~ ▷ sudo btrfs balance start -dusage=0 /
Done, had to relocate 0 out of 1853 chunks
johannes|~ ▷ sudo btrfs balance start -dusage=1 /
Done, had to relocate 0 out of 1853 chunks
johannes|~ ▷ sudo btrfs balance start -dusage=5 /
Done, had to relocate 0 out of 1853 chunks
johannes|~ ▷ sudo btrfs balance start -dusage=30 /
Done, had to relocate 0 out of 1853 chunks
johannes|~ ▷ sudo btrfs scrub status /
UUID:             38073a1e-e0c4-46e7-9f5e-d7175b820147
	no stats available
Total to scrub:   1.76TiB
Rate:             0.00B/s
Error summary:    no errors found
johannes|~ ▷ sudo btrfs scrub start /
scrub started on /, fsid 38073a1e-e0c4-46e7-9f5e-d7175b820147 (pid=10450)

Now anxiously waiting for scrub to finish. It’s been running for about 30 minutes now and nothing changed yet (despite high IO activity).

6

It looks like both your / and /home are snapshotted under .veeam_snapshots/... subvolumes, which means nothing you remove from your disk will get freed, because it’s still part of that snapshot. You have to delete the snapshot as well. But I don’t want to give any particular advice, because I have no idea what “veeam” is and what its purpose is.

3 Likes
7

Hats off to you Sir, you nailed it! :sunglasses: :100:
I fiddled with btrfs but eventually gave up because it wouldn’t let me delete these snapshots easily. btrfs-assistant did the trick though. The free disk space is now ramping up slowly, probably because the scrub is still underway for some hours.
Veeam is a backup software that I briefly tested and found it’s not what I want. I had no idea it would leave this mess behind. Maybe they are employing snapshots to have a consistent starting point for the backup but in my opinion a shortcut of this type should not live longer than the backup operation lasts. Whatever.
So I think we can consider this solved. I cannot thank you enough, you saved at least my week!

Cheers,
Joe

8

vmlinuz-6.1.12-200.fc37.x86_64

Unrelated but I recommend updating the kernel, this is quite old by Fedora standards. Current Fedora kernel is 6.5 series for both Fedora 37 and 38.

There have been many thousands of fixes and feature enhancements even in btrfs alone in this time frame, let alone all the other areas across the kernel.

9

I agree with Chris.
Not only is that very old for F37, but F37 itself is due to be EOL in about a month.
I would recommend an upgrade to at least F38 and potentially F39 as soon as the final is released.

It is recommended that you upgrade to the latest within the release version you are using before upgrading to a new version so that should be a priority for you soon.

10

Thanks again for all the advice, people. I’m really grateful that so much valuable information is shared here.

Well, it’s Fedora 38 Workstation actually. No idea what indicates it is older. But the kernel being on an outdated version is actually deliberate because newer kernels prevented VMWare from compiling its kernel modules so the entire software wouldn’t work anymore.
I am too little an expert to troubleshoot this and get it right so I decided to blacklist the kernel in dnf’s configuration and stay on the version mentioned. I might switch to the latest kernel to try and then switch back if it still doesn’t work but it’s always a bit intimidating.
VMWare is a vital part of my daily job so I cannot just do without it. There may be better alternatives such as Boxes but the images I already have will probably not just transfer to that new environment without any losses. I had a brief look at Boxes but compared to VMWare I couldn’t find some features, particularly transparent USB device routing from host to guest, attaching ISO images as DVD drives, etc. - it may all be there but I missed it at first glance.
But I digress. I’ll leave a note in my brain to take care of the kernel version.
Thanks a ton!

Cheers,
Joe

11

Without knowing exactly what your needs are it is difficult to be sure, but libvirt in fedora does the above seamlessly and should be able to get you away from the proprietary vmware lock-in.

12

Virt-Manager has even more options to control KVM/QEMU VMs, but that’s not VMWare… I think I saw somewhere that the conversion between VMWare and QEMU is possible, but never used it myself.