Full btrfs partitions == Killed Fedora 42 ? To the rescue of the system

Ask Fedora
, ,
1

Hello everyone.
This is my first topic on this forum, so please bear with me.

For context: some time ago I set up snapshots on my system using btrfs-assistant and Snapper. I configured the snapshot schedule as 3 hourly, 3 daily, 3 weekly, and the rest at 10. Because I work with large files, I didn’t notice that over time my entire 1 TB disk filled up. At some point while working (I assume when a snapshot was being created) Fedora froze, and the only option was a hard reset. After that it wouldn’t boot. It starts to boot, but gets stuck in an infinite loop when trying to start systemd-tmpfiles-setup.service/start. My suspicion fell on full metadata on the Btrfs partition. I booted a live system and indeed it turned out that virtually all allocated space was in use.

liveuser@localhost-live:~$ sudo btrfs filesystem usage /run/media/liveuser/fedora 
Overall:
    Device size:  952.00GiB
    Device allocated:  951.99GiB
    Device unallocated:    4.00MiB
    Device missing:      0.00B
    Device slack:      0.00B
    Used:  947.65GiB
    Free (estimated):  460.00KiB (min: 460.00KiB)
    Free (statfs, df):  460.00KiB
    Data ratio:       1.00
    Metadata ratio:       2.00
    Global reserve:  512.00MiB (used: 0.00B)
    Multiple profiles:         no

Data,single: Size:887.98GiB, Used:887.97GiB (100.00%)
   /dev/nvme1n1p6  375.61GiB
   /dev/nvme1n1p7    5.49GiB
   /dev/nvme1n1p8  264.67GiB
   /dev/nvme1n1p3  242.20GiB

Metadata,DUP: Size:32.00GiB, Used:29.84GiB (93.25%)
   /dev/nvme1n1p6   14.00GiB
   /dev/nvme1n1p8   12.00GiB
   /dev/nvme1n1p3   38.00GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/nvme1n1p6   16.00MiB

Unallocated:
   /dev/nvme1n1p6    1.00MiB
   /dev/nvme1n1p7    1.00MiB
   /dev/nvme1n1p8    1.00MiB
   /dev/nvme1n1p3    1.00MiB

So I tried metadata balancing, clearing, disabling quotas, and a bunch of other things, but unfortunately whenever a write operation is involved, the process enters state D (uninterruptible sleep) and never recovers—while filling up RAM and swap completely and forcing me to hard-reset the live system.

I tried many things—ddrescue, dd, rsync—to save the data and the system by copying it to a second 2 TB disk, but either a memory overflow/out-of-memory occurred, or I couldn’t extend the GPT, and I still couldn’t get the system to boot.

Is there any hope for me? If there’s any chance at all, I’d be grateful for help.

P.S. I tried running sudo btrfs check --readonly /dev/nvme1n1p6, but at the final stage the process gets killed automatically and I don’t get any useful information.

2

You should never balance the metadata, as it could make the situation worse. Read here.

In a situation where the filesystem is full (ENOSPC), making it impossible to delete files or snapshots, you need to add more space to fix it. You can do this by adding another Btrfs disk or partition to the system, giving you enough room to delete files or snapshots. Afterwards, you can remove the added device, whether it’s a USB drive, a partition, etc.
Read here for more details.

3 Likes
3

Thx for your reply. I have tried to add another device to the Btrfs filesystem to extend it and make some space for the ability to delete some files, but btrfs device add process goes to D state in the process list and exhausts RAM and only a hard reset helps.

4

Can you show the command you ran? The logs help understand what’s going on.

5

Yeah, sure.

Here are connected disks. I have attached 120GB SSD to the adapter and it is seen as sda1

root@localhost-live:~# lsblk -o NAME,SIZE,FSTYPE,LABEL,MOUNTPOINT,MODEL
NAME          SIZE FSTYPE  LABEL             MOUNTPOINT          MODEL
loop0           2G erofs                     /run/rootfsbase     
sda         111.8G                                               RTL9210B
└─sda1      111.8G                                               
sdb         119.5G iso9660 Fedora-WS-Live-42                     Flash Drive
├─sdb1        2.2G iso9660 Fedora-WS-Live-42 /run/initramfs/live 
└─sdb2         30M vfat    BOOT                                  
zram0           8G swap    zram0             [SWAP]              
nvme0n1     953.9G                                               PC611 NVMe SK hynix 1TB
├─nvme0n1p1   100M vfat                                          
├─nvme0n1p2    16M                                               
├─nvme0n1p3 280.2G btrfs   fedora                                
├─nvme0n1p4   778M ext4                                          
├─nvme0n1p5     1G ext4                                          
├─nvme0n1p6 389.6G btrfs   fedora                                
├─nvme0n1p7   5.5G btrfs   fedora                                
└─nvme0n1p8 276.7G btrfs   fedora

Next mounted target system to /mnt and running btrfs device add command which return nothing else.

oot@localhost-live:~# btrfs device add -f /dev/sda1 /mnt
Performing full device TRIM /dev/sda1 (111.79GiB) ...

Also there is return from top. As you can see preocess is in permament D state and uses more and more RAM.

root@localhost-live:~# cat top.txt | grep btrfs
   4584 root      20   0       0      0      0 R  90.9   0.0   1:15.19 btrfs-c+
   4585 root      20   0       0      0      0 D   0.0   0.0   0:00.00 btrfs-t+
   4598 root      20   0    9428   6024   3768 D   0.0   0.0   0:00.14 btrfs
   4584 root      20   0       0      0      0 R  99.3   0.0   1:18.18 btrfs-c+
   4585 root      20   0       0      0      0 D   0.0   0.0   0:00.00 btrfs-t+
   4598 root      20   0    9428   6024   3768 D   0.0   0.0   0:00.14 btrfs
   4584 root      20   0       0      0      0 R  98.7   0.0   1:21.15 btrfs-c+
   4585 root      20   0       0      0      0 D   0.0   0.0   0:00.00 btrfs-t+
   4598 root      20   0    9428   6024   3768 D   0.0   0.0   0:00.14 btrfs
   4584 root      20   0       0      0      0 R  98.0   0.0   1:24.10 btrfs-c+
   4585 root      20   0       0      0      0 D   0.0   0.0   0:00.00 btrfs-t+
   4598 root      20   0    9428   6024   3768 D   0.0   0.0   0:00.14 btrfs
6

So I assume everything went fine. You can check it with this command:

sudo btrfs filesystem usage -T /mnt

Maybe it’s running an interrupted balance? Unfortunately, the process list is truncated, so it’s hard to see what’s actually running.

You should be able to see it with this command:

sudo btrfs balance status /mnt

7

I used to be able to fix “no unallocated space” problem with temporary disk added and do data balance, but it felt that the problem of yours isn’t the same. It seems your ssd is just filled with metadata (maybe caused by too many snapshots) and data.

I have read once that it is not recommended to have more than 12 snapshots per (sub)volume or the performance can degrade severely. It was a pretty old article so things might have improved now, but I still think “3 hourly, 3 daily, 3 weekly” is too much. After your device recovered, maybe consider taking less snapshots? I usually kept it under 5 (all snapshots included), YMMV.

Also why did you created 4 partitions on one disk only to add all of them to a btrfs volume? btrfs has a volume manager so you don’t have to create one partition for /home and another for / (for example), just create one partition and many subvolumes on them for different mount points if you needed it.

8

The issue occurs when there are many write and delete operations involving large files. For example, if you delete a 5 GB file, that file is removed from the current subvolume but remains in a snapshot. As a result, the file still exists within the filesystem and continues to occupy 5 GB of space. Multiply this by many files and numerous hourly snapshots, and the disk usage can quickly grow.

If we take openSUSE as an example, it creates a large number of snapshots, one for every zypper or YaST action but their rootfs subvolume is very minimal, so the amount of space occupied over time between older snapshots is minimal.

1 Like
9

After rebooting system no changes.

root@localhost-live:~# mount -o ro,subvolid=5 /dev/nvme0n1p6 /mnt
root@localhost-live:~# ls
root@localhost-live:~# btrfs filesystem usage -T /mnt

Overall:
    Device size:  952.00GiB
    Device allocated:  951.99GiB
    Device unallocated:    4.00MiB
    Device missing:      0.00B
    Device slack:      0.00B
    Used:  947.65GiB
    Free (estimated):  456.00KiB (min: 456.00KiB)
    Free (statfs, df):  456.00KiB
    Data ratio:       1.00
    Metadata ratio:       2.00
    Global reserve:  512.00MiB (used: 0.00B)
    Multiple profiles:         no

                  Data      Metadata System                               
Id Path           single    DUP      DUP       Unallocated Total     Slack
-- -------------- --------- -------- --------- ----------- --------- -----
 1 /dev/nvme0n1p6 375.61GiB 14.00GiB  16.00MiB     1.00MiB 389.62GiB     -
 2 /dev/nvme0n1p7   5.49GiB        -         -     1.00MiB   5.49GiB     -
 3 /dev/nvme0n1p8 264.67GiB 12.00GiB         -     1.00MiB 276.67GiB     -
 4 /dev/nvme0n1p3 242.20GiB 38.00GiB         -     1.00MiB 280.20GiB     -
-- -------------- --------- -------- --------- ----------- --------- -----
   Total          887.98GiB 32.00GiB   8.00MiB     4.00MiB 952.00GiB 0.00B
   Used           887.97GiB 29.84GiB 128.00KiB                            
root@localhost-live:~# 
root@localhost-live:~# sudo btrfs balance status /mnt
No balance found on '/mnt'
10

sda1 appears to be an unformatted partition. To add it to the filesystem, it must be formatted as Btrfs.

One recommendation I have for you is this: once you’ve added “sda1” to the filesystem, make sure not to remove it physically. When you’re done and the other partitions have enough unallocated space, you can remove it with the command sudo btrfs device remove <device-or-devid> <mountpoint> and then disconnect it physically.

I strongly recommend making a backup before proceeding.

1 Like
11

To all: definitely please please please be very cautious about recommending adding a 2nd device to get out of disk full situations on btrfs. It’s not wrong but most folks do not appreciate the consequences of physically removing the 2nd device prior to running the btrfs device remove command. It will almost surely break the file system beyond repair.

OP, I suggest joining the discussion on https://matrix.to/#/#btrfs:fedoraproject.org where interactive discussion will work better for cases like this. Often there’s so much back and forth needed to fully understand the situation that it could take days of back and forth in the forum by comparison.

1 Like
12

Sorry, you’re right I should have specified and emphasized that from the start when I suggested it (even though it’s clearly highlighted in Forza’s blog, with all the steps explained one by one).
Next time, I’ll make sure to point that out right from my first message.

2 Likes
13

btrfs device add will format the device being added, as well as resize (grow) the file system. All of this is very fast.

btrfs device remove must first migrate block groups off the device being removed to devices remaining in the btrfs, and then resize (shrink) the flie system, then the kernel removes the device from the btrfs and the magic is cleared. Block group migration can take seconds, minutes, hours, or days - it depends on how many block groups there are to migrate.

NOTE: The device remove command does not return to a prompt until the entire process is finished. While this process is subject to COW and should be crash safe - it’s super risky to physically remove any devices from such a btrfs - not least of which is that when the device was added, the btrfs metadata was not converted to raid1 and spread across multiple drive. It is possible for critical metadata to only exist on the recently added device, since the remaining device was full, and physical removal means that critical metadata is no longer avaliable and btrfs can’t write it anywhere. So it’s just gone and that can break the file system permanently.

Please join us on https://matrix.to/#/#btrfs:fedoraproject.org in particular for discussing complex issues like this, and also running out of space. Everyone is welcome to hang out.

Thanks.

1 Like
14

You’re right, I thought it only applied to a whole block device, for example “sda”.

15

This is the problem Number 1:

    Device size:  952.00GiB
    Device allocated:  951.99GiB

You cannot do any operation with the BTRFS partition that needs to allocate more space (and cannot) when you have this. This is what you must fix first. You should never get to this point. If you do get to this point and BTRFS stops working, you must solve this first.

I had this happen once when I was decreasing the physical partition to fit the BTRFS partition inside on it and then tried to balance the data on this partition when I had almost all space allocated. The only thing that helped was to increase the device size, which is what I think people have already advised to do earlier in the thread. This wasn’t the system partition, so I had no issues with the system not booting or whatnot. But I think the crucial part in situations like this is to increase the device size here any whichever way. If your partition is on a disk that has free space on it that’s not used by any partition, you could try to do btrfs resize max to claim that free space. I think that’s what I did to solve this.

1 Like
16

@chrissadpepe

Looks like a 4 device Btrfs, data profile single, metadata profile DUP, and only 1 MiB unallocated on each device. Metadata block groups are pretty full.

And I’m not sure what the status is after adding /dev/sda1 I don’t see it showing up in the btrfs fi us /mnt command results. So maybe it hasn’t been added?

I strongly advice keeping this additional device physically connected until we’re certain whether it has or has not been added to the Btrfs.

What kernel version is being used? Do you have another system available to download a newer Fedora ISO and image a USB stick to boot from? I’m not sure we’ll need this but I’d like to have in our back pocket the option to boot a newer kernel and btrfs-progs, and do this from a Live USB system rather than booting from this file system.

17

Thank you all so much for your replies—I really appreciate the effort. I only wish I’d had the knowledge you’ve shared earlier. Responding to your hypotheses:

I’m almost certain that because the filesystem filled the entire disk, even adding another device didn’t actually expand the Btrfs pool. I’m basing this on the output of btrfs device add captured with strace. The disk doesn’t respond to a low-level call (I’ll post detailed logs tomorrow). As a result, btrfs device add—and any other command that would have triggered a write to the Btrfs filesystem—never completed. That’s why, there were no command outputs to show. Tomorrow, once I’m back at the broken machine, I’ll answer all the questions about the specs.

What have I tried so far?

I attempted to 1:1 clone the data onto a 2 TB disk with dd and then extend partition p6 by a few GB to free up space. Unfortunately, no success so far, but I can take another shot at it tomorrow.

Right now I’ve used rsync to copy /home and / (root) to the new disk and prepared the partitions for it. I haven’t booted it yet because I’m still wrestling with GRUB and efibootmgr.

Thanks again for your willingness to help—I’ll post an update tomorrow.

18

My pc specs.

liveuser@localhost-live:~$ sudo fastfetch
             .',;::::;,'.                 root@localhost-live
         .';:cccccccccccc:;,.             -------------------
      .;cccccccccccccccccccccc;.          OS: Fedora Linux 42 (Workstation Edition) x86_64
    .:cccccccccccccccccccccccccc:.        Host: G3 3500
  .;ccccccccccccc;.:dddl:.;ccccccc;.      Kernel: Linux 6.14.0-63.fc42.x86_64
 .:ccccccccccccc;OWMKOOXMWd;ccccccc:.     Uptime: 2 hours, 22 mins
.:ccccccccccccc;KMMc;cc;xMMc;ccccccc:.    Packages: 1995 (rpm)
,cccccccccccccc;MMM.;cc;;WW:;cccccccc,    Shell: bash 5.2.37
:cccccccccccccc;MMM.;cccccccccccccccc:    Display (DP-3): 6144x3456 @ 60 Hz in 32" [External]
:ccccccc;oxOOOo;MMM000k.;cccccccccccc:    Display (HDMI-3): 6144x3456 @ 60 Hz in 32" [External]
cccccc;0MMKxdd:;MMMkddc.;cccccccccccc;    Display (eDP-1): 3840x2160 @ 144 Hz in 15" [Built-in]
ccccc;XMO';cccc;MMM.;cccccccccccccccc'    WM: Mutter (X11)
ccccc;MMo;ccccc;MMW.;ccccccccccccccc;     Cursor: Adwaita
ccccc;0MNc.ccc.xMMd;ccccccccccccccc;      Terminal: Ptyxis 48.1
cccccc;dNMWXXXWM0:;cccccccccccccc:,       Terminal Font: Adwaita Mono (11pt)
cccccccc;.:odl:.;cccccccccccccc:,.        CPU: Intel(R) Core(TM) i7-10750H (12) @ 5.00 GHz
ccccccccccccccccccccccccccccc:'.          GPU 1: NVIDIA GeForce GTX 1660 Ti Mobile [Discrete]
:ccccccccccccccccccccccc:;,..             GPU 2: Intel UHD Graphics @ 1.15 GHz [Integrated]
 ':cccccccccccccccc::;,.                  Memory: 24.97 GiB / 31.10 GiB (80%)
                                          Swap: 562.26 MiB / 8.00 GiB (7%)
                                          Disk (/): 331.48 MiB / 6.22 GiB (5%) - overlay
                                          Disk (/run/initramfs/live): 2.20 GiB / 2.20 GiB (100%) - iso9660 [External, Read-only]
                                          Disk (/run/media/liveuser/fedora): 948.15 GiB / 952.00 GiB (100%) - btrfs
                                          Local IP (enp60s0): 192.168.32.185/24
                                          Battery (DELL JJRRD06): 81% [AC Connected]
                                          Locale: en_US.UTF-8

What kernel version is being used? Do you have another system available to download a newer Fedora ISO and image a USB stick to boot from? I’m not sure we’ll need this but I’d like to have in our back pocket the option to boot a newer kernel and btrfs-progs, and do this from a Live USB system rather than booting from this file system.

System that you see above is live booted from usb because “corrupted” fs is not bootable anymore. Fedora just stops booting on systemd, creating tempfiles.

As I said in my previous reply, here is log from strace. Pay attention to the last line, where there is no response from the kernel on ioctl command.

root@localhost-live:~# strace btrfs device add -f /dev/sda /run/media/liveuser/fedora/
execve("/usr/sbin/btrfs", ["btrfs", "device", "add", "-f", "/dev/sda", "/run/media/liveuser/fedora/"], 0x7fff94583a68 /* 33 vars */) = 0
brk(NULL)                               = 0x564d928b7000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f17ead01000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=75663, ...}) = 0
mmap(NULL, 75663, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f17eacee000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=36776, ...}) = 0
mmap(NULL, 32832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17eace5000
mmap(0x7f17eacea000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f17eacea000
mmap(0x7f17eacec000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f17eacec000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libblkid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=242128, ...}) = 0
mmap(NULL, 234392, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17eacab000
mmap(0x7f17eacd1000, 53248, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f17eacd1000
mmap(0x7f17eacde000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x33000) = 0x7f17eacde000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libudev.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=295360, ...}) = 0
mmap(NULL, 289088, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17eac64000
mmap(0x7f17eac8d000, 98304, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x29000) = 0x7f17eac8d000
mmap(0x7f17eaca5000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x41000) = 0x7f17eaca5000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libgcrypt.so.20", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1540080, ...}) = 0
mmap(NULL, 1524176, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17eaaef000
mmap(0x7f17eac0d000, 315392, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11e000) = 0x7f17eac0d000
mmap(0x7f17eac5a000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16b000) = 0x7f17eac5a000
mmap(0x7f17eac63000, 464, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f17eac63000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=140848, ...}) = 0
mmap(NULL, 139392, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17eaacc000
mmap(0x7f17eaae3000, 40960, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f17eaae3000
mmap(0x7f17eaaed000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x20000) = 0x7f17eaaed000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/liblzo2.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=143416, ...}) = 0
mmap(NULL, 143368, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17eaaa8000
mmap(0x7f17eaac5000, 20480, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d000) = 0x7f17eaac5000
mmap(0x7f17eaaca000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7f17eaaca000
mmap(0x7f17eaacb000, 8, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f17eaacb000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libzstd.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=807528, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f17eaaa6000
mmap(NULL, 798760, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17ea9e2000
mmap(0x7f17eaa93000, 69632, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb1000) = 0x7f17eaa93000
mmap(0x7f17eaaa4000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc2000) = 0x7f17eaaa4000
mmap(0x7f17eaaa5000, 40, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f17eaaa5000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\2607\0\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
fstat(3, {st_mode=S_IFREG|0755, st_size=2452416, ...}) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 2043000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17ea7ef000
mmap(0x7f17ea95f000, 479232, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x170000) = 0x7f17ea95f000
mmap(0x7f17ea9d4000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e4000) = 0x7f17ea9d4000
mmap(0x7f17ea9da000, 31864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f17ea9da000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20V\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=44992, ...}) = 0
mmap(NULL, 45128, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17ea7e3000
mmap(0x7f17ea7e9000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f17ea7e9000
mmap(0x7f17ea7ed000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x7f17ea7ed000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=182352, ...}) = 0
mmap(NULL, 176584, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17ea7b7000
mmap(0x7f17ea7da000, 28672, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f17ea7da000
mmap(0x7f17ea7e1000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2a000) = 0x7f17ea7e1000
mmap(0x7f17ea7e2000, 456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f17ea7e2000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libgpg-error.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=170448, ...}) = 0
mmap(NULL, 164248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f17ea78e000
mmap(0x7f17ea7a8000, 53248, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x7f17ea7a8000
mmap(0x7f17ea7b5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x27000) = 0x7f17ea7b5000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f17ea78c000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f17ea789000
arch_prctl(ARCH_SET_FS, 0x7f17ea789900) = 0
set_tid_address(0x7f17ea789bd0)         = 4047
set_robust_list(0x7f17ea789be0, 24)     = 0
rseq(0x7f17ea789680, 0x20, 0, 0x53053053) = 0
mprotect(0x7f17ea9d4000, 16384, PROT_READ) = 0
mprotect(0x7f17ea7b5000, 4096, PROT_READ) = 0
mprotect(0x7f17ea7e1000, 4096, PROT_READ) = 0
mprotect(0x7f17ea7ed000, 4096, PROT_READ) = 0
mprotect(0x7f17eaaa4000, 4096, PROT_READ) = 0
mprotect(0x7f17eaaca000, 4096, PROT_READ) = 0
mprotect(0x7f17eaaed000, 4096, PROT_READ) = 0
mprotect(0x7f17eac5a000, 20480, PROT_READ) = 0
mprotect(0x7f17eaca5000, 20480, PROT_READ) = 0
mprotect(0x7f17eacde000, 24576, PROT_READ) = 0
mprotect(0x7f17eacec000, 4096, PROT_READ) = 0
mprotect(0x564d6158b000, 32768, PROT_READ) = 0
mprotect(0x7f17ead3d000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7f17eacee000, 75663)           = 0
getrandom("\xce\xf2\x18\x4e\xb6\x5c\x51\x58", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x564d928b7000
brk(0x564d928d8000)                     = 0x564d928d8000
prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 1
prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1
prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument)
access("/etc/gcrypt/fips_enabled", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/proc/sys/crypto/fips_enabled", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(3, "0\n", 1024)                    = 2
close(3)                                = 0
newfstatat(AT_FDCWD, "/run/media/liveuser/fedora/", {st_mode=S_IFDIR|0755, st_size=114, ...}, 0) = 0
statfs("/run/media/liveuser/fedora/", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096, f_blocks=249559808, f_bfree=1007402, f_bavail=114, f_files=0, f_ffree=0, f_fsid={val=[0xe22f3c87, 0x27d918ff]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_RELATIME}) = 0
openat(AT_FDCWD, "/run/media/liveuser/fedora/", O_RDONLY) = 3
ioctl(3, BTRFS_IOC_FS_INFO, {max_id=4, num_devices=4, fsid=54688c57-4a77-4fce-b647-b0d06dae5734, nodesize=16384, sectorsize=4096, clone_alignment=4096, flags=0}) = 0
openat(AT_FDCWD, "/sys/fs/btrfs/54688c57-4a77-4fce-b647-b0d06dae5734/exclusive_operation", O_RDONLY) = 4
ioctl(3, BTRFS_IOC_FS_INFO, {max_id=4, num_devices=4, fsid=54688c57-4a77-4fce-b647-b0d06dae5734, nodesize=16384, sectorsize=4096, clone_alignment=4096, flags=0}) = 0
openat(AT_FDCWD, "/sys/fs/btrfs/54688c57-4a77-4fce-b647-b0d06dae5734/exclusive_operation", O_RDONLY) = 5
lseek(5, 0, SEEK_SET)                   = 0
read(5, "none\n", 32)                   = 5
close(5)                                = 0
close(4)                                = 0
ioctl(3, BTRFS_IOC_GET_FEATURES, {compat_flags=0, compat_ro_flags=BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE|BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID|BTRFS_FEATURE_COMPAT_RO_VERITY, incompat_flags=BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF|BTRFS_FEATURE_INCOMPAT_COMPRESS_ZSTD|BTRFS_FEATURE_INCOMPAT_BIG_METADATA|BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF|BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA|BTRFS_FEATURE_INCOMPAT_NO_HOLES}) = 0
newfstatat(AT_FDCWD, "/dev/sda", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}, 0) = 0
openat(AT_FDCWD, "/dev/sda", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 4
fadvise64(4, 0, 0, POSIX_FADV_RANDOM)   = 0
fstat(4, {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}) = 0
ioctl(4, BLKGETSIZE64, [120040980480])  = 0
ioctl(4, FDGETFDCSTAT, 0x7fffcb312f10)  = -1 EINVAL (Invalid argument)
openat(AT_FDCWD, "/sys/dev/block/8:0", O_RDONLY|O_CLOEXEC) = 5
openat(5, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(5)                                = 0
openat(AT_FDCWD, "/sys/dev/block/8:0", O_RDONLY|O_CLOEXEC) = 5
faccessat2(5, "partition", F_OK, 0)     = -1 ENOENT (No such file or directory)
openat(5, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(5)                                = 0
ioctl(4, CDROM_GET_CAPABILITY, 0)       = -1 EINVAL (Invalid argument)
ioctl(4, CDROM_LAST_WRITTEN, 0x7fffcb312e50) = -1 EINVAL (Invalid argument)
ioctl(4, BLKGETZONESZ, [0])             = 0
ioctl(4, BLKIOOPT, [33553920])          = 0
ioctl(4, BLKIOMIN, [512])               = 0
openat(AT_FDCWD, "/sys/dev/block/8:0", O_RDONLY|O_CLOEXEC) = 5
faccessat2(5, "partition", F_OK, 0)     = -1 ENOENT (No such file or directory)
openat(5, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
readlink("/sys/dev/block/8:0", "../../devices/pci0000:00/0000:00"..., 4095) = 129
close(5)                                = 0
close(4)                                = 0
openat(AT_FDCWD, "/sys/block/sda/queue/zoned", O_RDONLY) = 4
read(4, "none\n", 32)                   = 5
close(4)                                = 0
newfstatat(AT_FDCWD, "/dev/sda", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}, 0) = 0
openat(AT_FDCWD, "/proc/swaps", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(4, "Filename\t\t\t\tType\t\tSize\t\tUsed\t\tPr"..., 1024) = 105
newfstatat(AT_FDCWD, "/dev/zram0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xfb, 0), ...}, 0) = 0
read(4, "", 1024)                       = 0
close(4)                                = 0
openat(AT_FDCWD, "/dev/sda", O_RDONLY)  = 4
fstat(4, {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}) = 0
ioctl(4, BLKGETZONESZ, [0])             = 0
pread64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 65536) = 4096
openat(AT_FDCWD, "/proc/self/mounts", O_RDONLY|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(5, "LiveOS_rootfs / overlay rw,secla"..., 1024) = 1024
newfstatat(AT_FDCWD, "LiveOS_rootfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "rpc_pipefs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "devtmpfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "devpts", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "sysfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "securityfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "cgroup2", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "pstore", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "efivarfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "bpf", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "configfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "proc", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
read(5, "pfs /run tmpfs rw,seclabel,nosui"..., 1024) = 1024
newfstatat(AT_FDCWD, "tmpfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/dev/sdb1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0x11), ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/sda", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/sdb1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0x11), ...}, 0) = 0
readlink("/dev", 0x7fffcb312970, 1023)  = -1 EINVAL (Invalid argument)
readlink("/dev/sda", 0x7fffcb312970, 1023) = -1 EINVAL (Invalid argument)
readlink("/dev", 0x7fffcb312970, 1023)  = -1 EINVAL (Invalid argument)
readlink("/dev/sdb1", 0x7fffcb312970, 1023) = -1 EINVAL (Invalid argument)
newfstatat(AT_FDCWD, "/dev/sda", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/sdb1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0x11), ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/sda", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0
readlink("/dev", 0x7fffcb310840, 1023)  = -1 EINVAL (Invalid argument)
readlink("/dev/loop0", 0x7fffcb310840, 1023) = -1 EINVAL (Invalid argument)
openat(AT_FDCWD, "/sys/block//loop0/loop/backing_file", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
read(6, "/LiveOS/squashfs.img\n", 4096) = 21
close(6)                                = 0
readlink("/dev", 0x7fffcb312970, 1023)  = -1 EINVAL (Invalid argument)
readlink("/dev/sda", 0x7fffcb312970, 1023) = -1 EINVAL (Invalid argument)
readlink("/LiveOS", 0x7fffcb312970, 1023) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/dev/sda", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}, 0) = 0
newfstatat(AT_FDCWD, "/LiveOS/squashfs.img", 0x7fffcb312e50, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "selinuxfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "systemd-1", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tracefs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "debugfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "mqueue", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "hugetlbfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
read(5, "exec,relatime 0 0\ntmpfs /tmp tmp"..., 1024) = 843
newfstatat(AT_FDCWD, "fusectl", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "gvfsd-fuse", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "portal", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/dev/nvme0n1p6", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x103, 0x6), ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/sda", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/nvme0n1p6", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x103, 0x6), ...}, 0) = 0
readlink("/dev", 0x7fffcb312970, 1023)  = -1 EINVAL (Invalid argument)
readlink("/dev/sda", 0x7fffcb312970, 1023) = -1 EINVAL (Invalid argument)
readlink("/dev", 0x7fffcb312970, 1023)  = -1 EINVAL (Invalid argument)
readlink("/dev/nvme0n1p6", 0x7fffcb312970, 1023) = -1 EINVAL (Invalid argument)
newfstatat(AT_FDCWD, "/dev/sda", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}, 0) = 0
newfstatat(AT_FDCWD, "/dev/nvme0n1p6", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x103, 0x6), ...}, 0) = 0
newfstatat(AT_FDCWD, "tmpfs", 0x7fffcb316e70, 0) = -1 ENOENT (No such file or directory)
read(5, "", 1024)                       = 0
close(5)                                = 0
close(4)                                = 0
openat(AT_FDCWD, "/dev/sda", O_RDONLY|O_EXCL) = 4
fstat(4, {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}) = 0
close(4)                                = 0
openat(AT_FDCWD, "/dev/sda", O_RDWR)    = 4
fstat(4, {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}) = 0
ioctl(4, BLKGETSIZE64, [120040980480])  = 0
openat(AT_FDCWD, "/dev/sda", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 5
fadvise64(5, 0, 0, POSIX_FADV_RANDOM)   = 0
fstat(5, {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}) = 0
ioctl(5, BLKGETSIZE64, [120040980480])  = 0
ioctl(5, FDGETFDCSTAT, 0x7fffcb312e90)  = -1 EINVAL (Invalid argument)
openat(AT_FDCWD, "/sys/dev/block/8:0", O_RDONLY|O_CLOEXEC) = 6
openat(6, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(6)                                = 0
openat(AT_FDCWD, "/sys/dev/block/8:0", O_RDONLY|O_CLOEXEC) = 6
faccessat2(6, "partition", F_OK, 0)     = -1 ENOENT (No such file or directory)
openat(6, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(6)                                = 0
ioctl(5, CDROM_GET_CAPABILITY, 0)       = -1 EINVAL (Invalid argument)
ioctl(5, CDROM_LAST_WRITTEN, 0x7fffcb312dd0) = -1 EINVAL (Invalid argument)
ioctl(5, BLKGETZONESZ, [0])             = 0
ioctl(5, BLKIOOPT, [33553920])          = 0
ioctl(5, BLKIOMIN, [512])               = 0
openat(AT_FDCWD, "/sys/dev/block/8:0", O_RDONLY|O_CLOEXEC) = 6
faccessat2(6, "partition", F_OK, 0)     = -1 ENOENT (No such file or directory)
openat(6, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
readlink("/sys/dev/block/8:0", "../../devices/pci0000:00/0000:00"..., 4095) = 129
close(6)                                = 0
close(5)                                = 0
openat(AT_FDCWD, "/sys/block/sda/queue/discard_granularity", O_RDONLY) = 5
read(5, "512\n", 128)                   = 4
close(5)                                = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x1), ...}) = 0
write(1, "Performing full device TRIM /dev"..., 53Performing full device TRIM /dev/sda (111.80GiB) ...
) = 53
ioctl(4, BLKDISCARD, [0, 1073741824])   = 0
ioctl(4, BLKDISCARD, [1073741824, 1073741824]) = 0
ioctl(4, BLKDISCARD, [2147483648, 1073741824]) = 0
ioctl(4, BLKDISCARD, [3221225472, 1073741824]) = 0
ioctl(4, BLKDISCARD, [4294967296, 1073741824]) = 0
ioctl(4, BLKDISCARD, [5368709120, 1073741824]) = 0
ioctl(4, BLKDISCARD, [6442450944, 1073741824]) = 0
ioctl(4, BLKDISCARD, [7516192768, 1073741824]) = 0
ioctl(4, BLKDISCARD, [8589934592, 1073741824]) = 0
ioctl(4, BLKDISCARD, [9663676416, 1073741824]) = 0
ioctl(4, BLKDISCARD, [10737418240, 1073741824]) = 0
ioctl(4, BLKDISCARD, [11811160064, 1073741824]) = 0
ioctl(4, BLKDISCARD, [12884901888, 1073741824]) = 0
ioctl(4, BLKDISCARD, [13958643712, 1073741824]) = 0
ioctl(4, BLKDISCARD, [15032385536, 1073741824]) = 0
ioctl(4, BLKDISCARD, [16106127360, 1073741824]) = 0
ioctl(4, BLKDISCARD, [17179869184, 1073741824]) = 0
ioctl(4, BLKDISCARD, [18253611008, 1073741824]) = 0
ioctl(4, BLKDISCARD, [19327352832, 1073741824]) = 0
ioctl(4, BLKDISCARD, [20401094656, 1073741824]) = 0
ioctl(4, BLKDISCARD, [21474836480, 1073741824]) = 0
ioctl(4, BLKDISCARD, [22548578304, 1073741824]) = 0
ioctl(4, BLKDISCARD, [23622320128, 1073741824]) = 0
ioctl(4, BLKDISCARD, [24696061952, 1073741824]) = 0
ioctl(4, BLKDISCARD, [25769803776, 1073741824]) = 0
ioctl(4, BLKDISCARD, [26843545600, 1073741824]) = 0
ioctl(4, BLKDISCARD, [27917287424, 1073741824]) = 0
ioctl(4, BLKDISCARD, [28991029248, 1073741824]) = 0
ioctl(4, BLKDISCARD, [30064771072, 1073741824]) = 0
ioctl(4, BLKDISCARD, [31138512896, 1073741824]) = 0
ioctl(4, BLKDISCARD, [32212254720, 1073741824]) = 0
ioctl(4, BLKDISCARD, [33285996544, 1073741824]) = 0
ioctl(4, BLKDISCARD, [34359738368, 1073741824]) = 0
ioctl(4, BLKDISCARD, [35433480192, 1073741824]) = 0
ioctl(4, BLKDISCARD, [36507222016, 1073741824]) = 0
ioctl(4, BLKDISCARD, [37580963840, 1073741824]) = 0
ioctl(4, BLKDISCARD, [38654705664, 1073741824]) = 0
ioctl(4, BLKDISCARD, [39728447488, 1073741824]) = 0
ioctl(4, BLKDISCARD, [40802189312, 1073741824]) = 0
ioctl(4, BLKDISCARD, [41875931136, 1073741824]) = 0
ioctl(4, BLKDISCARD, [42949672960, 1073741824]) = 0
ioctl(4, BLKDISCARD, [44023414784, 1073741824]) = 0
ioctl(4, BLKDISCARD, [45097156608, 1073741824]) = 0
ioctl(4, BLKDISCARD, [46170898432, 1073741824]) = 0
ioctl(4, BLKDISCARD, [47244640256, 1073741824]) = 0
ioctl(4, BLKDISCARD, [48318382080, 1073741824]) = 0
ioctl(4, BLKDISCARD, [49392123904, 1073741824]) = 0
ioctl(4, BLKDISCARD, [50465865728, 1073741824]) = 0
ioctl(4, BLKDISCARD, [51539607552, 1073741824]) = 0
ioctl(4, BLKDISCARD, [52613349376, 1073741824]) = 0
ioctl(4, BLKDISCARD, [53687091200, 1073741824]) = 0
ioctl(4, BLKDISCARD, [54760833024, 1073741824]) = 0
ioctl(4, BLKDISCARD, [55834574848, 1073741824]) = 0
ioctl(4, BLKDISCARD, [56908316672, 1073741824]) = 0
ioctl(4, BLKDISCARD, [57982058496, 1073741824]) = 0
ioctl(4, BLKDISCARD, [59055800320, 1073741824]) = 0
ioctl(4, BLKDISCARD, [60129542144, 1073741824]) = 0
ioctl(4, BLKDISCARD, [61203283968, 1073741824]) = 0
ioctl(4, BLKDISCARD, [62277025792, 1073741824]) = 0
ioctl(4, BLKDISCARD, [63350767616, 1073741824]) = 0
ioctl(4, BLKDISCARD, [64424509440, 1073741824]) = 0
ioctl(4, BLKDISCARD, [65498251264, 1073741824]) = 0
ioctl(4, BLKDISCARD, [66571993088, 1073741824]) = 0
ioctl(4, BLKDISCARD, [67645734912, 1073741824]) = 0
ioctl(4, BLKDISCARD, [68719476736, 1073741824]) = 0
ioctl(4, BLKDISCARD, [69793218560, 1073741824]) = 0
ioctl(4, BLKDISCARD, [70866960384, 1073741824]) = 0
ioctl(4, BLKDISCARD, [71940702208, 1073741824]) = 0
ioctl(4, BLKDISCARD, [73014444032, 1073741824]) = 0
ioctl(4, BLKDISCARD, [74088185856, 1073741824]) = 0
ioctl(4, BLKDISCARD, [75161927680, 1073741824]) = 0
ioctl(4, BLKDISCARD, [76235669504, 1073741824]) = 0
ioctl(4, BLKDISCARD, [77309411328, 1073741824]) = 0
ioctl(4, BLKDISCARD, [78383153152, 1073741824]) = 0
ioctl(4, BLKDISCARD, [79456894976, 1073741824]) = 0
ioctl(4, BLKDISCARD, [80530636800, 1073741824]) = 0
ioctl(4, BLKDISCARD, [81604378624, 1073741824]) = 0
ioctl(4, BLKDISCARD, [82678120448, 1073741824]) = 0
ioctl(4, BLKDISCARD, [83751862272, 1073741824]) = 0
ioctl(4, BLKDISCARD, [84825604096, 1073741824]) = 0
ioctl(4, BLKDISCARD, [85899345920, 1073741824]) = 0
ioctl(4, BLKDISCARD, [86973087744, 1073741824]) = 0
ioctl(4, BLKDISCARD, [88046829568, 1073741824]) = 0
ioctl(4, BLKDISCARD, [89120571392, 1073741824]) = 0
ioctl(4, BLKDISCARD, [90194313216, 1073741824]) = 0
ioctl(4, BLKDISCARD, [91268055040, 1073741824]) = 0
ioctl(4, BLKDISCARD, [92341796864, 1073741824]) = 0
ioctl(4, BLKDISCARD, [93415538688, 1073741824]) = 0
ioctl(4, BLKDISCARD, [94489280512, 1073741824]) = 0
ioctl(4, BLKDISCARD, [95563022336, 1073741824]) = 0
ioctl(4, BLKDISCARD, [96636764160, 1073741824]) = 0
ioctl(4, BLKDISCARD, [97710505984, 1073741824]) = 0
ioctl(4, BLKDISCARD, [98784247808, 1073741824]) = 0
ioctl(4, BLKDISCARD, [99857989632, 1073741824]) = 0
ioctl(4, BLKDISCARD, [100931731456, 1073741824]) = 0
ioctl(4, BLKDISCARD, [102005473280, 1073741824]) = 0
ioctl(4, BLKDISCARD, [103079215104, 1073741824]) = 0
ioctl(4, BLKDISCARD, [104152956928, 1073741824]) = 0
ioctl(4, BLKDISCARD, [105226698752, 1073741824]) = 0
ioctl(4, BLKDISCARD, [106300440576, 1073741824]) = 0
ioctl(4, BLKDISCARD, [107374182400, 1073741824]) = 0
ioctl(4, BLKDISCARD, [108447924224, 1073741824]) = 0
ioctl(4, BLKDISCARD, [109521666048, 1073741824]) = 0
ioctl(4, BLKDISCARD, [110595407872, 1073741824]) = 0
ioctl(4, BLKDISCARD, [111669149696, 1073741824]) = 0
ioctl(4, BLKDISCARD, [112742891520, 1073741824]) = 0
ioctl(4, BLKDISCARD, [113816633344, 1073741824]) = 0
ioctl(4, BLKDISCARD, [114890375168, 1073741824]) = 0
ioctl(4, BLKDISCARD, [115964116992, 1073741824]) = 0
ioctl(4, BLKDISCARD, [117037858816, 1073741824]) = 0
ioctl(4, BLKDISCARD, [118111600640, 1073741824]) = 0
ioctl(4, BLKDISCARD, [119185342464, 855638016]) = 0
mmap(NULL, 2101248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f17ea588000
pwrite64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2097152, 0) = 2097152
munmap(0x7f17ea588000, 2101248)         = 0
pwrite64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 65536) = 4096
pwrite64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 67108864) = 4096
pwrite64(4, "", 0, 120040980480)        = 0
brk(0x564d92ad9000)                     = 0x564d92ad9000
pwrite64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2097152, 120038883328) = 2097152
fadvise64(4, 0, 0, POSIX_FADV_RANDOM)   = 0
fstat(4, {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0), ...}) = 0
ioctl(4, BLKGETSIZE64, [120040980480])  = 0
ioctl(4, FDGETFDCSTAT, 0x7fffcb312ed0)  = -1 EINVAL (Invalid argument)
openat(AT_FDCWD, "/sys/dev/block/8:0", O_RDONLY|O_CLOEXEC) = 5
openat(5, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(5)                                = 0
openat(AT_FDCWD, "/sys/dev/block/8:0", O_RDONLY|O_CLOEXEC) = 5
faccessat2(5, "partition", F_OK, 0)     = -1 ENOENT (No such file or directory)
openat(5, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(5)                                = 0
ioctl(4, CDROM_GET_CAPABILITY, 0)       = -1 EINVAL (Invalid argument)
ioctl(4, CDROM_LAST_WRITTEN, 0x7fffcb312e10) = -1 EINVAL (Invalid argument)
ioctl(4, BLKGETZONESZ, [0])             = 0
ioctl(4, BLKIOOPT, [33553920])          = 0
ioctl(4, BLKIOMIN, [512])               = 0
close(4)                                = 0
readlink("/dev", 0x7fffcb316c80, 1023)  = -1 EINVAL (Invalid argument)
readlink("/dev/sda", 0x7fffcb316c80, 1023) = -1 EINVAL (Invalid argument)
ioctl(3, BTRFS_IOC_ADD_DEV, {fd=0, name="/dev/sda"}
19

There’s an error in specifying the device — you’re using “/dev/sda” instead of the partition “/dev/sda1”.

I recommend that once you add the device “/dev/sda1” [1], Once you add the device “/dev/sda1”, do not remove it until you’ve freed up space on the other full partitions.. You can recover space by deleting files or snapshots. Make sure there’s unallocated space available on the other devices, and then you can remove “/dev/sda1” with the command sudo btrfs device remove /dev/sda1 /run/media/liveuser/fedora/ Wait for the removal process to finish, double-check sudo btrfs filesystem usage -T /run/media/liveuser/fedora/ that it’s no longer part of the filesystem, and only then remove it physically.

Do not remove the device physically before completing this process, as doing so may cause data loss or filesystem corruption.

[1]

EDIT

As an additional recommendation and precaution, given the instability of USB devices, we advise against using USB devices for this operation.

1 Like
20

I’m not sure why but I think your kernel became confused as a result of having a partitioned /dev/sda resulting in a /dev/sda1 existing, but you tried to force add /dev/sda.

I tested this with USB sticks, kernel 6.17.0, and btrfs-progs 6.17, and the btrfs dev add worked. As in, it wiped the partition map on sda, and then wrote supers on sda, and grew the file system.

Anyway, given the risk of adding USB sticks (they can get bumped and drop of the USB bus - and kaboom your filesystem is damaged) I really think it’s best if we have an interactive conversation on Matrix.

The things I’d like to check in order:

  1. mount the file system mount -o enospc_debug /dev/nvme0n1p6 /mnt
  2. does it mount read write? any kernel errors? maybe give it 5 minutes after mount - sometimes there’s a prior balance that gets kicked off or orphan cleanup and we want to see if that fails with an otherwise normal mount
  3. if it is failing to mount or eventually mounts read-only does mount -o skip_balance,enospc_debug /dev/nvme0n1p6 /mnt work any better
  4. Can you provide the output from: grep -R . /sys/fs/btrfs/$UUID/allocation/

$UUID should auto complete with tab key assuming only one btrfs on this system (no matter how many devices it’s on).

With a snapper setup, and I’m pretty unfamiliar how it organizes things, I’m not sure the snapper command is available with Live USB boot, and even if installed, I’m not sure it has access to its database for tracking the various snapshots. If you start deleting snapshots, I’m not sure how confused snapper will get. The file system will be fine, but there might be consequences to deleting one or more snapshots.

I’m wondering if there are any large files that could be deleted that are not snapshot? Or if most everything is now pinned in snapshots?

Note that Btrfs needs unused space to write to in order to delete files, due to COW. The space consumption increases on Btrfs initially. Later the extents are freed (a few minutes at most).

Eventually we need to get to a point of trying to balance one or two data block groups. This tool can help:

So that’s part of upstream btrfs-progs but we don’t include it in Fedora btrfs package, so you’ll need to grab it from github and set permissions. Usage is:

btrfs-debugfs -b /mnt

It’ll return nearly 1000 lines because you have about that many block gropus. If we’re lucky, there will be some data block groups that are barely full, and it’ll be possible to use a filtered balance to get at least 1 GiB unallocated. Ideally 2-3. Then it’s possible to reboot normally, and get snapper/btrfs-assistant to remove some snapshots, further freeing up more space, and also check the configuration to make sure it will remove snapshots well before the file system gets full again.

I’m not aware of any issues related to this in kernel 6.14 but it really should not hang. Therefore I’ll low priority suggest an ISO with a newer kernel in the off chance there’s an improvement.

These are the current test candidates of Fedora 43 ISOs and images. Newer than the beta. They all have kernel 6.17.0 so just pick your preferred DE.

2 Likes