Home dir gone

Ive been using F41 for a while, I have a home and boot subvolume. Early on, I setup snapper on boot and home, and btrfs assistant. And yesterday my 500 gb home was full despite having only 150 gb, I realized that snapper was taking up the rest and used btrfs assistant to remove all the home stuff. This recovered all the disk space and everything was fine.

But on next restart I got this loop:
You are in emergency mode. After logging in, type 'journalctl -xb' to view system logs, 'systemctl reboot' to reboot, 'systemctl default' or 'exit' to boot into default mode. Cannot open access to console, the root account is locked. See sulogin(8) man page for more details

I have read some potential avenues to fix this but my case, my guess is, it’s related to the home changes I just did, so chroot into my install from the live disk and doing those may not be the best thing to try next.

When I boot into a f41 live usb, I see my home volume, but when I mount it, it’s empty. (boot seems fine).

Would something on restart have kicked this off, any ideas for best ways to recover?

If any fstab mount points are missing, I think you run into this error.

I’m not familiar with snapper so I don’t know if it’s modifying /etc/fstab or if the clean up could have deleted the subvolume assigned to the /home mountpoint.

You could use a Live USB stick to recover. I suggest using -o ro,nodiscards mount option to limit the changes to the file system.

Another option you can modify+add boot parameters:

systemd.volatile=overlay fstab=no and the existing rootflags=subvol=root should become rootflags=subvol=root,nodiscards

This provides a LiveOS-like environment, all writes are directed to memory, they’re not persistent.

Note, since fstab is ignored, your /home definitely will be empty. That doesn’t necessarily mean there’s data loss. You’ll need to dig through the list of subvolumes and piece together what remains.

I kinda like the USB stick approach because you can definitely avoid any further writes to the file system as a whole. And that improves the possibility the btrees for deleted subvolumes/snapshots are still present and can be recovered.

Thing is, if this is an SSD, any deleted data and metadata is soon subject to discards and the SSD itself will erase those btrees and data. If discards have already been issued, the recovery options are probably limited to independent backups.

And that leads to the “a snapshot is not a backup” observation. And also leads to thinking about Btrfs snapshots differently, for example in terms of using them for cheap replication to another Btrfs file system either locally or remotely. This is a form of backup, because of the replication to an independent file system, ostensibly on another device.

Automating this is the function of btrbk which is in Fedora repo. It’s different from snapper. It does manage it’s own snapshots as well as the replication policy.

Thanks Chris, I backed up an image, and went down something like what you suggested, but couldnt get it to mount with anything. I found btrfs rescue is showing errors, perhaps the metadata is corrupted in some way, as mounting the partition shows no data, I am trying to recover what i can through foremost now.

btrfs restore? Can you post the errors?

While unmounted, you can run btrfs check $DEV and it will check for file system inconsistencies. This is safe and read only, so long as --repair is not used. That’s a last resort.

Ordinarily btrfs will not mount if the file system is inconsistent. So it would seem there’s some snapper confusion and it ended up deleting more than you wanted. I’m not familiar with it so I’m not sure how that’s possible.

There can be hidden subvolumes and snapshots in snapper managed btrfs file systems. It might be worth checking:

# mount $DEV /mount
# btrfs sub list /mount

That will list all subvolumes (and subvolume snapshots).

You can navigate into these subvolumes and snapshots as if they were directories.

I had an external backup that was pretty recent, so I just restored from there. I mostly just lost some cyberpunk save progress…so I’ll survive. :slight_smile: Thanks for the help.