Hello, yesterday I was playing a game via steam (Escape the Backrooms via the latest proton-GE if it matters at all) when suddenly the game froze. I managed to press the meta key to bring up my taskbar but it was not responding when clicked and I then got a popup message that was something along the lines of not being able to write to plasmashell. My screen then went completely black and unresponsive. After hard restarting my pc via the power button, I tried to boot into my PC but after getting past GRUB and past the notice that my GPU overdrive (9070 XT) is on that shows every time I boot into KDE I am met with a kernel panic every time with the following error:
KERNEL PANIC! Please reboot your computer. Attempted to kill init! exitcode=0x00007f00
I have had Fedora root snapshots set up since I installed the OS at F41 but after trying the three past kernels installed (and the rescue kernel from install) and snapshots dating all the way back to march, I am met with the exact same error. What strikes me the most is that I wasn’t even messing with the OS recently or doing anything that would have triggered the system breaking so badly (apart from maybe switching to the VR power profile via LACT to play VR for a few hours the same day? I switched it back to automatic after but I guess its worth noting my GPU does have a small overclock). My PC was updated to the latest (or close to) F43 version at the time (not F44) so it had plasma 6.6.4 and kernel version 7.0.4 installed.
Just wondering how I can fix this cause trying other kernels or using a past root snapshots did not work. Thank you
Well, I guess that confirms that the problem isn’t your hardware. The first thing you should do before getting too far along is to backup anything you don’t want to risk losing.
You should be able to access the logs from the Live environment. It’s not something I’ve done, but it looks like if you can mount your system’s root filesystem somewhere (e.g. /mnt), then you should be able to view the logs from the system with a command something along the lines of:
journalctl -D /mnt/var/log/journal -b
See if you can find a more detailed error message about why the system is failing to boot. Assuming you have an internet connection in your Live environment, you should be able to upload some log entries using the fpaste command and share the link here (if you are comfortable doing that).
Edit: Glancing back at your original post, my guess is that you have some sort of filesystem corruption. You mention snapshots, so I assume this is a Btrfs filesystem. You might want to use some of Btrfs’ read-only and/or recovery options when mounting the filesystem in the Live environment. I’m not a Btrfs expert. If your rootfs is Btrfs and you think the problem might in fact be filesystem corruption, feel free to add the Btrfs tag to your original post to try to get the attention of someone who might be more familiar with those recovery commands.
Yeah, after actually taking a look and trying to find that directory in dolphin, I found that most of the root folders (home, var, tmp, opt etc.) are there but all of them are empty apart from some files in var, usr, etc and boot. Really strange and concerning to me that this happened seemingly out of nowhere, the drive still shows as being nearly full on the free space status bar in Dolphin as it was pre corruption but only 54.1 GB of about 800GB are left showing in Firelight and Dolphin. The command you suggested also returned saying the journal folder did not exist. (the rest of the folders its contained in do but as mentioned, empty)
There might still be a snapshot somewhere that has all (or most all) of your files. I wouldn’t know how to find it though. If he is around, you might try to ping Chris Murphy (cmurf) in the #btrfs Matrix room.
I’m wondering if the snapshots are what has filled up the drive.
You could try installing btrfs-assistant within the live environment (just sudo dnf install btrfs-assistant from a terminal within your live session), and run that app. That should give you a friendly GUI view of what’s taking up the space on the partition.
I had a look using btfs-assistant and I still have 100GB left on the drive, even if snapshots filled up the drive I dont see how that would have caused the system breaking. I’ve got around 70 snapshots listed with the assistant going back to march but yeah the .snapshots folder just shows as completely empty otherwise in Dolphin and with ls.
At some point while working (I assume when a snapshot was being created) Fedora froze, and the only option was a hard reset. After that it wouldn’t boot. It starts to boot, but gets stuck in an infinite loop when trying to start systemd-tmpfiles-setup.service/start. My suspicion fell on full metadata on the Btrfs partition. I booted a live system and indeed it turned out that virtually all allocated space was in use.
Just gave up and tried to reinstall with the live image using the whole drive and just got this error on the first installation step (configuring storage) lol.
org.fedoraproject.Anaconda.StorageInstallationError: Process reported exit code 1: ERROR: Could not destroy subvolume/snapshot: Directory not empty
Maybe you could run wipefs -a /dev/sda? Be sure you specify the correct drive! That command will completely remove everything (all partitions, all filesystems) instantly from the whole drive.
That worked, thanks! Despite that its still so strange that the system randomly corrupted and I guess wiping the whole drive and reinstalling wasn’t really the best solution lol, thank god for cloud backups