How do I recover an unbootable Fedora Workstation system?

I’m a new Fedora user, and I actually haven’t had a bad update yet. So this question is for educational purposes only! Though, sometime around Fedora 36, my test laptop which I hadn’t used for a few months suddenly booted to the grub shell and I had no idea what to do because I was completely unfamiliar with the grub shell, so I just reinstalled it.

I’m coming from Arch Linux. Here’s what I do if my system is no longer bootable:

  1. Get my Arch LiveCD and boot from it.
  2. Mount the Swap, RootFS, and Home partitions.
  3. Use arch-chroot to get into the system.
  4. Solve the problem, which usually means upgrading to the latest packages, or re-installing GRUB.

And that’s all there is to it. I never bothered setting up snapshots/recovery images for my system because I knew there was no chance of being unable to recover the system except in case of a hard drive failure.

I checked docs.fedoraproject.org, but there didn’t seem to be any information about system rescue in Fedora Workstation or Quick Docs. I couldn’t find much useful information on Fedora Project Wiki either.

I’m looking for a fairly detailed guide or pointers to related topics… I never really understood the Arch process beyond the basics; I just followed a ritual.

Thanks!

Hello @diligence3569 ,
There is a rescue kernel, which it is a good idea to boot into on occasion, as you update your system over time. The live image has everything necessary for recovery of most common boot issues. The troubleshoot section (of the installer image) also has a “recover existing Fedora system” option.

1 Like

When Linux breaks, it is often necessary to use command-line tools, if only to analyze the reasons behind the breakage so they can be addressed. Too many users resort to drastic measures such as a reinstall, which may resurrect a broken system but doesn’t help others understand what caused the problem. I recommend becoming familiar with Linux Command so you can refer to it when trying to understand requests to run commands in a terminal.

The basic outline of your description the Arch recovery procedure is the same for most linux systems. You need a way to boot the system, either from a USB drive or memory “key” or using a “rescue” kernel. “chroot” is a way to use kernel booted from other media with the non-booting filesystem in order to make repairs. Every linux system needs backups, alternative bootable media, and alternative network access (e.g., USB WiFi key or ethernet cable).

3 Likes

Thanks for the information, both of you!

I’m very comfortable with the command line; I just need a resource to learn the commands. My understanding is arch-chroot simplifies a lot of the process compared to the manual work you need to do with other distributions.

Maybe I should look at the documentation out there for RHEL, actually?

But it’s good to know there’s a handy recovery tool in the live image!

The link above provides a very good reference for learning the commands and how to use them.

Having the reference, however, does not replace the learning experience of actually trying various commands and becoming comfortable with using the command line.

The primary documentation are the man pages, although the quality varies a lot. For example man bash tells you everything about bash and much more.

Next, for some commands you have the info pages. The info pages are a GNU invention. As an example you have info grub2 or info gcc. The pinfo command is an alternative command with a different user interface. You can read both the man pages and info pages offline as they are found on your own system.

Both man pages and info pages are available or the net, for example https://www.gnu.org/software/grub/manual/grub/grub.html contains the same information as info grub2. Search for “info grub” in your search engine to find it. Sometime these pages doesn’t match the exact version you have on your own system, so keep that in mind. This is especially true for the Fedora version of grub2 which uses a special grub2 version of the BLS configuration files. You won’t find that on most other systems.

Tha Arch wiki pages has some very good information about some common linux programs.

1 Like

Fedora has systemd-nspawn which is a modern take on the chroot process. It also works for containers and SELinux bounds.

All you would need to do is systemd-nspawn -D if you add a -b you can attempt to boot a system if you are unsure.

1 Like

The recovery process is similar to what you described for Arch adding some points specific to Fedora such as SELinux and RPM packages:

2 Likes

Awesome tip, thank you! This is something that will work for any distribution, as I understand it. (Well, minus the SystemD-free distributions)

@vgaetera Ahh! I can’t believe I missed that section in the Fedora docs. Thank you for pointing it out to me! I was a bit concerned about the SELinux stuff as I know nothing about it and it has a reputation for being inscrutable, so that link should be helpful.

systemd-nspawn sounds like the solution I want to use, but I’ll mark @vgaetera’s answer as the solution since it does refer specifically to Fedora/RHEL/RPM docs for chrooting and the recovery process.

4 Likes