Rescuing a kinoite or silverblue install after breaking /etc - how?

Suppose I make a mistake with a modification to /etc in kinoite or silverblue, and the system no longer boots. What do I do?

I tried booting the install iso (Fedora-Kinoite-ostree-x86_64-36-1.5.iso) on a “CD” (it’s qemu/kvm, so a virtual CD) with my Kinoite installation present on /dev/sda in that VM, and the iso has a “Troubleshoot Fedora 36” option. That looked promising. Trying that, I next get to “Rescue a Fedora system” with the even more promising caption: “If the system will not boot, this lets you access files and edit config files to try to get it booting again.” Great! So I try that, and it tells me it will mount my installation at /mnt/sysroot, and I can either have it read-writable by selecting “1) Continue” or read-only by selecting “2) Read-only mount”. From there on, however, things go downhill. Either of those selections produces “You don’t have any Linux partitions. …” and offers me a shell from which I cannot see any part of my installation immediately. I manually mounted /dev/sda2, and I can see /home, /root and /var. OK - /home has the user dirs, which is useful if I need to copy them off a broken install. But, nowhere do I see /etc. I made changes to /etc in this install, so there must be some persistent state on the HD associated with that. I was assuming it would be in the form of the upper part of an overlay. But where is it?

I realize that the “Troubleshoot Fedora 36” item on the iso’s top menu is probably still targeted to Fedora Workstation, not Silverblue nor Kinoite, even though the iso is for installing Kinoite. I assume that will change at some point as these things mature.

But what about now? Is there a roadmap for how to get to /etc to make changes to rescue a Kinoite or Silverblue install?

/etc is on the root subvolume mounted at / you can find the default etc in /var/etc I believe.

I did find /etc (by using find, I should have thought of that first), but not in var. It is in root/ostree/deploy/fedora/deploy/*/ under the /dev/sda2 mount point, where * matches a long UID named dir. Looks like the entire root partition of the install is there, including all of /etc.

Is it safely editable? I guess I’ll find out…

Should be but it does get written (again) after an update since it is part of the three way merge of the server side commit + your layered stuff + your configs (etc…)

It seems to work if I edit files in /etc that I put there myself. Maybe then also any that I modified? Maybe not others? Or is /etc always safe to edit in all such cases? But /usr isn’t safe?

Ideally, I’d like to be able to chroot to do rescue work. Would that ever be possible?

/etc should be mutable and so should /var.
If you look at the ostree doc’s you can chroot into a commit I believe. Maybe start here for some guidance. As well @miabbott had a blog post about ostree which was quite informational, but I can’t seem to find the link today. Maybe he can help point you further along this road.

It’s been a while since I’ve written any blogs, but the ones that I have written should still be available at https://miabbott.github.io/

The “Getting to Know Fedora Silverblue” entry is a bit dated, but might be of use?

1 Like

Thanks for those links!