Cannot boot from initramfs (unable to switch root, sysroot directory is empty)

Ask Fedora
, ,
1

Greetings everyone.

Okay, so I have a problem with Fedora 34. More specifically, I have a problem with a custom built image that is part of the services that we deliver.

The Problem

The problem has to do with initramfs. See, we’ve been using initrd up until now, but the amount of packages and binaries have increased since our last upgrade from Fedora 32 to Fedora 34. Without manually adding new packages, the base, release and dependency packages have now increased to a point where initrd has become over 500MB in size.

The size of this initrd makes this image very unstable: legacy PXE doesn’t work for instance, because this image is too big for TFTP and will freeze on most networks. However, the biggest, most crucial issue we have, is that GRUB2 is not allowing this initrd to boot anymore, mainly on HP laptops. Whenever we go back to Fedora 32, the images seem to be more stable. We think it’s because of initrd being smaller here. The default Fedora Workstation (34) images also work on systems where our custom image is getting kernel panics.

Our image uses a /vmlinuz and /initrd file to boot into a RAMDisk using either USB (GRUB2) or PXE (iPXE). This is by design: Our application is a Chromium-based, integrated kiosk web browser, running on Fedora (34). It is supposed to be volatile and stateless.

Initramfs

I’ve been looking into initramfs as a replacement to initrd, since it’s much newer, much lighter, and the size of this file may not trigger any bugs in GRUB2. Perhaps this will also enable us the use of legacy PXE implementations like PXELINUX and Windows Deployment Services. I figured I could easily replace the use of CPIO + XZ with Dracut. But I’ve been stuck on an issue I can’t really wrap my head around…

The Paradox

Replacing initrd with initramfs in GRUB2 isn’t that hard. If I use the same name for both files (not at the same time), it would look for the file name in the config and find the file. Easy, right? But then, the kernel messages pop up, and it fails to boot reading something like: Failed to switch root: Specified switch root path /sysroot does not seem to be an OS tree. os-release file is missing.

Needless to say, I checked out the contents of /sysroot: empty. There should be an os-release file in /etc. Reading up on “Sysroot”, it’s supposed to mount the OS directory structure, which is probably why it’s telling me that it’s missing some files.

Sysroot

Here’s where I got lost: The directory structure is inside initramfs, right?. But in order to run initramfs through GRUB2, I need to mount the directory structure to Sysroot. That means I can only reach /sysroot from inside initramfs. But I can’t mount initramfs, because I need the OS-release file in /sysroot. I also can’t mount the directories inside initramfs, because it’s an image file. Can you see my paradox? How am I supposed to mount to /sysroot, when this requires something inside initramfs itself? I’ve already tried the root= kernel parameter with a PARTUUID or LABEL, but I can’t use UUIDs since it’s vFAT over UEFI.

Did I miss something here? Am I even supposed to use initramfs, when using bootable USB RAMDisks? If not, should I look into splitting up initrd into multiple files? How do I even do this?

NOTE: We are already using the highest possible compression (XZ level 9). I’ve tried splitting initrd before, but I have not been able to do it properly.

I personally don’t like asking for help, but when all the Google search results are in purple, I may need a point in the right direction. A lot of systems are not working after our upgrade to F:34 and I would really like to get this fixed. Does anybody know how to use Dracut properly? Anyone with experience in generating initramfs images? Why is my /sysroot not already mounted?

Help is much appreciated, thank you.

1 Like
2

I do not work with custom images, but if you do “lsinitrd /boot/initramfs-<kernel>.img” on the fedora 34 workstation you get a full listing of the content of the image. The initramfs image is about 35 MB which is considerably smaller than the size you posted.

I think you are building something wrong since the associated vmlinuz image is only about 10 MB.

I don’t think the initrd or initramfs image need to contain the entire OS binaries, but only the amount needed to start the system up to the point where switch root takes over and looks at the drive content directly.

3

Yes, I remember. Using this command:

lsinitramfs initramfs-5.13.7-200_34.img | grep os-release

Returns these values:

etc/os-release
usr/lib/os-release

So the file “Switch Root” is looking for (os-release) is right there! But how do I mount it through GRUB2? How do reach these files inside initramfs, while trying to boot from the same file?

I have difficulty grasping the workings of this. Anybody have an idea?

4

It sounds like what you want to do is create a custom Live CD (or “Live Image”, it doesn’t have to be written to a CD, it can be written to a USB drive). I’ve never done it, but there is some documentation on how to do that here: Creating and using a live installation image :: Fedora Docs

That is normally done by setting the root=… parameter on the kernel command line. The root=… parameter should never be pointed at the initramfs. Although the initramfs does have a copy of the os-release file, it is not a full rootfs. The initramfs is a minimal image that is only intended for some early initialization (mainly loading the drivers needed to mount the rootfs).

5

Thank you for your response,

That LiveCD stuff doesn’t look like what we’re doing at all… Our product is already working with initrd. We just want to try using initramfs, to work around a bug in GRUB2 caused by having a too large initrd file.

All we are doing is using CPIO + XZ to create an initrd, and supply this with a proper vmlinuz (kernel). These two files can then be booted into RAM by GRUB2 on most systems, but some of them are getting kernel panics now.

Your remark about the root= parameter: alright, it’s good to know that I don’t need to mount something inside initramfs. But where can I find the files that I need to mount to /sysroot then? Whatever I supply for root=, becomes mounted in /sysroot? This makes sense to me, but how do I populate /sysroot with the right files/mount?

6

The filesystem that you point root= at should be the same one that you were in when you ran dracut to generate the initramfs. It is typically a partition. It could also be a network share over a protocol like NFS, iSCSI, or NBD.

7

Ah I see… Unfortunately, our initramfs file is generated inside a container, which no longer exists after the build has been completed.

Any other solutions which require all of our clients to install additional network shares, is also out of the question.

Does this mean initramfs is not the right method, and we should look into splitting up initrd?

8

Maybe. I’m not sure that you are going to have much luck splitting up initrd either though. It will ultimately have to be combined to be usable as a rootfs.

Personally, I would consider formatting the USB drive with a filesystem that will work with Linux (e.g. ext4) and then writing your “initrd” files directly to that. Then you wouldn’t need an initrd (or initramfs) at all. Just something like root=PARTLABEL=USB001 passed to the kernel should be sufficient.

Another option that might be a little closer to what you are doing might be to use the loop and loopfstype options explained here: linux - Can I make SYSLINUX load a root fs from an image file on an NTFS partition? - Super User

I have no experience with the latter though. Also, either way, you would need to install the readonly-root package and set READONLY=yes in /etc/sysconfig/readonly-root if you want to be sure that user changes do not persist.

P.S. In addition to setting READONLY=yes in /etc/sysconfig/readonly-root, you will need to add the ro option to the list of kernel parameters to be sure that the root filesystem is mounted readonly.