Don’t know where I should go with this. I expect the kernel.spec file needs to be adjusted but that is quite a file. As kernel-install is part of systemd that may come into play.
I am expermenting with installing such that the ESP is automounted at /boot which works fine when I install sd-boot and the kernel in a kickstart %post
%post --nochroot
cp -p /run/install/repo/Packages/k/kernel-core-* /mnt/sysroot/root
cp -p /run/install/repo/Packages/k/kernel-modules-core-* /mnt/sysroot/root
%end
%post
bootctl install
printf "root=UUID=$(findmnt -no UUID /) rootflags=subvol=r ro audit=0 bgrt_disable plymouth.enable=0 systemd.hostname=INSERT_HOSTNAME_HERE\n" | tee /etc/kernel/cmdline
rpm -iv /root/kernel-*rpm
Even though everything works there is one detail I would like to see change.
# rpm -qf /usr/lib/modules/6.18.0-0.rc1.251017g98ac9cc4b4452.20.fc44.x86_64/symvers.xz /boot/symvers-6.18.0-0.rc1.251017g98ac9cc4b4452.20.fc44.x86_64.xz
kernel-core-6.18.0-0.rc1.251017g98ac9cc4b4452.20.fc44.x86_64
kernel-core-6.18.0-0.rc1.251017g98ac9cc4b4452.20.fc44.x86_64
The symvers file is copied from /usr/lib/modules to /boot taking up precious space on the ESP. When symlinks are supported it is a symlink but vfat does not support symlinks. Previously this operation would fail due to an attempt to make a symlink in a vfat so kudos to those who fixed that.
Does the symvers really need to be on the ESP at all?
If it has value being on the ESP, it would be better to see it next to the kernel and initrd rather than at the top level.
/boot/5d04dc52cb0e481bbbf072dce22d24f7/6.18.0-0.rc1.251017g98ac9cc4b4452.20.fc44.x86_64/linux
/boot/5d04dc52cb0e481bbbf072dce22d24f7/6.18.0-0.rc1.251017g98ac9cc4b4452.20.fc44.x86_64/initrd
/boot/symvers-6.18.0-0.rc1.251017g98ac9cc4b4452.20.fc44.x86_64.xz
With this work I would like to follow the bls. Fedora seems to be in favor of following specification in general but we’ll see on this one as there is a lot of technical debt carried at the behest of this project’s wonderful corporate sponsor.
The bls would have the ESP mounted at /boot in this scenario. I do not want an XBOOTLDR but if I also had an XBOOTLDR (also vfat as the bls prefers and may soon require) at /boot the path the ESP would mount at becomes /efi. Then the symvers would be
/boot/symvers-6.18.0-0.rc1.251017g98ac9cc4b4452.20.fc44.x86_64.xz
rather than next to the kernel and initrd which seems out of place.
If I just had /boot on the btrfs root subvolume and the ESP at /efi it would be out of spec with the bls.
A workaround that would be very straightforward would be to delete the symvers file from the ESP in the %post. It would still exists in /usr/lib/modules. The ESP would have to be sized large enough to hold it though and each kernel update would need the workaround reapplied. This workaround falls apart if having the symvers on the ESP/XBOOTLDR has value.
While kickstarting Server to produce images that can be deployed to many VMs safely, the handling of /etc/machine-id and the systemd-firtboot logic comes into play. Seems like systemd presets in fedora work differently from this change or these packaging guildlines. Looking at installs using the webui rather than kickstart also have deviations from the packaged presets though the machine-id is assigned at install time thus not ready for use on many VMs.
This investigation came about after perusing dmesg output upon first boot.
Starting initrd-switch-root.service - Switch Root...
Switching root.
. . .
Failed to preset unit /etc/systemd/system/sockets.target.wants/systemd-resolved-varlink.socket: Read-only file system
Failed to preset unit /etc/systemd/system/sockets.target.wants/systemd-resolved-monitor.socket: Read-only file system
Failed to preset unit /etc/systemd/system/sockets.target.wants/systemd-networkd.socket: Read-only file system
Failed to preset unit /etc/systemd/system/sockets.target.wants/systemd-networkd-varlink.socket: Read-only file system
Failed to preset unit /etc/systemd/system/multi-user.target.wants/systemd-networkd.service: Read-only file system
Failed to preset unit /etc/systemd/system/dbus-org.freedesktop.network1.service: Read-only file system
Failed to preset unit /etc/systemd/system/network-online.target.wants/systemd-networkd-wait-online.service: Read-only file system
Failed to preset unit /etc/systemd/system/default.target: Read-only file system
Failed to preset unit /etc/systemd/system/sshd-keygen@rsa.service: Read-only file system
Failed to preset unit /etc/systemd/system/sshd-keygen@ecdsa.service: Read-only file system
Failed to preset unit /etc/systemd/system/multi-user.target.wants/remote-cryptsetup.target: Read-only file system
Failed to preset unit /etc/systemd/system/multi-user.target.wants/remote-integritysetup.target: Read-only file system
Failed to preset unit /etc/systemd/system/multi-user.target.wants/remote-veritysetup.target: Read-only file system
Failed to preset unit: Unit /etc/systemd/system/systemd-homed.service.wants/systemd-homed-activate.service does not exist
Failed to preset unit /etc/systemd/system/dbus-org.freedesktop.home1.service: Read-only file system
Failed to preset unit /etc/systemd/system/multi-user.target.wants/systemd-homed.service: Read-only file system
Failed to preset unit /etc/systemd/system/dbus-org.freedesktop.oom1.service: Read-only file system
Failed to preset unit /etc/systemd/system/multi-user.target.wants/systemd-oomd.service: Read-only file system
Failed to preset unit /etc/systemd/system/sockets.target.wants/systemd-oomd.socket: Read-only file system
Failed to preset unit /etc/systemd/system/sysinit.target.wants/systemd-pstore.service: Read-only file system
Failed to preset unit /etc/systemd/system/timers.target.wants/dnf-makecache.timer: Read-only file system
. . .
initrd-switch-root.service: Deactivated successfully.
Stopped initrd-switch-root.service - Switch Root.
Currently the kickstart ends up with uninitialized in /etc/machine-id both on the rootfs and in the initrd. The same messages are logged if /etc/machine-id is an empty file on both. More combinations are possible also noting that no /etc/machine-id is supposed to behave differently yet.
Looking at a freshly installed server the systemd-delta program shows unit file deviations but provides no information concerning preset deviations.
One attempt to find preset deviations:
systemctl status -a 2>/dev/null | grep '[L]oaded:.*preset' | grep -vE 'disabled; preset: disabled|enabled; preset: enabled'
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-oomd.service; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-userdbd.service; indirect; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd-varlink.socket; enabled; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.socket; enabled; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-resolved-monitor.socket; enabled; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-resolved-varlink.socket; enabled; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/multi-user.target; indirect; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/remote-cryptsetup.target; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/remote-integritysetup.target; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/remote-veritysetup.target; disabled; preset: enabled)
Another attempt
find /usr/lib/systemd/system -type f | while read f;do echo $(basename $f);done | while read u;do echo $(systemctl status $u 2>&1 | grep Loaded:);done | grep preset | grep -vE 'disabled; preset: disabled|enabled; preset: enabled'
Loaded: loaded (/usr/lib/systemd/system/multi-user.target; indirect; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-sysupdate-reboot.service; indirect; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-sysupdate.service; indirect; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-userdbd.service; indirect; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd-varlink.socket; enabled; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd-wait-online.service; enabled; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.socket; enabled; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/remote-cryptsetup.target; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/remote-integritysetup.target; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/remote-veritysetup.target; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-homed-activate.service; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-homed.service; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-oomd.service; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-pstore.service; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/dnf-makecache.timer; disabled; preset: enabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-resolved-monitor.socket; enabled; preset: disabled)
Loaded: loaded (/usr/lib/systemd/system/systemd-resolved-varlink.socket; enabled; preset: disabled)
So what is causing these unit deviations from the presets?
With machine-id being set during the webui installation process and ending up both on the rootfs and in the initrd, systemd-firstboot should never run: is that expected as this would seem to eliminate fedora presets from ever being applied?
The systemd-firstboot has ConditionPathIsReadWrite=/etc so how is it that these preset failures ended up being logged on the first boot?