The rpm-ostreed service errors out when with bootc bare-metal box

Ask Fedora
,
1

I have been playing around with creating a very basic bootc image based off Almalinux. I have my public repo here, and I am using the following kickstart file to provision the bare-metal machine.

I can boot the machine without any problem, however eventually the rpm-ostreed-automatic timer starts the rpm-ostreed-automatic service, which starts the rpm-ostreed and that will error out with the following:

May 04 18:06:02 alma-test systemd[1]: Starting rpm-ostree System Management Daemon...
May 04 18:06:02 alma-test rpm-ostree[7001]: Reading config file '/etc/rpm-ostreed.conf'
May 04 18:06:02 alma-test rpm-ostree[7001]: error: Couldn't start daemon: Error setting up sysroot: /var/lib/rpm must be a symbolic link
May 04 18:06:02 alma-test systemd[1]: rpm-ostreed.service: Main process exited, code=exited, status=1/FAILURE
May 04 18:06:02 alma-test systemd[1]: rpm-ostreed.service: Failed with result 'exit-code'.
May 04 18:06:02 alma-test systemd[1]: Failed to start rpm-ostree System Management Daemon.

So I’m not exactly sure the best way of handling the rpm-ostreed config… any help would be super appreciated!

I know that this is from almalinux, but was suggest to post here for support:

I have a fedora version that I can test with has the same issue, but the alma is one I was testing with over the weekend

2

Hello @ascii17 and welcome to :fedora: !

As you already mentioned, this forum is more appropriate to ask about Fedora/CentOS based bootable containers. I would suggest trying to build a simple (or without any customizations) container image based on Fedora bootc, install it, and see if the issue is still there.

Also, out of curiosity, what approach and tools did you use (using the Kickstart file) to create the disk image and install it on the bare metal machine?

3

Thanks @hricky , I can check as it is fairly easy to swap the os with bootc.

what approach and tools did you use (using the Kickstart file) to create the disk image and install it on the bare metal machine?

So what I did was take the alma linux iso and bake in my kickstart file:

text --non-interactive
eula --agreed

keyboard us
lang en_US.UTF-8

network --noipv6 --onboot=yes --bootproto=dhcp --activate

rootpw --lock
firewall --enabled --ssh
selinux --enforcing

bootloader --location=mbr
zerombr
clearpart --all --initlabel
part /boot     --fstype xfs  --size=1024 --label=BOOTFS
part /boot/efi --fstype vfat --size=1024 --label=EFIFS

part pv.01     --size=100    --grow

volgroup sysvg --pesize=4096 pv.01

logvol /              --fstype xfs --name=lv_root  --vgname=sysvg --size=30720 --label=ROOTFS
logvol /home          --fstype xfs --name=lv_home  --vgname=sysvg --size=13312 --label=HOMEFS   --fsoptions="nodev,nosuid"
logvol /tmp           --fstype xfs --name=lv_tmp   --vgname=sysvg --size=5120  --label=TMPFS    --fsoptions="nodev,noexec,nosuid"
logvol /var           --fstype xfs --name=lv_var   --vgname=sysvg --size=15360 --label=VARFS    --fsoptions="nodev"
logvol /var/lib       --fstype xfs --name=lv_lib   --vgname=sysvg --size=51200 --label=VARLIBFS --fsoptions="nodev"

services --enabled=NetworkManager,sshd

ostreecontainer --url ghcr.io/a1994sc/bootc-images/alma:latest

user --name=sysadmin --plaintext --password=changeit --groups=wheel
# a1994sc public ssh keys
sshkey --username=sysadmin "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBACa5MIyu4mLLLc0D5Y0eOWV1JnvvSo68pDJAh4SyC1WyMVK1eOIlpyDlfFNu7wev8fPELJEwbT+pCsjH2FVU8qRNAH17nW1EBn9xWOX7rEnpxOp6X485+jeA0t/a2jB6e7Bcn86Xwa1tPEbIKS6eo530KMLagaCFpl9arv1SGWeh6/YAw=="

reboot --eject

ref

Here is my github action for creating the iso with my stuff backed in

1 Like
4

So I was able to able to determine that is an issue with the enterprise linux 9.5, both rhel and alma, show the issue when trying start the rpm-ostreed service…

When I manually create the symbolic link, (and before cleaning out the existing /var/lib/rpm), it is happy… very strange

$ rm -rf /var/lib/rpm
$ ln -s ../../usr/lib/rpm/ /var/lib/

I will see if I can find the right place to submit the issue, thanks

5

Also to leave a little bit of debug information here for other people:

Once I verified that everything was good I did the same but with the almalinux:

6

I see rm -rf /var/{cache,log} /var/lib/{dnf,rhsm} in your Containerfiles. I’m not sure if it’s relevant and it’s just a wild guess, but could you try something like the following at the end of your Containerfiles to see if that would solve the issue?

rpm-ostree cleanup --repomd
dnf clean all
rm --recursive --force /var/* /tmp/*
bootc container lint

This is what I put at the end of my Containerfiles, but I currently only use Fedora/CentOS based bootable containers.

7

So I did as you suggested and got the same issue where there is no symbolic link created during the install process…

I guess I could have a post script in the kickstart to remove the existing /var/lib/rpm and create the symbolic link… but that feels hacky

8

So I did test if adding a post-install script would do it… and my system is happy… how silly

9

Well, I would say silly or not, as long as it works and doesn’t break anything, it would probably be good enough. I would also suggest keeping the cleanup and especially the bootc container lint command as it is very useful.

10

Yes sir, I will probably leave it as for now, because it is working; having read more on bootc website and the fedora docs, having the lint command is very important.

Thank you Hristo!

1 Like