Enabling services after layering without reboot

Ask Fedora
1

I am layering a bunch of packages through a systemd-unit as part of the initial boot and I am able to do it with only one reboot, based on the solution here: Reboot once post ignition in Fedora CoreOS? - #2 by dustymabe

E.g. I am layering greenboot on top.

However, I have the general issue, that I cannot use a) systemd ignition snippets to enable the service and b) not use a script invocation after the overlaying but before reboot to enable the service, because systemd does not yet know about the service units and thus has no clue in which target to install the symlinks.

As a solution I now propagate a script, that will enable the service if the service unit is present.

This way I keep the whole reboot cycle to only one and at the same time have a more generic approach.

Even if I would go with a secondary reboot, I would still have to seperate the service enabling from ignition, since I need to boot into the overlay that has the units in it.

There doesn’t really seem to be another solution, even if I would build a custom ostree, layer my packages in there and then rebase to that layer at ignition. I would still have the issue, that when ignition configures systemd, the units are not yet present and thus it won’t enable them.

What is a good solution to this?

My goal is to have all the necessary software being installed through ignition to have the node ready so that higher orchestration can manage the workload on top. And thus I don’t want to do the layering with yet another tool (e.g. ansible).

2

Hello @maha and welcome to :fedora: !

Could you share your Butane config so we can try to reproduce the issue?

3

Hi @hricky

sure:

variant: fcos
version: 1.6.0

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 foo

storage:
  directories:
    - path: /var/cache/rpm-ostree-install

systemd:   
  units:   
    - name: rpm-ostree-install-greenboot.service
      enabled: true
      contents: | 
        [Unit]
        Description=Layer greenboot with rpm-ostree
        Wants=network-online.target
        After=network-online.target
        Before=rpm-ostree-install-all-packages.service
        ConditionPathExists=!/var/cache/rpm-ostree-install/greenboot.stamp
      
        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/flock /var/cache/rpm-ostree-install/.lock -c "/usr/bin/rpm-ostree install --assumeyes --idempotent --allow-inactive --reboot greenboot greenboot-default-health-checks | tee /var/cache/rpm-ostree-install/greenboot.stamp"
        
        [Install]
        WantedBy=multi-user.target

    - name: greenboot-task-runner.service
      enabled: true
    - name: greenboot-healthcheck.service
      enabled: true
    - name: greenboot-status.service
      enabled: true
    - name: greenboot-loading-message.service
      enabled: true
    - name: greenboot-grub2-set-counter.service
      enabled: true
    - name: greenboot-grub2-set-success.service
      enabled: true
    - name: greenboot-rpm-ostree-grub2-check-fallback.service
      enabled: true
    - name: redboot-auto-reboot.service
      enabled: true
    - name: redboot-task-runner.service
      enabled: true

This gives me after one reboot a system with greenboot installed, but (for obvious reasons) not yet with greenboot running:

[core@localhost ~]$ systemctl status greenboot-status.service 
○ greenboot-status.service - greenboot MotD Generator
     Loaded: loaded (/usr/lib/systemd/system/greenboot-status.service; disabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: inactive (dead)
[core@localhost ~]$ ls -l /etc/systemd/system/multi-user.target.wants/
total 0
lrwxrwxrwx. 1 root root 46 Mar 31 18:28 NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.service
lrwxrwxrwx. 1 root root 49 Mar 31 18:28 afterburn-checkin.service -> /usr/lib/systemd/system/afterburn-checkin.service
lrwxrwxrwx. 1 root root 59 Mar 31 18:28 afterburn-firstboot-checkin.service -> /usr/lib/systemd/system/afterburn-firstboot-checkin.service
lrwxrwxrwx. 1 root root 43 Mar 31 18:28 audit-rules.service -> /usr/lib/systemd/system/audit-rules.service
lrwxrwxrwx. 1 root root 38 Mar 31 18:28 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx. 1 root root 39 Mar 31 18:28 chronyd.service -> /usr/lib/systemd/system/chronyd.service
lrwxrwxrwx. 1 root root 83 Mar 31 18:28 console-login-helper-messages-gensnippet-os-release.service -> /usr/lib/systemd/system/console-login-helper-messages-gensnippet-os-release.service
lrwxrwxrwx. 1 root root 81 Mar 31 18:28 console-login-helper-messages-gensnippet-ssh-keys.service -> /usr/lib/systemd/system/console-login-helper-messages-gensnippet-ssh-keys.service
lrwxrwxrwx. 1 root root 60 Mar 31 18:28 coreos-check-cgroups-version.service -> /usr/lib/systemd/system/coreos-check-cgroups-version.service
lrwxrwxrwx. 1 root root 53 Mar 31 18:28 coreos-check-ssh-keys.service -> /usr/lib/systemd/system/coreos-check-ssh-keys.service
lrwxrwxrwx. 1 root root 63 Mar 31 18:28 coreos-check-wireless-firmwares.service -> /usr/lib/systemd/system/coreos-check-wireless-firmwares.service
lrwxrwxrwx. 1 root root 57 Mar 31 18:28 coreos-fix-selinux-labels.service -> /usr/lib/systemd/system/coreos-fix-selinux-labels.service
lrwxrwxrwx. 1 root root 61 Mar 31 18:28 coreos-ignition-delete-config.service -> /usr/lib/systemd/system/coreos-ignition-delete-config.service
lrwxrwxrwx. 1 root root 60 Mar 31 18:28 coreos-ignition-write-issues.service -> /usr/lib/systemd/system/coreos-ignition-write-issues.service
lrwxrwxrwx. 1 root root 54 Mar 31 18:28 coreos-liveiso-success.service -> /usr/lib/systemd/system/coreos-liveiso-success.service
lrwxrwxrwx. 1 root root 61 Mar 31 18:28 coreos-platform-chrony-config.service -> /usr/lib/systemd/system/coreos-platform-chrony-config.service
lrwxrwxrwx. 1 root root 42 Mar 31 18:28 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root 39 Mar 31 18:28 machines.target -> /usr/lib/systemd/system/machines.target
lrwxrwxrwx. 1 root root 41 Mar 31 18:28 mdmonitor.service -> /usr/lib/systemd/system/mdmonitor.service
lrwxrwxrwx. 1 root root 41 Mar 31 18:28 nfs-client.target -> /usr/lib/systemd/system/nfs-client.target
lrwxrwxrwx. 1 root root 51 Mar 31 18:28 ostree-finalize-staged.path -> /usr/lib/systemd/system/ostree-finalize-staged.path
lrwxrwxrwx. 1 root root 48 Mar 31 18:28 remote-cryptsetup.target -> /usr/lib/systemd/system/remote-cryptsetup.target
lrwxrwxrwx. 1 root root 40 Mar 31 18:28 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx. 1 root root 56 Mar 31 18:28 rpm-ostree-install-greenboot.service -> /etc/systemd/system/rpm-ostree-install-greenboot.service
lrwxrwxrwx. 1 root root 36 Mar 31 18:28 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx. 1 root root 36 Mar 31 18:28 sssd.service -> /usr/lib/systemd/system/sssd.service
lrwxrwxrwx. 1 root root 45 Mar 31 18:28 systemd-homed.service -> /usr/lib/systemd/system/systemd-homed.service
lrwxrwxrwx. 1 root root 39 Mar 31 18:28 zincati.service -> /usr/lib/systemd/system/zincati.service
[core@localhost ~]$ rpm -ql greenboot | grep systemd
/usr/lib/systemd/system/greenboot-grub2-set-counter.service
/usr/lib/systemd/system/greenboot-grub2-set-success.service
/usr/lib/systemd/system/greenboot-healthcheck.service
/usr/lib/systemd/system/greenboot-loading-message.service
/usr/lib/systemd/system/greenboot-rpm-ostree-grub2-check-fallback.service
/usr/lib/systemd/system/greenboot-status.service
/usr/lib/systemd/system/greenboot-task-runner.service
/usr/lib/systemd/system/redboot-auto-reboot.service
/usr/lib/systemd/system/redboot-task-runner.service
/usr/lib/systemd/system/redboot.target

I can do anoter enablement and reboot and it works:

[root@localhost ~]#           systemctl enable greenboot-task-runner.service
          systemctl enable greenboot-healthcheck.service
          systemctl enable greenboot-status.service
          systemctl enable greenboot-loading-message.service
          systemctl enable greenboot-grub2-set-counter.service
          systemctl enable greenboot-grub2-set-success.service
          systemctl enable greenboot-rpm-ostree-grub2-check-fallback.service
          systemctl enable redboot-auto-reboot.service
          systemctl enable redboot-task-runner.service
Created symlink '/etc/systemd/system/multi-user.target.wants/greenboot-task-runner.service' → '/usr/lib/systemd/system/greenboot-task-runner.service'.
Created symlink '/etc/systemd/system/multi-user.target.wants/greenboot-healthcheck.service' → '/usr/lib/systemd/system/greenboot-healthcheck.service'.
Created symlink '/etc/systemd/system/boot-complete.target.requires/greenboot-healthcheck.service' → '/usr/lib/systemd/system/greenboot-healthcheck.service'.
Created symlink '/etc/systemd/system/multi-user.target.wants/greenboot-status.service' → '/usr/lib/systemd/system/greenboot-status.service'.
Created symlink '/etc/systemd/system/multi-user.target.wants/greenboot-loading-message.service' → '/usr/lib/systemd/system/greenboot-loading-message.service'.
Created symlink '/etc/systemd/system/ostree-finalize-staged.service.requires/greenboot-grub2-set-counter.service' → '/usr/lib/systemd/system/greenboot-grub2-set-counter.service'.
Created symlink '/etc/systemd/system/multi-user.target.wants/greenboot-grub2-set-success.service' → '/usr/lib/systemd/system/greenboot-grub2-set-success.service'.
Created symlink '/etc/systemd/system/greenboot-healthcheck.service.requires/greenboot-rpm-ostree-grub2-check-fallback.service' → '/usr/lib/systemd/system/greenboot-rpm-ostree-grub2-check-fallback.service'.
Created symlink '/etc/systemd/system/redboot.target.wants/redboot-auto-reboot.service' → '/usr/lib/systemd/system/redboot-auto-reboot.service'.
Created symlink '/etc/systemd/system/redboot.target.requires/redboot-task-runner.service' → '/usr/lib/systemd/system/redboot-task-runner.service'.
[root@localhost ~]# systemctl reboot 

Broadcast message from root@localhost on pts/1 (Mon 2025-03-31 18:32:56 UTC):

[....]


[core@localhost ~]$ systemctl status greenboot-status.service 
● greenboot-status.service - greenboot MotD Generator
     Loaded: loaded (/usr/lib/systemd/system/greenboot-status.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: active (exited) since Mon 2025-03-31 18:35:26 UTC; 33s ago
 Invocation: d1b912e21b5a4fddb8d1b74b94a0711b
    Process: 1710 ExecStart=/usr/libexec/greenboot/greenboot-status (code=exited, status=0/SUCCESS)
   Main PID: 1710 (code=exited, status=0/SUCCESS)
   Mem peak: 1.7M
        CPU: 23ms

Mar 31 18:35:26 localhost.localdomain systemd[1]: Starting greenboot-status.service - greenboot MotD Generator...
Mar 31 18:35:26 localhost.localdomain greenboot-status[1721]: Script '01_update_platforms_check.sh' FAILURE (exit code '1'). Continuing...
Mar 31 18:35:26 localhost.localdomain greenboot-status[1721]: Boot Status is GREEN - Health Check SUCCESS
Mar 31 18:35:26 localhost.localdomain systemd[1]: Finished greenboot-status.service - greenboot MotD Generator.

[core@localhost ~]$ ls -l /etc/systemd/system/multi-user.target.wants/ | grep greenboot
lrwxrwxrwx. 1 root root 59 Mar 31 18:32 greenboot-grub2-set-success.service -> /usr/lib/systemd/system/greenboot-grub2-set-success.service
lrwxrwxrwx. 1 root root 53 Mar 31 18:32 greenboot-healthcheck.service -> /usr/lib/systemd/system/greenboot-healthcheck.service
lrwxrwxrwx. 1 root root 57 Mar 31 18:32 greenboot-loading-message.service -> /usr/lib/systemd/system/greenboot-loading-message.service
lrwxrwxrwx. 1 root root 48 Mar 31 18:32 greenboot-status.service -> /usr/lib/systemd/system/greenboot-status.service
lrwxrwxrwx. 1 root root 53 Mar 31 18:32 greenboot-task-runner.service -> /usr/lib/systemd/system/greenboot-task-runner.service
lrwxrwxrwx. 1 root root 56 Mar 31 18:28 rpm-ostree-install-greenboot.service -> /etc/systemd/system/rpm-ostree-install-greenboot.service

So to me it’s obvious you cannot use the ignition systemd definitions for things that are not yet installed and I end up thing something like:

variant: fcos
version: 1.6.0
storage:
  files:
    - path: /var/local/greenboot/activate-service.sh
      mode: 0700
      contents:
        inline: |
          #!/bin/bash -ex

          # activate greenboot
          systemctl enable greenboot-task-runner.service
          systemctl enable greenboot-healthcheck.service
          systemctl enable greenboot-status.service
          systemctl enable greenboot-loading-message.service
          systemctl enable greenboot-grub2-set-counter.service
          systemctl enable greenboot-grub2-set-success.service
          systemctl enable greenboot-rpm-ostree-grub2-check-fallback.service
          systemctl enable redboot-auto-reboot.service
          systemctl enable redboot-task-runner.service

          systemctl start --no-block greenboot-healthcheck.service
          systemctl start --no-block greenboot-status.service

          systemctl disable activate-greenboot.service
  directories:
    - path: /var/cache/rpm-ostree-install
    - path: /var/local/greenboot
systemd:   
  units:   
   - name: rpm-ostree-install-greenboot.service
     enabled: true
     contents: | 
       [Unit]
       Description=Layer greenboot with rpm-ostree
       Wants=network-online.target
       After=network-online.target
       Before=rpm-ostree-install-all-packages.service
       ConditionPathExists=!/var/cache/rpm-ostree-install/greenboot.stamp
     
       [Service]
       Type=oneshot
       RemainAfterExit=yes
       ExecStart=/usr/bin/flock /var/cache/rpm-ostree-install/.lock -c "/usr/bin/rpm-ostree install --assumeyes --idempotent --allow-inactive greenboot greenboot-default-health-checks | tee /var/cache/rpm-ostree-install/greenboot.stamp"
       
       [Install]
       WantedBy=multi-user.target

   - name: activate-greenboot.service
     enabled: true
     contents: | 
       [Unit]
       Description=Activate greenboot services
       ConditionPathExists=!/etc/systemd/system/multi-user.target.wants/greenboot-status.service
       ConditionPathExists=/usr/libexec/greenboot/greenboot-status
     
       [Service]
       Type=oneshot
       RemainAfterExit=yes
       ExecStart=/usr/bin/bash /var/local/greenboot/activate-service.sh
       
       [Install]
       WantedBy=multi-user.target

Is this the way to go or how are others handling this?

greenboot is just an example here, why I am trying to do is to have everything in ignition up to the point where the system is managed by higher levels of orchestration…

4

I tested the following Butane configuration in a virtual machine and it installed and enabled the GreenBoot services. However, I’m not familiar with this application and it will probably need more configuration to get it working, but for the sake of the demo/example it should be good enough.

Install GreenBoot and enable the services 
variant: fcos
version: 1.6.0

passwd:
  users:
    - name: core
      ssh_authorized_keys: [ssh-ed25519 AAAA ...]

storage:
  files:
    - path: /usr/local/bin/enable_greenboot_services.sh
      mode: 0755
      contents:
        inline: |
          #! /usr/bin/env bash

          set -euo pipefail

          main() {

            greenboot_services="
              greenboot-task-runner.service
              greenboot-healthcheck.service
              greenboot-status.service
              greenboot-loading-message.service
              greenboot-grub2-set-counter.service
              greenboot-grub2-set-success.service
              greenboot-rpm-ostree-grub2-check-fallback.service
              redboot-auto-reboot.service
              redboot-task-runner.service
            "

            for service in ${greenboot_services}
            do
              systemctl enable ${service}
            done

          }

          main 

systemd:
  units:
    # Install GreenBoot
    - name: rpm-ostree-install-greenboot.service
      enabled: true
      contents: |
        [Unit]
        Description=Install GreenBoot
        Wants=network-online.target
        After=network-online.target
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/rpm-ostree install -yA greenboot greenboot-default-health-checks
        ExecStart=/usr/local/bin/enable_greenboot_services.sh
        ExecStart=/usr/bin/touch /var/lib/%N.stamp
        ExecStart=/usr/bin/systemctl --no-block reboot

        [Install]
        WantedBy=multi-user.target
5

Hi @hricky

Yes, this works because you install the RPM also into the liveOS and so systemd knows about the units and can enable them directly.

What is the general recommendation with --apply-live with rpm-ostree? Is it something that is ok (since it has an advantage in this case) or should it be avoided?

Or asked differently: why is it not the default?

6

I’m not sure what the general recommendation is for the rpm-ostree install --apply-live option, if there is one. I think the only downside is that to install some apps the system needs to be restarted for them to work properly and maybe that’s why it’s not the default. There are more technical aspects to this and my personal suggestion would be to restart the machine after every new software installation. As you mentioned, in this particular case it’s only beneficial since we’re rebooting after installation anyway.