Rpm-ostree kargs change not in boot options

Project Discussion
1

I used the following on a silverblue 32 beta (20200424): (twice)

[ mike@localhost ~]$ rpm-ostree kargs --append=systemd.unified_cgroup_hierachy=0
   Checking out tree 0750349... done
   Resolving dependencies... done
   Checking out packages... done
   Running pre scripts... done
   Running post scripts... done
   Running posttrans scripts... done
   Writing rpmdb... done
   Writing OSTree commit... done
   Staging deployment... done
   Freed: 44.4 MB (pkgcache branches: 0)
   Kernel arguments updated.
   Run "systemctl reboot" to start a reboot
   [mike@localhost ~]$ rpm-ostree kargs
   resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet root=/dev/mapper/fedora-root ostree=/ostree/boot.1/fedora/ab7bfac27a7cb9333c9fb1b234887f359ff0b5fb8fe357178a73e3bcf25e0ed3/0 systemd.unified_cgroup_hierachy=0
   [mike@localhost ~]$ systemctl reboot
   ..... reboot here
   mike@localhost ~]$ rpm-ostree kargs
   resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet root=/dev/mapper/fedora-root ostree=/ostree/boot.1/fedora/ab7bfac27a7cb9333c9fb1b234887f359ff0b5fb8fe357178a73e3bcf25e0ed3/0
   [mike@localhost ~]$

to turn off cgroups-v2 so I could run moby. However, after I reboot, the kernel argument is not there and the system is still running cgroups-v2. I have added exactly the same kernel argument to a fedora32 server installation, and it works fine. Journalctl shows no sign of a failed boot for a bad kernel argument.

So why has the kernel argument disappeared?

BTW, manually adding the kernel argument by editing the kernel boot line works. It still doesn’t show up in the rpm-ostree kargs output, though.

2

After the reboot, does rpm-ostree status show a message about failing to finalize the previous deployment? You can also check with journalctl -b -1 -u ostree-finalize-staged.service.

3

You are right:

[mike@localhost ~]$ rpm-ostree status
State: idle
Warning: failed to finalize previous deployment
         error: Bootloader write config: grub2-mkconfig: Child process exited with code 127
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/32/x86_64/updates/silverblue
                   Version: 32.20200424.n.0 (2020-04-24T08:16:12Z)
                BaseCommit: 0750349ac06deb6399e2c292f9269686406a1bbd0d0157934aa6244283d771f5
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
           LayeredPackages: emacs-nox moby-engine

  ostree://fedora:fedora/32/x86_64/updates/silverblue
                   Version: 32.20200424.n.0 (2020-04-24T08:16:12Z)
                BaseCommit: 0750349ac06deb6399e2c292f9269686406a1bbd0d0157934aa6244283d771f5
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
           LayeredPackages: moby-engine

[mike@localhost ~]$ journalctl -b -1 -u ostree-finalize-staged.service
-- Logs begin at Wed 2020-04-22 14:53:04 EDT, end at Mon 2020-04-27 20:11:43 EDT. --
Apr 27 10:54:34 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged Deployment.
Apr 27 10:55:34 localhost.localdomain systemd[1]: Stopping OSTree Finalize Staged Deployment...
Apr 27 10:55:34 localhost.localdomain ostree[18767]: Finalizing staged deployment
Apr 27 10:55:36 localhost.localdomain ostree[18767]: Copying /etc changes: 10 modified, 0 removed, 30 added
Apr 27 10:55:36 localhost.localdomain ostree[18767]: Copying /etc changes: 10 modified, 0 removed, 30 added
Apr 27 10:55:38 localhost.localdomain ostree[18767]: error: Bootloader write config: grub2-mkconfig: Child process exited with code 127
Apr 27 10:55:38 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
Apr 27 10:55:38 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Apr 27 10:55:38 localhost.localdomain systemd[1]: Stopped OSTree Finalize Staged Deployment.

Does that explain why the reboot sat there for a long time after the shutdown before actually rebooting? So how would I go about determining exactly why it failed? Where is grub2-mkconfig going for it’s source since that doesn’t seem to be in /etc/default/grub?

I also tried other args:

rpm-ostree kargs --append=kvm-intel.nested=1 --append=intel_iommu=on

Same thing happened. So it’s not the args themselves. Could the kernel args line be too long? Here’s what I get in dmesg:

[0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/ostree/fedora-ab7bfac27a7cb9333c9fb1b234887f359ff0b5fb8fe357178a73e3bcf25e0ed3/vmlinuz-5.6.6-300.fc32.x86_64 resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet root=/dev/mapper/fedora-root ostree=/ostree/boot.1/fedora/ab7bfac27a7cb9333c9fb1b234887f359ff0b5fb8fe357178a73e3bcf25e0ed3/0

4

I realized that /etc/default/grub is there after all. Strangely, after trying the last kernel args for nested vms, and rebooting, I decided to check grub2-mkconfig:

[mike@localhost ~]$ sudo grub2-mkconfig
[sudo] password for mike: 
/etc/default/grub: line 6: systemd.unified_cgroup_hierarchy=0: command not found
[mike@localhost ~]$ sudo cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet" systemd.unified_cgroup_hierarchy=0
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

Clearly, this seems to be a bug. Somewhere I read that in Silverblue, we shouldn’t use the /etc/default/grub method to add kernel arguments, that rpm-ostree kargs was the only right way to do it. I suppose that’s because rpm-ostree does a commit. Could this be done manually?

5

You most likely don’t have this setup anymore but notice that the desired kernel boot parameter is put behind the quotation mark…