Ostree-finalize-staged fails with an SELinux error

Ask Fedora
,
1

Hello !

I am trying to rebase my system from a sericea install to a kinoite image, derived from ublus-os/nividia-kinoite.

My custom image is built from here.

The rebase works well when I run rpm-ostree rebase ostree-unverified-image:registry:quay.io/jbtrystram/fedora-kinoite-nvidia:39

Then I reboot, and nothing appears to be applied: the GRUB menu stays the same, and I am back to the state before I ran the rebase command.

I pinned the latest known good deployment for safety. Here is the state after I ran rpm-ostree reset:

bash-5.2$ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  fedora:fedora/38/x86_64/sericea
                  Version: 38.20231130.0 (2023-11-30T01:10:25Z)
                   Commit: 37842dd2139bbcdeca262bd6180d992bf2709870f3f5b77d84da0b47a40121de
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
                     Diff: 119 upgraded, 325 removed, 1 added

● fedora:fedora/38/x86_64/sericea
                  Version: 38.20231114.0 (2023-11-14T00:50:43Z)
               BaseCommit: 51f4a2509bfc544c6aef0236de53db35d720d255f4af722b6785ddf9612e588a
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: akmod-nvidia alacritty autojump-zsh fira-code-fonts ipxe-roms-qemu mozilla-openh264 vim virt-install virt-manager vulkan-validation-layers xarchiver
                           xorg-x11-drv-nvidia zsh
            LocalPackages: redhat-internal-cert-install-0.1-29.el7.noarch redhat-internal-NetworkManager-openvpn-profiles-non-gnome-0.1-59.el7.noarch
                   Pinned: yes

Then I run the rebase. The log is rather long, here is the full output : paste

But in essence, nothing goes wrong. A bunch of updated packages, a bunch of deleted ones, a bunch of new ones, and

Changes queued for next boot. Run "systemctl reboot" to start a reboot

At the end.

Before reboot I have :

State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  ostree-unverified-image:registry:quay.io/jbtrystram/fedora-kinoite-nvidia:39
                   Digest: sha256:6230a9564c255f010fb44d8298c3d76bee88a9e58bd9e81f1fe4bb51bc498006
                  Version: 39.20231129.0 (2023-11-30T00:17:47Z)
                     Diff: 1104 upgraded, 8 downgraded, 136 removed, 817 added

● fedora:fedora/38/x86_64/sericea
                  Version: 38.20231130.0 (2023-11-30T01:10:25Z)
                   Commit: 37842dd2139bbcdeca262bd6180d992bf2709870f3f5b77d84da0b47a40121de
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464

  fedora:fedora/38/x86_64/sericea
                  Version: 38.20231114.0 (2023-11-14T00:50:43Z)
               BaseCommit: 51f4a2509bfc544c6aef0236de53db35d720d255f4af722b6785ddf9612e588a
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: akmod-nvidia alacritty autojump-zsh fira-code-fonts
                           ipxe-roms-qemu mozilla-openh264 vim virt-install virt-manager
                           vulkan-validation-layers xarchiver xorg-x11-drv-nvidia zsh
            LocalPackages: redhat-internal-cert-install-0.1-29.el7.noarch
                           redhat-internal-NetworkManager-openvpn-profiles-non-gnome-0.1-59.el7.noarch
                   Pinned: yes

So it seems like the deployment is ready and all good.
Then I reboot, and I get :

State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● fedora:fedora/38/x86_64/sericea
                  Version: 38.20231130.0 (2023-11-30T01:10:25Z)
                   Commit: 37842dd2139bbcdeca262bd6180d992bf2709870f3f5b77d84da0b47a40121de
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464

  fedora:fedora/38/x86_64/sericea
                  Version: 38.20231114.0 (2023-11-14T00:50:43Z)
               BaseCommit: 51f4a2509bfc544c6aef0236de53db35d720d255f4af722b6785ddf9612e588a
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: akmod-nvidia alacritty autojump-zsh fira-code-fonts
                           ipxe-roms-qemu mozilla-openh264 vim virt-install virt-manager
                           vulkan-validation-layers xarchiver xorg-x11-drv-nvidia zsh
            LocalPackages: redhat-internal-cert-install-0.1-29.el7.noarch
                           redhat-internal-NetworkManager-openvpn-profiles-non-gnome-0.1-59.el7.noarch
                   Pinned: yes

Back to the beginning.
How can i investigate this more ? Thanks for any pointers

2

I tried to run grub2-mkconfig after running the rebase to update grub, but no luck.

Looks like somthing related to SELinux :

Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Finalizing staged deployment
Copying /etc changes: 31 modified, 0 removed, 129 added
Copying /etc changes: 31 modified, 0 removed, 129 added
Refreshing SELinux policy
libsemanage.semanage_direct_get_module_info: Unable to open procmail module lang ext file at /etc/selinux/targeted/tmp/modules/100/procmail/lang_ext. (No such file or directory).
semodule:  Failed!
Refreshed SELinux policy in 156 ms
error: Finalizing deployment: Finalizing SELinux policy: Child process exited with code 1
ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
ostree-finalize-staged.service: Failed with result 'exit-code'.
Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.

from sudo journalctl -b -1 -u ostree-finalize-staged.service

This is weird. I see procmail have not been maintained for more than 20 years, and is not installed in my image (neither the one I’m trying to rebase from).

Also, the /etc/selinux/targeted/tmp directory does not exist on my system so I am not sure what to do. This service is executed quite late in the shutdown sequence so I don’t know how I could fix it manually.

I know I can interrupt the initramfs sequence with rd.break but not how to look around interrupting the shutdown sequence.
Maybe add OnFailure=emergency.target to ostree-finalize-staged.service ?

Here are some more things I tried :

  • sudo rsync -rclv /usr/etc/selinux/ /etc/selinux/ after reading this
  • touch /etc/selinux/targeted/tmp/modules/100/procmail/lang_ext so selinux will find the file, but it doesn’t work. The message stays the same after rebooting.
3

For kernel arg’s and the like, you use rpm-ostree karg’s.

Yeah that won’t work unless you do it with a live filesystem. Check your EFIVars area to see if you have any files with DUMP in their title there, you could be running out of space there. Should be at /sys/firmware/efi/efivars. One of my Lenovo laptops (E530) Thinkpad was filled up in that area which prevented me from installing Silverblue onto it (or pretty much any other os), until I deleted the files with “DUMP” in their names that I found in this location. I was originally thinking of flashing my Thinkpads BIOS with LibreBoot for an open BIOS and was doing research into the how to when I came across the fact that some BIOS’s actually are on two chips, like with my Lenovo. So some of the boot info gets stored (potentially) onto the second chip. In the end I came to the realization that there was not enough space on my BIOS hardware due to the aforementioned “DUMP” files. So I deleted those and ONLY those files as I am still uncertain how the decision is made on what gets stored on what chip. Oh yeah, in my case the system would do just exactly what you are experiencing, not write the bootloader stuff but not give any warning or error that it failed.

4

Thanks for your input. I just had a look around in the /sys/firmware/efi and the subfolders, no dump-* folders or files to be found unfortunately

Free-space wise It seems to be fine :

Filesystem      Size  Used Avail Use% Mounted on
efivarfs        256K   82K  170K  33% /sys/firmware/efi/efivars
5

I see you are already onto my next thought that it was related to selinux. Were there not some elevation of security measures (options in selinux) introduced in fedora recently around user level ACL’s that maybe wouldn’t account for the real fact an rpm-ostree based system is worked on in user mode not super user.

6

You could do the download only of the upgrade and manually deploy it after the update is finished.

7

After trying a bunch of things I was able to solve my problem, with something I should I tried way earlier.

I was running sericea 38, and tried to rebase to a custom image, derived from ublue-os 39.
So i did a rebase to sericea 39 first, which worked, then rebased to ublue-os/kinoite which works.
So the problem lies with my custom image I’m afraid.

8

I’m running into this issue again today.
It seems to be due to my derive of the ublue-os image.

I get this selinux policy that fails to apply on ostree-finalize-staged.
I can’t figure out where this policy comes from :

# find /etc -name lang_ext | grep procmail
/etc/selinux/targeted/active/modules/100/procmail/lang_ext

# rpm -qf  /etc/selinux/targeted/active/modules/100/procmail/lang_ext
file /etc/selinux/targeted/active/modules/100/procmail/lang_ext is not owned by any package

# ls -l /etc/selinux/targeted/active/modules/100/procmail/        
total 20
-rw-------. 1 root root  2737 Dec 14 10:45 cil
-rw-------. 1 root root 11121 Dec 14 10:45 hll
-rw-------. 1 root root     2 Dec 14 10:45 lang_ext

When running ostree finalize manually I get the same error :

sudo /usr/bin/ostree admin finalize-staged -v                  
OT: remounted /sysroot writable
OT: Deployment 182862618f3f7e1baf74e48fe3386ea054e5c3a07be15c46f71b95f1958012ca.0 unlocked=0
OT: Deployment ce266a106575953fb7ddaf0a819c77cd38ed28c878cce010d3ec6970b8ed2173.0 unlocked=0
Copying /etc changes: 21 modified, 0 removed, 103 added
libsemanage.semanage_direct_get_module_info: Unable to open procmail module lang ext file at /etc/selinux/targeted/tmp/modules/100/procmail/lang_ext. (No such file or directory).
semodule:  Failed!
error: Child process exited with code 1

According to this log /etc/selinux/targeted/tmp/modules/100/procmail/lang_ext is not found, but maybe the tmp in the path should not be there ?
I don’t know what to look for to solve this

9

Perhaps it is a temporary dir created (then destroyed) during the new image created at update after the three-way merge. Isn’t the finalize stage the cleanup and deploy point?