Silverblue missing boot option

[lyz (Thomas Lynema)]

Changes to the tree and restarts were are taking effect.

The root cause of this may be this:

The .1 location doesn’t exist anymore. There are two .0 locations. I’m wondering if this was due to upgrading to Fedora 32.

Probably relevant is that I had this issue from running livefs https://jlelse.blog/dev/fix-broken-silverblue/ . The livefs command was run after a few update failures to test and see if that would work.

After looking at grub configs, this bug still exists as well grub2-mkconfig when invoked directly generates bad config on EFI · Issue #1009 · ostreedev/ostree · GitHub . That seems unlikely to be related. Still, it isn’t clear if the /boot/loader/entries are correct. There is only one entry in that directory now, ostree-2-fedora-atomic.conf. For whatever reason, there is another one created in the /boot/efi/EFI/fedora/grub.cfg file that also points to the :0 image.

I could manually perform filesystem operations to make the dirs, but don’t know if that is the correct way to go. Does ostree have a clean way to fix this?

[lyz (Thomas Lynema)]

A bit more detail. The system is hitting this issue Common F31 bugs - Fedora Project Wiki . Also, when performing an install
rpm-ostree install fuse-exfat exfat-utils
The result is that no new kernels are added to /boot/ostree and no entry is added in grub.
This may be a twisted manifestation of the issues discussed here Boot entries gone after upgrade . Many of the people on that thread ended up reinstalling.

[lyz (Thomas Lynema)]

Tried disabling selinux due to this message in the journal:

May 27 11:42:46 audit[43320]: AVC avc:  denied  { nnp_transition nosuid_transition } for  pid=43320 comm="dracut" scontext=system_u:system_r:install_t

Performed some updates:

State: idle
Deployments:
  ostree://fedora:fedora/32/x86_64/silverblue
                   Version: 32.20200527.0 (2020-05-27T03:48:57Z)
                BaseCommit: f4d503fdd5382ba0efca2379b7c25ec42e8e6eba13cfcaae8dd0eaec44afbc53
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
                      Diff: 33 upgraded, 2 added

But /boot/ostree only has the old image. No new location for the initramfs nor the kernel image are made.

find / -name vmlinuz*64
/boot/ostree/fedora-atomic-4ca22d3cc75ddd335c7d06d5cbe84676a1b27c561421eaceb9a0bc201288f322/vmlinuz-5.6.13-300.fc32.x86_64

The journal says an initrd was attempted

May 27 11:42:46 rpm-ostree[28985]: dracut: *** Creating initramfs image file '/tmp/initramfs.img' >

Going to try to manually build the grub entries and copy files from

/sysroot/ostree/deploy/fedora-atomic/deploy/b6dfd326fa076abead91e2a3b2ac2062754516e33c831e7c88e75cb223edf083.0/usr/lib/modules/5.6.14-300.fc32.x86_64/

[lyz (Thomas Lynema)]

Created the boot entry manually. System somewhat booted, but most services were unable to start journalctl didn’t log anything.

[lyz@bubu ~]$ cat /boot/loader.0/entries/ostree-1-fedora-atomic.conf
title Fedora 32.20200524.0 (Silverblue) (ostree:1)
version 2
options resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap root=/dev/mapper/fedora-root consoleblank=600 pcie_aspm=off quiet amd_iommu=on video=efifb:off rd.driver.pre=vfio-pci  vfio-pci.ids=10de:1c03,10de:10f1,1022:145f ostree=/ostree/boot.0/fedora-atomic/b6dfd326fa076abead91e2a3b2ac2062754516e33c831e7c88e75cb223edf083/0
linux /ostree/fedora-atomic-b6dfd326fa076abead91e2a3b2ac2062754516e33c831e7c88e75cb223edf083/vmlinuz-5.6.14-300.fc32.x86_64
initrd /ostree/fedora-atomic-b6dfd326fa076abead91e2a3b2ac2062754516e33c831e7c88e75cb223edf083/initramfs-5.6.14-300.fc32.x86_64.img

This is what the files on the non-booting revision look like.

[lyz (Thomas Lynema)]

Everything mentioned in the previous posts is normal. I tested this on a VM. The issue was that the finalization process was failing with

May 27 16:46:34 ostree[44286]: error: Unexpected state: /run/ostree-booted found, but no /boot/loader
directory
May 27 16:46:34 systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, stat
us=1/FAILURE
May 27 16:46:34 systemd[1]: ostree-finalize-staged.service: Failed with result ‘exit-code’.
May 27 16:46:34 systemd[1]: Stopped OSTree Finalize Staged Deployment.
May 27 16:46:34 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=‘unit=ostree-finalize-staged comm=“systemd” exe=“/usr/lib/systemd/systemd” hostname=? addr=? terminal=? res=failed’

After reading this discussion, I ran finalize manually and it worked

My Silverbox OS no longer updates, probably due to an error with grub2-mkconfig (?)

Its still broken for me. I did a “rpm-ostree install gnome-tweaks” and after a reboot it seems as i had never done that. So i did it again and the result is: State: idle AutomaticUpdates: disabled Deployments: ostree://fedora:fedora/31/x86_64/silverblue Version: 31.20191211.0 (2019-12-11T01:41:34Z) BaseCommit: 500e1a1bd0c3a5d1e98857852ad1fce56ab476a95923fb765ad3ce276d8af780 GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C335…

So the issue is /boot being unmounted about 10 seconds before the ostree-finalize-staged service is run.

May 27 16:46:24 systemd[1]: boot.mount: Succeeded.
May 27 16:46:24 systemd[1]: Unmounted /boot.

The bug is somewhere in the ostree-finalize-staged.service file. Maybe RequireMountsFor=/boot would work?

[walters (Colin Walters)]

Maybe RequireMountsFor=/boot would work?

We actually just did that.

What’s in your /etc/fstab? Or systemctl status boot.mount?

[walters (Colin Walters)]

To elaborate, you can use e.g.
systemctl edit ostree-finalize-staged.service
and then paste in:

[Unit]
RequiresMountsFor=/boot

to see if that helps.

[lyz (Thomas Lynema)]

fstab

/dev/mapper/fedora-root / xfs defaults 0 0
UUID=41120299-9366-4262-9aef-3f7ab527045b /boot ext4 defaults 1 2
UUID=C69B-3D18 /boot/efi vfat umask=0077,shortname=winnt 0 2
/dev/mapper/fedora-swap swap swap defaults 0 0
UUID=2136271a-d069-4ee8-a7d0-fdf114eed5cb /home/media ext4 defaults 0 0
UUID=1C5D-FD94 /var/lib/docker/volumes/plex exfat defaults 0 0

systemctl status boot.mount

● boot.mount - /boot
Loaded: loaded (/etc/fstab; generated)
Active: active (mounted) since Wed 2020-05-27 16:47:11 EDT; 14h ago
Where: /boot
What: /dev/sdb2
Docs: man:fstab(5)
man:systemd-fstab-generator(8)
Tasks: 0 (limit: 38356)
Memory: 176.0K
CPU: 11ms
CGroup: /system.slice/boot.mount
Warning: some journal files were not opened due to insufficient permissions.

[lyz (Thomas Lynema)]

Ok, performed the update of the service and updated with systemctl daemon-reload

Looks promising

[root@bubu lyz]# systemctl list-dependencies ostree-finalize-staged.service
ostree-finalize-staged.service
● ├─-.mount
● ├─boot.mount
● └─system.slice

Staged this change:

ostree://fedora:fedora/32/x86_64/silverblue
Version: 32.20200528.0 (2020-05-28T01:22:03Z)
BaseCommit: 0574a6a0da357c4ebade7b07455726eb8295e1b92526cf45bdff613157768871
GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
Diff: 2 upgraded, 1 removed

Hurray!

[lyz@bubu ~]$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/32/x86_64/silverblue
Version: 32.20200528.0 (2020-05-28T01:22:03Z)

Log

May 28 07:20:22 systemd[1]: Stopped OSTree Finalize Staged Deployment.
May 28 07:20:22 systemd[1]: ostree-finalize-staged.service: Consumed 7.880s CPU time.
May 28 07:20:22 systemd[1]: Stopped target Local File Systems.
May 28 07:20:22 kernel: audit: type=1131 audit(1590664822.794:782): pid=1 uid=0 auid=4294967295 ses=4294967
295 subj=system_u:system_r:init_t:s0 msg=‘unit=ostree-finalize-staged comm=“systemd” exe=“/usr/lib/systemd/systemd” hostnam
e=? addr=? terminal=? res=success’
…
May 28 07:20:22 systemd[1]: Unmounting /boot…

So yeah, that fixes it. Thanks much.

[lyz (Thomas Lynema)]

@walters

[Unit]
RequiresMountsFor=/boot

Does this stack, or should it be

[Unit]
RequiresMountsFor=/sysroot /boot

The dependency tree is the same either way.