Fedora 42 upgrade - failed to login

ophil · May 5, 2025, 7:48am

hi colleagues
yesterday my fedora 41 auto upgraded by zincati, here is status:

core@fc41dev:~/temp$ rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; trying to stage 42.20250410.3.2 (failed attempts: 5)
Deployments:
  fedora:fedora/x86_64/coreos/stable
                  Version: 42.20250410.3.2 (2025-04-30T17:43:22Z)
                   Commit: baea63252b3d3836668ff0dba57d9f368b591e1387b223293953f2c3f08794cc
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944
                     Diff: 433 upgraded, 3 removed, 5 added

● fedora:fedora/x86_64/coreos/stable
                  Version: 41.20250331.3.0 (2025-04-11T10:53:51Z)
                   Commit: 1f21e2c4c522f97eea5d4383e831f8dc8e5a512e9de00d0fd679b689a1d507d3
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1

It runs as VM in KVM/libvirt/qemu on Ubuntu 24.04. I catch the grub and started the previous version. Some info regarding failed services from v.42 startup log, starting with failed messages:

$ cat 0505v42failed.console 
[DEPEND] Dependency failed for NetworkManager.service - Network Manager.
[DEPEND] Dependency failed for nmstate.service - Apply nmstate on-disk state.
[DEPEND] Dependency failed for NetworkManag…vice - Network Manager Wait Online.
[  OK  ] Reached target network.target - Network.
[  OK  ] Reached target network-online.target - Network is Online.
         Starting dbus-broker.service - D-Bus System Message Bus...
[FAILED] Failed to start dbus-broker.service - D-Bus System Message Bus.
See 'systemctl status dbus-broker.service' for details.
         Starting dbus-broker.service - D-Bus System Message Bus...
[FAILED] Failed to start dbus-broker.service - D-Bus System Message Bus.
See 'systemctl status dbus-broker.service' for details.
         Starting dbus-broker.service - D-Bus System Message Bus...
[FAILED] Failed to start dbus-broker.service - D-Bus System Message Bus.
See 'systemctl status dbus-broker.service' for details.
         Starting dbus-broker.service - D-Bus System Message Bus...
[FAILED] Failed to start dbus-broker.service - D-Bus System Message Bus.
See 'systemctl status dbus-broker.service' for details.
[FAILED] Failed to start dbus-broker.service - D-Bus System Message Bus.
See 'systemctl status dbus-broker.service' for details.
[  OK  ] Reached target basic.target - Basic System.
[  OK  ] Reached target afterburn-sshkeys.t…shkeys@.service template instances.
         Starting console-login-helper-mess…ia console-login-helper-messages...
         Starting coreos-check-wireless-fir…k For Wireless Firmware Packages...
         Starting coreos-ignition-write-iss…eate Ignition Status Issue Files...
         Starting coreos-platform-chrony-co…ure Chrony Based On The Platform...
         Starting dracut-shutdown.service -…store /run/initramfs on shutdown...
         Starting k3s.service - Lightweight Kubernetes...
         Starting ssh-host-keys-migration.s…ate OpenSSH host key permissions...
[  OK  ] Reached target sshd-keygen.target.
         Starting console-login-helper-mess…ia console-login-helper-messages...
[  OK  ] Reached target nss-user-lookup.target - User and Group Name Lookups.
         Starting systemd-homed.service - Home Area Manager...
         Starting systemd-logind.service - User Login Management...
[  OK  ] Finished dracut-shutdown.service - Restore /run/initramfs on shutdown.
[FAILED] Failed to start console-login-help… via console-login-helper-messages.
See 'systemctl status console-login-hel…nippet-os-release.service' for details.
[FAILED] Failed to start systemd-homed.service - Home Area Manager.

More logs from journalctl about NetworkManager.service:

May 05 05:46:43 fc41dev audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dbus-broker comm="systemd" ex
e="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
May 05 05:46:43 fc41dev systemd[1]: Dependency failed for NetworkManager.service - Network Manager.
May 05 05:46:43 fc41dev systemd[1]: Dependency failed for nmstate.service - Apply nmstate on-disk state.
May 05 05:46:43 fc41dev systemd[1]: nmstate.service: Job nmstate.service/start failed with result 'dependency'.
May 05 05:46:43 fc41dev systemd[1]: Dependency failed for NetworkManager-wait-online.service - Network Manager Wait Online.
May 05 05:46:43 fc41dev systemd[1]: NetworkManager-wait-online.service: Job NetworkManager-wait-online.service/start failed with result 'dependency'.
May 05 05:46:43 fc41dev systemd[1]: NetworkManager.service: Job NetworkManager.service/start failed with result 'dependency'.

I’m unable login to console - both in virt-manager and ssh, login service also failed:

journalctl --since '06:56:39' -u systemd-logind
May 05 06:56:39 fc41dev systemd[1]: Starting systemd-logind.service - User Login Management...
May 05 06:56:39 fc41dev (d-logind)[1001]: Failed to initialize SELinux labeling handle: No such file or directory
May 05 06:56:39 fc41dev systemd-logind[1001]: Failed to initialize SELinux labeling handle: No such file or directory
May 05 06:56:39 fc41dev systemd-logind[1001]: Failed to connect to system bus: Connection refused
May 05 06:56:39 fc41dev systemd-logind[1001]: Failed to fully start up daemon: Connection refused
May 05 06:56:39 fc41dev systemd[1]: systemd-logind.service: Main process exited, code=exited, status=1/FAILURE
May 05 06:56:39 fc41dev systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
May 05 06:56:39 fc41dev systemd[1]: Failed to start systemd-logind.service - User Login Management.
May 05 06:56:39 fc41dev systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 1.
May 05 06:56:39 fc41dev systemd[1]: Starting systemd-logind.service - User Login Management...
May 05 06:56:39 fc41dev (d-logind)[1080]: Failed to initialize SELinux labeling handle: No such file or directory
May 05 06:56:40 fc41dev systemd-logind[1080]: Failed to initialize SELinux labeling handle: No such file or directory
May 05 06:56:40 fc41dev systemd-logind[1080]: Watching system buttons on /dev/input/event0 (Power Button)
May 05 06:56:40 fc41dev systemd-logind[1080]: Watching system buttons on /dev/input/event1 (AT Translated Set 2 keyboard)
May 05 06:56:40 fc41dev systemd-logind[1080]: New seat seat0.
May 05 06:56:40 fc41dev systemd[1]: Started systemd-logind.service - User Login Management.
-- Boot 4a0bc7dd61fb4fd7805951f98fecb78b --
May 05 07:01:12 fc41dev systemd[1]: Starting systemd-logind.service - User Login Management...
May 05 07:01:13 fc41dev systemd-logind[1034]: Watching system buttons on /dev/input/event0 (Power Button)
May 05 07:01:13 fc41dev systemd-logind[1034]: Watching system buttons on /dev/input/event1 (AT Translated Set 2 keyboard)
May 05 07:01:13 fc41dev systemd-logind[1034]: New seat seat0.
May 05 07:01:13 fc41dev systemd[1]: Started systemd-logind.service - User Login Management.
May 05 07:01:25 fc41dev systemd-logind[1034]: New session 1 of user root.
May 05 07:01:35 fc41dev systemd-logind[1034]: Removed session 1.
May 05 07:25:38 fc41dev systemd-logind[1034]: New session 2 of user core.
May 05 07:25:38 fc41dev systemd-logind[1034]: New session 3 of user core.

dustymabe · May 5, 2025, 6:57pm

The Failed to initialize SELinux labeling handle: No such file or directory is certainly interesting.

Any chance you could try to reproduce this on:

a newly provisioned system that is then upgraded?
- You can get older boot media from the unofficial FCOS builds browser.
a newly provisioned F42 system (non-upgraded)?

If we can get a reproducer it will be much easier to triage/fix.

ophil · May 5, 2025, 9:18pm

Thanks a lot Dusty, you help me to track down the guilty.

I checked the same upgrade on other VM, not new, but slightly cleaner - no problems with upgrade from 41.20250331.3.0 to 42.20250410.3.2 . Then I recall one change to rpm-ostree I made manually - I installed the k3s by recommended script, it detected correctly coreos and even installed k3s-selinux.rpm, see GitHub - k3s-io/k3s-selinux: SELinux policy for k3s. After some time I rolled back this package from rpm-ostree - found workaround to run k3s without it. However, for some time my setup was marked with this package added.

So, if you want to reproduce this error, install package k3s-selinux.rpm and then rollback this commit, or just check install script provided by https://k3s.io/

However, before latest upgrade the same setup worked correctly:

$ journalctl -u zincati|grep 'update applied'
Apr 03 15:26:58 fc41dev zincati[1625]: [INFO  zincati::update_agent::actor] update applied, waiting for reboot: 41.20250315.3.0
Apr 17 08:18:44 fc41dev zincati[1801]: [INFO  zincati::update_agent::actor] update applied, waiting for reboot: 41.20250331.3.0
May 02 08:51:31 fc41dev zincati[1790]: [INFO  zincati::update_agent::actor] update applied, waiting for reboot: 42.20250410.3.2

I can provide the zincati log on failed server, after today upgrade it full of errors

dustymabe · May 5, 2025, 10:30pm

I imagine the policy in k3s-selinux/policy/coreos at master · k3s-io/k3s-selinux · GitHub needs to be updated? Maybe open an issue against that repo to see if they have hit this problem before?

vwbusguy · May 5, 2025, 11:39pm

Yup - I had it happen once on CentOS Stream 9 running k3s once as well. Fedora and CentOS aren’t technically supported distros for k3s, but probably still something they should know about through their community issues.

ophil · May 6, 2025, 6:01am

Yes, checked k3s-selinux repo, seems to be outdated and hardly active.
Nevertheless , still some issues with rpm-ostree. As me said, I rolled back subj. package commit, technically, that would wipe out all changes. Since then 3 updates run correctly. Errors happened only after major upd. to fc42.

Found some info in rpm-ostreed.service log. First, fragment where additional package k3s-selinux-1.6-1.coreos.noarch.rpm installed

Mar 27 08:38:23 fc41dev rpm-ostree[1784]: Downloading: https://rpm.rancher.io/k3s/stable/common/coreos/noarch/k3s-selinux-1.6-1.coreos.noarch.rpm
Mar 27 08:38:24 fc41dev rpm-ostree[1784]: Imported 1 pkg
Mar 27 08:38:24 fc41dev rpm-ostree[1784]: Executed %prein for k3s-selinux in 107 ms
Mar 27 08:38:34 fc41dev rpm-ostree[1784]: Executed %post for k3s-selinux in 10485 ms
Mar 27 08:38:34 fc41dev rpm-ostree[1784]: Executed %posttrans for k3s-selinux in 58 ms
Mar 27 08:38:34 fc41dev rpm-ostree[1784]: No files matched %transfiletriggerin(lib) for glibc-common
Mar 27 08:38:34 fc41dev rpm-ostree[1784]: No files matched %transfiletriggerin(lib64) for glibc-common
Mar 27 08:38:35 fc41dev rpm-ostree[1784]: Executed %transfiletriggerin(glibc-common) for lib, lib64, usr/lib, usr/lib64 in 893 ms; 13115 matched files
Mar 27 08:38:35 fc41dev rpm-ostree[1784]: No files matched %transfiletriggerin(usr/lib64/gio/modules) for glib2
Mar 27 08:38:35 fc41dev rpm-ostree[1784]: No files matched %transfiletriggerin(usr/share/glib-2.0/schemas) for glib2
Mar 27 08:38:35 fc41dev rpm-ostree[1784]: Executed %transfiletriggerin(shared-mime-info) for usr/share/mime in 81 ms; 908 matched files
Mar 27 08:38:36 fc41dev rpm-ostree[1784]: Executed %transfiletriggerin(systemd-udev) for usr/lib/udev/hwdb.d in 1043 ms; 32 matched files
Mar 27 08:38:37 fc41dev rpm-ostree[1784]: Executed %transfiletriggerin(systemd-udev) for usr/lib/udev/rules.d in 107 ms; 78 matched files
Mar 27 08:38:37 fc41dev rpm-ostree[1784]: sanitycheck(/usr/bin/true) successful
Mar 27 08:38:37 fc41dev rpm-ostree[1784]: Regenerating rpmdb for target
Mar 27 08:38:54 fc41dev rpm-ostree[1784]: Wrote commit: c1bffd5c88abbcda2d45412aaf82be2f78fcb7de8b0e32f15ab76e5c2afc1ef2; New objects: meta:24 content:14 totaling 40.2 MB)
Mar 27 08:38:57 fc41dev rpm-ostree[1784]: Created deployment; subtasks: checkout=1.1s composefs=917ms etc=933ms
Mar 27 08:38:57 fc41dev rpm-ostree[1784]: Created new deployment /ostree/deploy/fedora-coreos/deploy/c1bffd5c88abbcda2d45412aaf82be2f78fcb7de8b0e32f15ab76e5c2afc1ef2.0
Mar 27 08:38:58 fc41dev rpm-ostree[1784]: Txn PkgChange on /org/projectatomic/rpmostree1/fedora_coreos successful
Mar 27 08:38:58 fc41dev rpm-ostree[1784]: Unlocked sysroot
Mar 27 08:38:58 fc41dev rpm-ostree[1784]: Process [pid: 1780 uid: 0 unit: session-2.scope] disconnected from transaction progress
Mar 27 08:38:58 fc41dev rpm-ostree[1784]: client(id:cli dbus:1.31 unit:session-2.scope uid:0) vanished; remaining=0
Mar 27 08:38:58 fc41dev rpm-ostree[1784]: In idle state; will auto-exit in 64 seconds
Mar 27 08:39:12 fc41dev rpm-ostree[1784]: Allowing active client :1.33 (uid 1000)
Mar 27 08:39:12 fc41dev rpm-ostree[1784]: client(id:cli dbus:1.33 unit:session-2.scope uid:1000) added; new total=1
Mar 27 08:39:12 fc41dev rpm-ostree[1784]: client(id:cli dbus:1.33 unit:session-2.scope uid:1000) vanished; remaining=0
Mar 27 08:39:12 fc41dev rpm-ostree[1784]: In idle state; will auto-exit in 60 seconds
Mar 27 08:40:11 fc41dev rpm-ostree[1784]: In idle state; will auto-exit in 64 seconds
Mar 27 08:40:11 fc41dev systemd[1]: rpm-ostreed.service: Deactivated successfully.
Mar 27 08:40:11 fc41dev systemd[1]: rpm-ostreed.service: Consumed 51.265s CPU time, 886.2M memory peak.

Another where I’d rolled it back:

Mar 27 11:02:14 fc41dev rpm-ostree[5746]: Initiated txn Rollback for client(id:cli dbus:1.100 unit:session-2.scope uid:0): /org/projectatomic/rpmostree1/fedora_coreos
Mar 27 11:02:14 fc41dev rpm-ostree[5746]: Process [pid: 5886 uid: 0 unit: session-2.scope] connected to transaction progress
Mar 27 11:02:14 fc41dev rpm-ostree[5746]: Txn Rollback on /org/projectatomic/rpmostree1/fedora_coreos failed: Staged deployment (remove with cleanup -p)
Mar 27 11:02:14 fc41dev rpm-ostree[5746]: Unlocked sysroot
Mar 27 11:02:14 fc41dev rpm-ostree[5746]: Process [pid: 5886 uid: 0 unit: session-2.scope] disconnected from transaction progress
Mar 27 11:02:14 fc41dev rpm-ostree[5746]: client(id:cli dbus:1.100 unit:session-2.scope uid:0) vanished; remaining=0
Mar 27 11:02:14 fc41dev rpm-ostree[5746]: In idle state; will auto-exit in 63 seconds
Mar 27 11:03:12 fc41dev rpm-ostree[5746]: client(id:cli dbus:1.105 unit:session-2.scope uid:0) added; new total=1
Mar 27 11:03:12 fc41dev rpm-ostree[5746]: Loaded sysroot
Mar 27 11:03:12 fc41dev rpm-ostree[5746]: Locked sysroot
Mar 27 11:03:12 fc41dev rpm-ostree[5746]: Initiated txn Rollback for client(id:cli dbus:1.105 unit:session-2.scope uid:0): /org/projectatomic/rpmostree1/fedora_coreos
Mar 27 11:03:12 fc41dev rpm-ostree[5746]: Process [pid: 5955 uid: 0 unit: session-2.scope] connected to transaction progress
Mar 27 11:03:12 fc41dev rpm-ostree[5746]: Txn Rollback on /org/projectatomic/rpmostree1/fedora_coreos failed: Staged deployment (remove with cleanup -p)
Mar 27 11:03:12 fc41dev rpm-ostree[5746]: Unlocked sysroot

see 2 messages Txn Rollback on /org/projectatomic/rpmostree1/fedora_coreos failed: Staged deployment (remove with cleanup -p)

vwbusguy · May 6, 2025, 8:58pm

IIRC, it’s really only targeting their supported distros (so SLEMicro, RHEL, Rocky, but not CentOS Stream), which also don’t require a lot of changes, but that also means it’s unfortunately more likely to break for Fedora policies.

You could either put it in permissive and use audit2allow to fix it or disable SELinux on those hosts. k3s will run without selinux enabled, but of course it’s better to not disable it, especially on internet facing hosts.

ophil · May 7, 2025, 4:39am

Thanks Scott, this is what I done. The problem now not in k3s, but why rmp-ostree rollback failed to restore the clean state of previous commit.

vwbusguy · May 8, 2025, 12:46am

It should for the things that are immutable. However, stuff under /var/lib/rancher/k3s is not immutable. I suggest using btrfs snapshots if you want to revert the filesystem to a point in time (as well as data store backups for k3s).

dustymabe · May 8, 2025, 7:54pm

rpm-ostree rollback will revert changes in /etc/ (i.e. configuration files) but not data in /var/

Topic		Replies	Views
Unable to Upgrade to 35.20211029.3.0 Project Discussion coreos-wg	17	1392	January 22, 2022
CoreOs fails to boot with composefs after upgrade to 41 Ask Fedora coreos	12	224	January 24, 2025
Upgrade LXQt 41 -> 42 Failed ### SOS! Ask Fedora	4	226	April 22, 2025
F41 Error updating kernel-core package Ask Fedora f41	3	296	February 4, 2025
Can't check signature: public key not found Ask Fedora coreos	2	1933	June 1, 2023

Fedora 42 upgrade - failed to login

Related topics