Incredibly long sudo delays, lockscreen freezes because of failing systemd units

Related: Fixing slow sudo (failed systemd services)

I’m having the exact same problem with pam as well, and this is really frustrating, because it really affects a LOT of aspects of the system, and contrary to that post, restarting the units do not fix the problem at all.

First of all, the obvious ones include the 30s sudo authentication delay and 30s freezes on the lock screen. Additionally, gnome-calendar will not launch anymore because systemd-timedated.service is failing, gnome-control-center will also lock up for 30s if I click on some tabs such as “Sharing”, gnome’s dock and application menu will stay blank for 30s on startup as well.

This is a fresh F32 installation that is only 3 weeks old, I don’t know why it’s already breaking down. I’m pretty sure this is a selinux related problem but I have no experience with selinux, so I don’t know where should I begin to start the troubleshooting process. Please let me know if you need more info.

$ systemctl list-units --state=failed
  UNIT                      LOAD   ACTIVE SUB    DESCRIPTION                      
● fprintd.service           loaded failed failed Fingerprint Authentication Daemon
● systemd-timedated.service loaded failed failed Time & Date Service 

$ sudo systemctl status systemd-timedated.service
● systemd-timedated.service - Time & Date Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-timedated.service; static; vendor preset: disabled)
     Active: failed (Result: exit-code) since 4min 42s ago
       Docs: man:systemd-timedated.service(8)
    Process: 11314 ExecStart=/usr/lib/systemd/systemd-timedated (code=exited, status=226/NAMESPACE)
   Main PID: 11314 (code=exited, status=226/NAMESPACE)
        CPU: 10ms

systemd[1]: Starting Time & Date Service...
systemd[11314]: systemd-timedated.service: Failed to set up mount namespacing: /run/systemd/unit-root/: Permission denied
systemd[11314]: systemd-timedated.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-timedated: Permission denied
systemd[1]: systemd-timedated.service: Main process exited, code=exited, status=226/NAMESPACE
systemd[1]: systemd-timedated.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Time & Date Service.

$ sudo systemctl status fprintd.service
● fprintd.service - Fingerprint Authentication Daemon
     Loaded: loaded (/usr/lib/systemd/system/fprintd.service; static; vendor preset: disabled)
     Active: failed (Result: exit-code) since 5min ago
       Docs: man:fprintd(1)
    Process: 11285 ExecStart=/usr/libexec/fprintd (code=exited, status=226/NAMESPACE)
   Main PID: 11285 (code=exited, status=226/NAMESPACE)
        CPU: 7ms

systemd[1]: Starting Fingerprint Authentication Daemon...
systemd[1]: fprintd.service: Main process exited, code=exited, status=226/NAMESPACE
systemd[1]: fprintd.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Fingerprint Authentication Daemon.

Welcome to the forum! Please read the “Getting Started” topic to be up to date on how to best get help and help others here.

You said this is a new install so:

  1. Have you done a full update since the install? If not then please do so.
    “sudo dnf update -y” will handle that.

If the problem is still there then try the steps below.

  1. If you suspect selinux then the first thing I would suggest is to do is check that out.
    a) Check the status of selinux. “sudo getenforce” will return the current status, which should be one of enforcing, permissive, or disabled
    If that result is “enforcing” you can temporarily disable enforcing to see if that changes anything by doing “sudo setenforce 0” then see if there are changes in the system response. The next reboot will reset selinux to the default enforcing type as defined in /etc/selinux/config.
    b) try relabeling the system in case selinux has something to do with causing the hangs. “sudo touch /.autorelabel” followed by a reboot will handle that.
    c) To change the default selinux setting you can change the value in /etc/selinux/config to which ever type you want, enforcing, permissive, or disabled.

For detailed information on selinux and how to manage it you can look at the man pages for selinux, setenforce, getenforce, and others that will tell you how to use them.

1 Like
  1. I just did that and rebooted right before I wrote the post to make sure it wasn’t something that has already been fixed, so I’m sure everything is up to date.

  2. Switching selinux to permissive made the two failing systemd services able to be started once again, which got rid of the sudo and lockscreen delays as expected, and gnome-calendar works once again. However, gnome-control-center is still hanging if I click on the “Sharing” tab, and I haven’t tested the blank dock/application menu because I haven’t rebooted yet.
    Additionally, I wasn’t aware of the /.autorelabel thing before, so I simply tried reinstalling selinux-policy-targeted, which did not work. I’ll reboot in a bit and see if relabelling the fs (with /.autorelabel placed) fixes the problem.

The filesystem has been relabeled but the situation did not improve, I’m seeing 3 failing units now:

$ systemctl list-units --state=failed
  UNIT                      LOAD   ACTIVE SUB    DESCRIPTION                      
● fprintd.service           loaded failed failed Fingerprint Authentication Daemon
● systemd-hostnamed.service loaded failed failed Hostname Service                 
● systemd-timedated.service loaded failed failed Time & Date Service       

After setting selinux back to permissive, and restarting the 3 services, everything works once again. I really don’t want to disable selinux permanently, is there anything else I can try other than reinstalling?

If you set selinux to permissive it will still give you warnings about errors. That would at least allow the system to operate while you identify the cause.

There is the possibility that the install somehow caused an error that is triggering selinux to halt the services and a reinstall could fix that.
There is also the possibility that a disk error caused it, and you might be able to test for that using smartctl and look for errors already reported or run a test yourself just in case.
Having a new service that is failing causes me to suspect some form of filesystem corruption that may be growing…

What is the output of grep AVC /var/log/audit/audit.log ?

This seems to be a common problem, see this:

A common cause seems issues around /etc/hosts and hostname…

1 Like

This is what I get after running sudo ausearch -i -m avc:

type=AVC msg=audit(09/17/2020 00:00:52.830:319) : avc:  denied  { remount } for  pid=5399 comm=(fprintd) scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=filesystem permissive=0 
type=AVC msg=audit(09/17/2020 00:01:11.118:323) : avc:  denied  { remount } for  pid=5493 comm=(ostnamed) scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=filesystem permissive=0 
type=AVC msg=audit(09/17/2020 00:01:49.483:330) : avc:  denied  { remount } for  pid=5557 comm=(imedated) scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=filesystem permissive=0 
type=AVC msg=audit(09/17/2020 00:02:14.667:338) : avc:  denied  { remount } for  pid=5603 comm=(coredump) scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=filesystem permissive=0 
type=AVC msg=audit(09/17/2020 00:05:01.550:371) : avc:  denied  { remount } for  pid=6001 comm=(ostnamed) scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=filesystem permissive=1 
type=AVC msg=audit(09/17/2020 00:05:54.659:390) : avc:  denied  { remount } for  pid=6147 comm=(imedated) scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=filesystem permissive=1 

This case is different because setting selinux to permissive resolves the problem, so I’m pretty sure it’s selinux related.

Additionally, I did check the SMART and there are no errors. I don’t have any disk/fs related problems in the kernel log buffer either. Since my / is on btrfs, I also did a scrub, and that returned clean as well.

Update: the problem was mysteriously fixed after a system update.
Here’s a list of packages which were updated in this particular transaction:

 System time: 
 Succeeded: True
 Role: update-packages
 Duration: 81828 (seconds)
 Command line: 
 User ID: 0
 Username: root
 Real name: root
 Affected packages:
 - updating grub2-common-1:2.04-23.fc32.noarch
 - updating libxml2-2.9.10-7.fc32.x86_64
 - updating grub2-tools-minimal-1:2.04-23.fc32.x86_64
 - updating grub2-tools-1:2.04-23.fc32.x86_64
 - installing kernel-core-5.8.9-200.fc32.x86_64
 - installing kernel-modules-5.8.9-200.fc32.x86_64
 - installing sane-airscan-0.99.16-1.fc32.x86_64
 - updating sane-backends-drivers-cameras-1.0.31-2.fc32.x86_64
 - updating sane-backends-libs-1.0.31-2.fc32.x86_64
 - updating sane-backends-1.0.31-2.fc32.x86_64
 - updating sane-backends-drivers-scanners-1.0.31-2.fc32.x86_64
 - updating grub2-pc-modules-1:2.04-23.fc32.noarch
 - updating pipewire-libs-0.3.11-2.fc32.x86_64
 - updating pipewire-0.3.11-2.fc32.x86_64
 - updating gtk-update-icon-cache-3.24.23-1.fc32.x86_64
 - updating gtk3-3.24.23-1.fc32.x86_64
 - updating inkscape-1.0.1-2.fc32.x86_64
 - updating code-1.49.1-1600299354.el7.x86_64
 - updating grub2-pc-1:2.04-23.fc32.x86_64
 - installing kernel-5.8.9-200.fc32.x86_64
 - installing kernel-modules-extra-5.8.9-200.fc32.x86_64
 - updating grub2-efi-ia32-1:2.04-23.fc32.x86_64
 - updating grub2-efi-x64-1:2.04-23.fc32.x86_64
 - updating grub2-tools-extra-1:2.04-23.fc32.x86_64
 - updating python3-libxml2-2.9.10-7.fc32.x86_64
 - updating grub2-efi-ia32-cdboot-1:2.04-23.fc32.x86_64
 - updating grub2-efi-x64-cdboot-1:2.04-23.fc32.x86_64
 - updating grub2-tools-efi-1:2.04-23.fc32.x86_64
 - updating sudo-1.9.2-1.fc32.x86_64
 - updating container-selinux-2:2.145.0-1.fc32.noarch
 - installing kernel-devel-5.8.9-200.fc32.x86_64
 - updating libxml2-2.9.10-7.fc32.i686
 - updating sane-backends-drivers-cameras-1.0.31-2.fc32.i686
 - updating sane-backends-libs-1.0.31-2.fc32.i686
 - updating sane-backends-drivers-scanners-1.0.31-2.fc32.i686
 - cleanup sane-backends-drivers-scanners-1.0.31-1.fc32.i686
 - cleanup sane-backends-libs-1.0.31-1.fc32.i686
 - cleanup sane-backends-drivers-cameras-1.0.31-1.fc32.i686
 - cleanup grub2-pc-1:2.04-22.fc32.x86_64
 - cleanup grub2-efi-x64-1:2.04-22.fc32.x86_64
 - cleanup grub2-efi-ia32-1:2.04-22.fc32.x86_64
 - cleanup kernel-5.8.6-201.fc32.x86_64
 - cleanup grub2-pc-modules-1:2.04-22.fc32.noarch
 - removing kmod-VirtualBox-5.8.6-201.fc32.x86_64-6.1.14-1.fc32.x86_64
 - cleanup grub2-efi-x64-cdboot-1:2.04-22.fc32.x86_64
 - cleanup grub2-efi-ia32-cdboot-1:2.04-22.fc32.x86_64
 - cleanup libxml2-2.9.10-3.fc32.i686
 - cleanup kernel-devel-5.8.6-201.fc32.x86_64
 - cleanup container-selinux-2:2.144.0-3.fc32.noarch
 - cleanup python3-libxml2-2.9.10-3.fc32.x86_64
 - cleanup sane-backends-1.0.31-1.fc32.x86_64
 - cleanup inkscape-1.0-2.fc32.x86_64
 - cleanup kernel-modules-extra-5.8.6-201.fc32.x86_64
 - cleanup pipewire-0.3.10-1.fc32.x86_64
 - cleanup grub2-tools-extra-1:2.04-22.fc32.x86_64
 - cleanup grub2-tools-minimal-1:2.04-22.fc32.x86_64
 - cleanup kernel-modules-5.8.6-201.fc32.x86_64
 - cleanup grub2-tools-1:2.04-22.fc32.x86_64
 - cleanup code-1.49.0-1599744701.el7.x86_64
 - cleanup gtk3-3.24.22-1.fc32.x86_64
 - cleanup grub2-tools-efi-1:2.04-22.fc32.x86_64
 - cleanup grub2-common-1:2.04-22.fc32.noarch
 - cleanup gtk-update-icon-cache-3.24.22-1.fc32.x86_64
 - cleanup kernel-core-5.8.6-201.fc32.x86_64
 - cleanup pipewire-libs-0.3.10-1.fc32.x86_64
 - cleanup sudo-1.9.0-0.1.b4.fc32.x86_64

Update: the units have started failing once again on next reboot. I did dnf reinstall *selinux* and another autorelabel, now it’s working fine again. I’m pretty sure units will start failing once again on next reboot.

If your problems continue you might look at several of the helps that can be found by a quick google search for “selinux troubleshooter fedora” this is one that popped up for me. As you know you can see the errors posted by selinux by using the cli command

 sudo ausearch -m AVC,USER_AVC,SELINUX_ERR -ts recent 

Those errors will lead you to the specific problem if you read them but other than telling you what the error is they are not a lot of help.

I find selinux tools that can be very helpful are the troubleshooter and toolbox. “sudo dnf install setroubleshoot setools”.
These are tools that will help you identify and fix problems related to selinux. If you are using the gnome desktop and have both those installed they should give you alerts when they occur, and suggest how to fix the problem.

This is more detailed info on how to troubleshoot selinux and tools to help.

Something is wrong in your file system if you have files with the unlabled_t type. In this blog post Dan Walsh discusses a number of possible causes. I don’t know if any might apply to you.

Unfortunately, your AVC:s don’t mention more exactly what it is that is unlabeled. When I’ve seen it previously, the message has typically included a path or inode that could be used to find the problem. I’m not sure why you don’t see it. A brute force method would be

sudo find / -context '*unlabeled_t*'

Run this in when in enforcing mode, so the mount will actually fail.

1 Like

I set selinux back to enforcing, restarted all 3 aforementioned services, but they don’t seem to be failing anymore. I tried looking for unlabelled files under / but didn’t find any:

$ sudo find / -context '*unlabeled_t*' -not -path "/run/media/*"
find: ‘/proc/18188/task/18188/net’: Invalid argument
----- further entries under /proc removed -----
find: ‘/run/user/1000/doc’: Permission denied
find: ‘/run/user/1000/gvfs’: Permission denied

This problem is incredibly intermittent and I still have not been able to pin it down. I have set selinux to permissive for now because I don’t really have enough time to waste on this at the moment.

The only thing I ever have mounted under /run/media are USB devices (files and actual media) and those are mounted as /run/media/myusername/devicename

It may easily be that those errors occur when you have a USB device plugged in and mounted that does not have the correct selinux context labeling.

1 Like

Problem here is that anything under /run/media should not prevent the aforementioned 3 services from starting at all. These are mounts that typically a DE would mount drives to. My files on these removable drives certainly do not have the correct selinux context, but why should they? I’m not running any binaries or having any system services depend on my files on there.

Having binaries or system services dependent upon the files has no relevance to the need for selinux context. Simply look at the files in and under your user home directory with “ls -lZ” and you will see what I mean. I do not think the DE handles the mounting there, but rather I think systemd is involved in that.

Selinux enforces the security context on all files possible on all accessible devices and warns you or prevents access if the context is not correct. It obviously cannot write the context to files in vfat or ntfs formatted filesystems, but it can warn or prevent access there dependent upon the policy in effect.

Try booting with the USB devices unplugged; then, after you have signed in, attach them and see if the same errors occur. It seems possible that selinux may be interfering before it fully initializes the system if it cannot properly read the device during startup.

Wow, I really wasn’t expecting that but when I removed the USB SD card reader I had plugged in and rebooted, it all started working again. I was having the exact same failed units as @ahrb but now settings/lock screen/sudo are back to normal speed.