Unable to log in after last fedora 41 update

I had done an update to my system 2 days ago but not rebooted or logged out/in.
Today we had a power outage and of course that forced a restart.

When I attempt to log in I get a failed authentication on the plymouth login screen.

I tried using a text screen to log in and can do so as root, but not as my regular user.
My regular user gets the authentication failure message when trying to log in that way.

I tried creating a new user with adduser and that user also gets the authentication failure message when trying to log in.

I have tried resetting my password but there is no change in the issue.
I also performed a restorecon -R / to reset selinux context on the entire system and have tried setting selinux to permissive with setenforce 0 but that also has had no effect.

Does anyone have an idea where I may look for changes that would prevent a regular user from logging in but still allow root to connect via text screen login?

Do you have way to see you have user files still?

I had on openSUSE some month ago where updates somehow deleted user/home those where never there and only way to get back was reinstall

Not sure if this is helpful or even related, but that happened multiple times in gnome

Even I had snapshots but not single snapshot allowed to login only root was working

You happen to be systemd boot btw?

Long shot but maybe

There may be some hints in journalctl such as fsck running for a long time on first boot after the power hit.

Living in an area prone to hurricanes and winter storms, a UPS has largely eliminated unsafe shutdowns, so all my practical experience with unsafe shutdowns is in the distant past. I do recall having to restore the last full backup followed by nearly a month of daily incremental backups, all on a robotic tape library with a constant stream of users asking when the system would be back.

Do the users run the same shell program as root? Is there garbage in any of /etc/passwd, /etc/shadow, etc. root’s entries are usually at the start of these files, so might have escaped corruption affecting the entries near the end of the file.

No corruption seen on any of the files
I watched the boot process and it successfully ran fsck during the startup
Just for assurance I repeated the fsck on /home with fsck -f and no errors are reported.

I also cannot ssh into or out of that system with either password or my usual key.

The only clue I can find in journalctl I am unable to cut&paste since I have no way to get the data to my other system.

I have selinux in permissive mode and I see an audit message that reads something like msg='op=PAM:accounting grantors=? acct=ME exe="/usr/bin/login" hostname=.... adr=? terminal=/dev/tty4 res=failed'
The line immediately before is similar with op=PAM:authentication grantors=pam_unix with res=success
and the line following is from login with the authentication failure

“grantors=?” is suspect. I have “grantors=pam_unix,pam_localuser”.

That could be something related the /etc/pam.d, in particular ls -dl /etc/pam.d/system-auth which normally would be a symbolic link

  ls -dl /etc/pam.d/system-auth
lrwxrwxrwx. 1 root root 27 Oct 22 19:21 /etc/pam.d/system-auth -> /etc/authselect/system-auth

and the content of the target

cat /etc/authselect/system-auth
# Generated by authselect
# Do not modify this file manually, use authselect instead. Any user changes will be overwritten.
# You can stop authselect from managing your configuration by calling 'authselect opt-out'.
# See authselect(8) for more details.

auth        required                                     pam_env.so
auth        required                                     pam_faildelay.so delay=2000000
auth        [default=1 ignore=ignore success=ok]         pam_usertype.so isregular
auth        [default=1 ignore=ignore success=ok]         pam_localuser.so
auth        sufficient                                   pam_unix.so
auth        [default=1 ignore=ignore success=ok]         pam_usertype.so isregular
auth        sufficient                                   pam_sss.so forward_pass
auth        required                                     pam_deny.so

account     required                                     pam_unix.so
account     sufficient                                   pam_localuser.so
account     sufficient                                   pam_usertype.so issystem
account     [default=bad success=ok user_unknown=ignore] pam_sss.so
account     required                                     pam_permit.so

password    requisite                                    pam_pwquality.so local_users_only
password    sufficient                                   pam_unix.so yescrypt shadow  use_authtok
password    [success=1 default=ignore]                   pam_localuser.so
password    sufficient                                   pam_sss.so use_authtok
password    required                                     pam_deny.so

session     optional                                     pam_keyinit.so revoke
session     required                                     pam_limits.so
-session    optional                                     pam_systemd.so
session     optional                                     pam_oddjob_mkhomedir.so
session     [success=1 default=ignore]                   pam_succeed_if.so service in crond quiet use_uid
session     required                                     pam_unix.so
session     optional                                     pam_sss.so

This appears to only be affecting authentication for regular users, both existing and new.

Once logged in as root I can also switch to my regular user with su but cannot get the gui to run.

That file is much smaller than what you show here.
I wonder why and how it may have gotten modified.

My other system that still works has the file /etc/authselect/system-auth that matches your post.

A reinstall of the authselect-libs package installs only the limited file that I currently have.

The one that fails has

auth        required                                     pam_env.so
auth        required                                     pam_faildelay.so delay=2000000
auth        sufficient                                   pam_unix.so
auth        required                                     pam_deny.so

account     required                                     pam_unix.so

password    requisite                                    pam_pwquality.so 
password    sufficient                                   pam_unix.so yescrypt shadow  use_authtok
password    required                                     pam_deny.so

session     optional                                     pam_keyinit.so revoke
session     required                                     pam_limits.so
-session    optional                                     pam_systemd.so
session     [success=1 default=ignore]                   pam_succeed_if.so service in crond quiet use_uid
session     required                                     pam_unix.so

and nothing else.

That one also looks fine. It is the “local” profile, which you can verify with authselect current. Previous Fedora releases would select “sssd” as the default profile.

Your /etc/authselect/system-auth looks like what I have on 2 desktop systems. I recall some changes related to pam a couple moths ago, but won’t have time to investigate for a while.

Edit: I was thinking of migrating to the new authselect local profile.
I have:

% authselect current                                         
Profile ID: local
Enabled features:
- with-silent-lastlog
- with-mdns4

The date shown with `ls -ld /etc/pam.d/system-auth is different than yours. I have Nov 14 while you show Oct 22.

I did updates on Nov 15 and Nov 17 and a distro-sync on Nov 11
I had rebooted on Nov 15 with no problems until the power outage today. So the suspect is something that was changed with my update on Nov 17.
I cannot provide the list of packages updated as text until I am able to use ssh/rsync to transfer the data.

But this is the list in image form

It seems pretty obvious that something in that update caused this problem but as yet I cannot identify it. I won’t be upgrading my other 2 systems until after this problem is resolved.

That is the time when the authselect was asked to update or change the profile.
This also happens when authselect-libs is (re)installed or updated.

See rpm -q --scripts authselect-libs and the last line is

/usr/bin/authselect apply-changes &> /dev/null

which will create or recreate the symbolic links in /etc/pam.d.

Still no progress.
I have done a full dnf reinstall * after doing a dnf distro-sync as well as switching authselect between the sssd and local profiles yet any regular user is prevented from login.

ssh and rsync are prevented from operation as well, probably a result of the same issue.

This is the only related data I can find in the logs

Nov 19 08:35:03  audit[2872]: USER_AUTH pid=2872 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_usertype,pam_localuser,pam_unix acct="jeff" exe="/usr/bin/login" hostname=eagle.home.domain addr=? terminal=/dev/tty4 res=success'
Nov 19 08:35:03  audit[2872]: USER_ACCT pid=2872 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=? acct="jeff" exe="/usr/bin/login" hostname=eagle.home.domain addr=? terminal=/dev/tty4 res=failed'
Nov 19 08:35:35  audit[2891]: USER_AUTH pid=2891 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_usertype,pam_localuser,pam_unix acct="jeff" exe="/usr/bin/login" hostname=eagle.home.domain addr=? terminal=/dev/tty4 res=success'
Nov 19 08:35:35 audit[2891]: USER_ACCT pid=2891 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=? acct="jeff" exe="/usr/bin/login" hostname=eagle.home.domain addr=? terminal=/dev/tty4 res=failed'

I also tried to connect from another system using ssh and this is the log.

Nov 19 09:15:59 audit[3233]: CRYPTO_KEY_USER pid=3233 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:XXXXXXXXX direction=? spid=3233 suid=0  exe="/usr/libexec/openssh/sshd-session" hostname=? addr=192.168.4.113 terminal=? res=success'
Nov 19 09:15:59 audit[3233]: CRYPTO_KEY_USER pid=3233 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:XXXXXXXXX direction=? spid=3233 suid=0  exe="/usr/libexec/openssh/sshd-session" hostname=? addr=192.168.4.113 terminal=? res=success'
Nov 19 09:15:59 audit[3233]: CRYPTO_KEY_USER pid=3233 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:XXXXXXXX direction=? spid=3233 suid=0  exe="/usr/libexec/openssh/sshd-session" hostname=? addr=192.168.4.113 terminal=? res=success'
Nov 19 09:15:59 audit[3232]: CRYPTO_SESSION pid=3232 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-server cipher=aes256-gcm@openssh.com ksize=256 mac=<implicit> pfs=curve25519-sha256 spid=3233 suid=74 rport=58590 laddr=192.168.4.111 lport=22  exe="/usr/libexec/openssh/sshd-session" hostname=? addr=192.168.4.113 terminal=? res=success'
Nov 19 09:15:59 audit[3232]: CRYPTO_SESSION pid=3232 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-client cipher=aes256-gcm@openssh.com ksize=256 mac=<implicit> pfs=curve25519-sha256 spid=3233 suid=74 rport=58590 laddr=192.168.4.111 lport=22  exe="/usr/libexec/openssh/sshd-session" hostname=? addr=192.168.4.113 terminal=? res=success'
Nov 19 09:16:00 audit[3232]: USER_AUTH pid=3232 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=pubkey acct="jeff" exe="/usr/libexec/openssh/sshd-session" hostname=? addr=192.168.4.113 terminal=ssh res=failed'
Nov 19 09:16:06 audit[3232]: USER_AUTH pid=3232 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_unix acct="jeff" exe="/usr/libexec/openssh/sshd-session" hostname=192.168.4.113 addr=192.168.4.113 terminal=ssh res=success'
Nov 19 09:16:06 audit[3232]: USER_ACCT pid=3232 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=? acct="jeff" exe="/usr/libexec/openssh/sshd-session" hostname=192.168.4.113 addr=192.168.4.113 terminal=ssh res=failed'
Nov 19 09:16:06 sshd-session[3232]: Failed password for jeff from 192.168.4.113 port 58590 ssh2
Nov 19 09:16:06 sshd-session[3232]: fatal: Access denied for user jeff by PAM account configuration [preauth]

Both entries above, the first on the local system and the second with a remote ssh attempt) clearly show that PAM (or authselect) is denying access even though I am 100% certain that the passwords are correct.

This is almost to the point that I am considering a full reinstall on my daily driver. Not a pleasant task with all the extras I have in use!

It might be that read only issue did you check my link above like a similar situation where fix was

touch /forcefsck

Also just got wild thought is the keyboard layout correct

Are you using btrfs?

Maybe try running a scrub

My file system is ext4

I did not. I had already run a forced fsck on the system and home partitions manually. The file systems are not mounting read-only.

The only errors seem to be failed authentication by any user other than root.

Keyboard layout is the same as it has been for years. I have even tried different keyboards, and the same keyboard will log me in as root but not as my other user.

1 Like
Authentication Success, Accounting Failure: The logs consistently show "USER_AUTH res=success" followed by "USER_ACCT res=failed". This means the system successfully verifies the user's password (authentication), but something goes wrong during the accounting phase (which typically involves logging the login attempt and updating system resources).

It shows password is correct, but accounting failed

Pam.d? So something on accounting fails to log in user in

Got also possible idea systemctl status multi-user.target

If you have a file named /etc/nologin that could explain it. Simply remove that file.

2 Likes

It has been years since I have dealt with that file and its effects. Thanks tremendously for the reminder as that was indeed the issue. :+1:

The real question remaining is how that file was created though.?
I have no clue at this point.

We used to use that when preparing for an update on multi-user systems to keep the system idle during the update, but this is not one of those situations.

Perhaps as part of the shutdown command

If the time argument is used, 5 minutes before the system goes down the /run/nologin file is created to ensure that further logins shall not be allowed.