After upgrading a server from Fedora 38 to 39 SSH started rejecting password-authenticated connection attempts with “Permission denied”. Luckily it was still possible to log in with a Kerberos ticket and I discovered that sssd_pam was crashing:
Relevant section from the journal:
Nov 28 15:37:56 server001 krb5_child[27022]: Pre-authentication failed: Invalid argument
Nov 28 15:37:56 server001 audit[10812]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=10812 comm="sssd_pam" exe="/usr/libexec/sssd/sssd_pam" sig=11 res=1
Nov 28 15:37:56 server001 kernel: sssd_pam[10812]: segfault at 0 ip 00005654a80e5723 sp 00007ffe77592d90 error 4 in sssd_pam[5654a80c1000+28000] likely on CPU 6 (core 6, socket 0)
Nov 28 15:37:56 server001 kernel: Code: c2 48 8d 05 77 67 00 00 48 89 45 80 48 89 c7 31 c0 e8 f1 d7 fd ff 48 c7 85 70 ff ff ff 01 00 00 00 49 8b 44 24 18 48 8b 5d a0 <48> 8b 30 44 0f b6 7b 48 48 89 df e8 3d cc fd ff 48 c7 45 98 00 00
Nov 28 15:37:56 server001 audit: BPF prog-id=126 op=LOAD
Nov 28 15:37:56 server001 audit: BPF prog-id=127 op=LOAD
Nov 28 15:37:56 server001 audit: BPF prog-id=128 op=LOAD
Nov 28 15:37:57 server001 systemd[1]: Started systemd-coredump@1-27023-0.service - Process Core Dump (PID 27023/UID 0).
Nov 28 15:37:57 server001 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-coredump@1-27023-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 28 15:37:57 server001 systemd-coredump[27024]: Process 10812 (sssd_pam) of user 0 dumped core.
Module tdb.so from rpm libldb-2.8.0-1.fc39.x86_64
Module skel.so from rpm libldb-2.8.0-1.fc39.x86_64
Module server_sort.so from rpm libldb-2.8.0-1.fc39.x86_64
Module sample.so from rpm libldb-2.8.0-1.fc39.x86_64
Module rdn_name.so from rpm libldb-2.8.0-1.fc39.x86_64
Module paged_searches.so from rpm libldb-2.8.0-1.fc39.x86_64
Module memberof.so from rpm sssd-2.9.3-1.fc39.x86_64
Module mdb.so from rpm libldb-2.8.0-1.fc39.x86_64
Module liblmdb.so.0.0.0 from rpm lmdb-0.9.31-2.fc39.x86_64
Module libldb-tdb-err-map.so from rpm libldb-2.8.0-1.fc39.x86_64
Module libldb-key-value.so from rpm libldb-2.8.0-1.fc39.x86_64
Module libldb-mdb-int.so from rpm libldb-2.8.0-1.fc39.x86_64
Module libldb-tdb-int.so from rpm libldb-2.8.0-1.fc39.x86_64
Module ldb.so from rpm libldb-2.8.0-1.fc39.x86_64
Module libcrypt.so.2 from rpm libxcrypt-4.4.36-2.fc39.x86_64
Module libssl.so.3 from rpm openssl-3.1.1-4.fc39.x86_64
Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-11.fc39.x86_64
Module libevent-2.1.so.7 from rpm libevent-2.1.12-9.fc39.x86_64
Module ldap.so from rpm libldb-2.8.0-1.fc39.x86_64
Module asq.so from rpm libldb-2.8.0-1.fc39.x86_64
Module libpath_utils.so.1 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libz.so.1 from rpm zlib-1.2.13-4.fc39.x86_64
Module libzstd.so.1 from rpm zstd-1.5.5-4.fc39.x86_64
Module liblzma.so.5 from rpm xz-5.4.4-1.fc39.x86_64
Module liblz4.so.1 from rpm lz4-1.9.4-4.fc39.x86_64
Module libcap.so.2 from rpm libcap-2.48-7.fc39.x86_64
Module libsss_cert.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libcollection.so.4 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libref_array.so.1 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libbasicobjects.so.0 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libini_config.so.5 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libpcre2-8.so.0 from rpm pcre2-10.42-1.fc39.2.x86_64
Module libunistring.so.5 from rpm libunistring-1.1-5.fc39.x86_64
Module libdbus-1.so.3 from rpm dbus-1.14.10-1.fc39.x86_64
Module libcrypto.so.3 from rpm openssl-3.1.1-4.fc39.x86_64
Module libkeyutils.so.1 from rpm keyutils-1.6.1-7.fc39.x86_64
Module libkrb5support.so.0 from rpm krb5-1.21.2-2.fc39.x86_64
Module libcom_err.so.2 from rpm e2fsprogs-1.47.0-2.fc39.x86_64
Module libk5crypto.so.3 from rpm krb5-1.21.2-2.fc39.x86_64
Module libkrb5.so.3 from rpm krb5-1.21.2-2.fc39.x86_64
Module libeconf.so.0 from rpm libeconf-0.5.2-1.fc39.x86_64
Module libaudit.so.1 from rpm audit-3.1.2-5.fc39.x86_64
Module libtalloc.so.2 from rpm libtalloc-2.4.1-1.fc39.x86_64
Module libtevent.so.0 from rpm libtevent-0.15.0-1.fc39.x86_64
Module libdhash.so.1 from rpm ding-libs-0.6.2-54.fc39.x86_64
Module libsss_sbus.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libsss_iface.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libsystemd.so.0 from rpm systemd-254.5-2.fc39.x86_64
Module libsss_debug.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libsss_child.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libsss_crypt.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libtdb.so.1 from rpm libtdb-1.4.9-1.fc39.x86_64
Module libselinux.so.1 from rpm libselinux-3.5-5.fc39.x86_64
Module libldb.so.2 from rpm libldb-2.8.0-1.fc39.x86_64
Module libpopt.so.0 from rpm popt-1.19-3.fc39.x86_64
Module libsss_util.so from rpm sssd-2.9.3-1.fc39.x86_64
Module libsss_certmap.so.0 from rpm sssd-2.9.3-1.fc39.x86_64
Module libgssapi_krb5.so.2 from rpm krb5-1.21.2-2.fc39.x86_64
Module libpam.so.0 from rpm pam-1.5.3-3.fc39.x86_64
Module sssd_pam from rpm sssd-2.9.3-1.fc39.x86_64
Stack trace of thread 10812:
#0 0x00005654a80e5723 pam_passkey_auth_send.isra.0 (sssd_pam + 0x2d723)
#1 0x00005654a80e68a0 pam_passkey_get_user_done (sssd_pam + 0x2e8a0)
#2 0x00005654a80e3a11 pam_passkey_get_mapping_done (sssd_pam + 0x2ba11)
#3 0x00007f839f88be40 tevent_common_invoke_immediate_handler (libtevent.so.0 + 0xbe40)
#4 0x00007f839f88bea2 tevent_common_loop_immediate (libtevent.so.0 + 0xbea2)
#5 0x00007f839f88fa22 epoll_event_loop_once (libtevent.so.0 + 0xfa22)
#6 0x00007f839f887894 std_event_loop_once (libtevent.so.0 + 0x7894)
#7 0x00007f839f889e1b _tevent_loop_once (libtevent.so.0 + 0x9e1b)
#8 0x00007f839f889f6b tevent_common_loop_wait (libtevent.so.0 + 0x9f6b)
#9 0x00007f839f887914 std_event_loop_wait (libtevent.so.0 + 0x7914)
#10 0x00007f839fb10a6f server_loop (libsss_util.so + 0x50a6f)
#11 0x00005654a80c6a63 main (sssd_pam + 0xea63)
#12 0x00007f839f6b014a __libc_start_call_main (libc.so.6 + 0x2814a)
#13 0x00007f839f6b020b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2820b)
#14 0x00005654a80c7245 _start (sssd_pam + 0xf245)
ELF object binary architecture: AMD x86-64
Nov 28 15:37:57 server001 audit[26944]: USER_AUTH pid=26944 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=PAM:authentication grantors=? acct="user001" exe="/usr/sbin/sshd" hostname=192.168.2.69 addr=192.168.2.69 terminal=ssh res=failed'
Nov 28 15:37:57 server001 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-coredump@1-27023-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 28 15:37:57 server001 systemd[1]: systemd-coredump@1-27023-0.service: Deactivated successfully.
Nov 28 15:37:57 server001 sssd_pam[27032]: Starting up
Nov 28 15:37:57 server001 audit: BPF prog-id=128 op=UNLOAD
Nov 28 15:37:57 server001 audit: BPF prog-id=127 op=UNLOAD
Nov 28 15:37:57 server001 audit: BPF prog-id=126 op=UNLOAD
Nov 28 15:37:57 server001 abrt-server[27037]: Deleting problem directory ccpp-2023-11-28-15:37:57.230849-10812 (dup of ccpp-2023-11-28-13:33:41.300493-2042)
Nov 28 15:37:58 server001 abrt-notification[27135]: Process 2042 (sssd_pam) crashed in pam_passkey_auth_send.isra.0()
Nov 28 15:37:59 server001 sshd[26944]: Failed password for user001 from 192.168.2.69 port 44444 ssh2
Also, while trying to understand what’s wrong I tried logging in from a local account into an AD one, and instead of being asked for a password I’m getting this message that I’ve never seen before:
Kerberos TGT will not be granted upon login, user experience will be affected.
Insert your passkey device, then press ENTER.
Communication with AD seems to be fine, at least things like getent passwd $USERNAME behave as expected and I can get a ticket with kinit.
What’s wrong and how do I go about diagnosing/fixing this problem? Or should I report the issue to the SSSD devs?