Generate new password to synchronized KVNO in /etc/krb5.keytab

Ask Fedora
1

Hello,

I am getting this error in my log:

Sep 11 22:51:36 myhost.ad.domain.lan ldap_child[221671]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Preauthentication failed. Unable to create GSSAPI-encrypted LDAP connection.

Reading about I found that this can happen if the key version number is desynchronized, which seems to be exactly my situation. Using the commands mentioned in that post:

# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   1 host/myhost.ad.domain.lan@AD.DOMAIN.LAN
   1 host/MYHOST@AD.DOMAIN.LAN
   1 host/myhost.ad.domain.lan@AD.DOMAIN.LAN
   1 host/MYHOST@AD.DOMAIN.LAN
   1 host/myhost.ad.domain.lan@AD.DOMAIN.LAN
   1 host/MYHOST@AD.DOMAIN.LAN
   1 restrictedkrbhost/myhost.ad.domain.lan@AD.DOMAIN.LAN
   1 restrictedkrbhost/MYHOST@AD.DOMAIN.LAN
   1 restrictedkrbhost/myhost.ad.domain.lan@AD.DOMAIN.LAN
   1 restrictedkrbhost/MYHOST@AD.DOMAIN.LAN
   1 restrictedkrbhost/myhost.ad.domain.lan@AD.DOMAIN.LAN
   1 restrictedkrbhost/MYHOST@AD.DOMAIN.LAN
   1 MYHOST$@AD.DOMAIN.LAN
   1 MYHOST$@AD.DOMAIN.LAN
   1 MYHOST$@AD.DOMAIN.LAN
   2 MYHOST$@AD.DOMAIN.LAN
   2 MYHOST$@AD.DOMAIN.LAN
   2 host/myhost.ad.domain.lan@AD.DOMAIN.LAN
   2 host/myhost.ad.domain.lan@AD.DOMAIN.LAN
   2 host/MYHOST@AD.DOMAIN.LAN
   2 host/MYHOST@AD.DOMAIN.LAN
   2 restrictedkrbhost/myhost.ad.domain.lan@AD.DOMAIN.LAN
   2 restrictedkrbhost/myhost.ad.domain.lan@AD.DOMAIN.LAN
   2 restrictedkrbhost/MYHOST@AD.DOMAIN.LAN
   2 restrictedkrbhost/MYHOST@AD.DOMAIN.LAN

Clearly the latest version is 2 (running with -kt args shows that it is 4 months old) so somehow the computer account key has not been updated in a while.

It seems like the version should be 3:

> kvno -S host MYHOST
host/MYHOST@: kvno = 3

Now, according to this post, the solution is to refresh the computer account key but the the instructions given are for IPA.

After searching for how this is done via sssd it appears that the keytab would be updated using adcli update as per, but I get:

root@myhost.ad.domain.lan /v/h/myuser@ad.domain.lan# kinit Administrator
Password for Administrator@AD.DOMAIN.LAN: 

root@myhost.ad.domain.lan /v/h/myuser@ad.domain.lan [1]# klist
Ticket cache: KCM:0
Default principal: administrator@AD.DOMAIN.LAN

Valid starting     Expires            Service principal
11/09/24 21:54:18  12/09/24 07:54:18  krbtgt/AD.DOMAIN.LAN@AD.DOMAIN.LAN
	renew until 18/09/24 21:54:15

root@myhost.ad.domain.lan /v/h/myuser@ad.domain.lan# adcli update -v
 * Found realm in keytab: AD.DOMAIN.LAN
 * Found service principal in keytab: host/myhost.ad.domain.lan
 * Found host qualified name in keytab: myhost.ad.domain.lan
 * Found service principal in keytab: host/myhost
 * Found service principal in keytab: restrictedkrbhost/myhost.ad.domain.lan
 * Found service principal in keytab: restrictedkrbhost/myhost
 * Found computer name in keytab: myhost
 * Calculated domain name from host fqdn: ad.domain.lan
 * Using computer account name: myhost
 * Using domain realm: ad.domain.lan
 * Discovering domain controllers: _ldap._tcp.ad.domain.lan
 * Sending NetLogon ping to domain controller: dc1.ad.domain.lan
 * Received NetLogon info from: dc1.ad.domain.lan
 * Wrote out krb5.conf snippet to /tmp/adcli-krb5-eOVJzZ/krb5.d/adcli-krb5-conf-PinApq
 ! Couldn't authenticate as machine account: myhost: Preauthentication failed
adcli: couldn't connect to ad.domain.lan domain: Couldn't authenticate as machine account: myhost: Preauthentication failed

So, how can I sync up the key version number to fix this? Short of leaving/rejoining the domain, is there something else I could do?

2

@akarypid, this issue reminds me of times I had working in a kerberos environment where we had both an AD kdc and a separate kdc on Linux (two realms).

As I recall once the kvno changes on the kdc the older ones will never be valid again. Also the AD kdc needed the various principal forms for the host to operate properly.

I think the hosts krb5.conf can be adjusted such that kinit’ing a new host keytab will result in all the necessary principal forms to be made. Or it should be able to be done on the kinit command line.

Also, removing the no-longer-usable principals from the /etc/krb5.keytab may be helpful (make a backup copy first).

I do not have any kerberos setup now so I cannot be as helpful as I prefer. After the exceptional passwordless fedora presentation at flock2024, I’m thinking kerberos may become relavent again as a solution that runs a local kdc was quite interesting.

1 Like
3

I gave up and simply did a leave/re-join domain process. Part of the problem here is that this is hard to replicate, so there is little documentation around it…

Just reading upon this shows how difficult it is to debug.

For example, I will quote from this post:

We have about ~20K Linux servers that are sssd-enabled. A mix of RHEL6,
RHEL7, RHEL8, Oracle Linux 6, 7 and 8. We have ~7K Linux servers still on
the old commercial product. …

… Again, this communication failure must be very infrequent, as it’s
occurring only 0.3% of the time or so. On random Linux servers – no
discernible geographic or build location pattern.

This problem could even be caused by restoring a VM image to a previous snapshot (or a backup/restore of any sort) that includes the /etc/krb5.keytab file. Even this would cause a Linux instance that had updated to a newer key version to suddenly run with an older key version and get into this state…

It seems that in these cases the “leave then rejoin domain” operation is probably the easiest solution…

1 Like
4

Sounds great. Having the authority to leave/join makes you powerful;-)

I did find that wireshark was helpful in debugging as it does quite a good job decoding kerberos communication exchanges. I’ve also worked in environments where sniffing packets on a computer was forbidden to mere mortals.

1 Like