Cannot login per ssh via terraform connection

Hello,

with the most recent version of CoreOS, 33.20210117.3.2, I cannot use the terraform ssh connection provisioner anymore. It used to work in prior versions.

Problem is that a SSH connection cannot be established by means of the tf connector, but it works with a simple “ssh core@”. Here are two log entries, acquired with journalctl while trying to login with either method:

When trying with terraform connector (failure):

Feb 13 14:27:10 k1.local.vlan audit[18899]: USER_LOGIN pid=18899 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="core" exe="/usr/sbin/sshd" hostname=? addr=192.168.56.1 terminal=ssh res=failed'

When trying with “ssh core@…” (success):

Feb 13 14:29:00 k1.local.vlan audit[20006]: USER_LOGIN pid=20006 uid=0 auid=1000 ses=12 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=? addr=192.168.56.1 terminal=/dev/pts/1 res=success'

The user is in both cases the same (core) and ssh is via RSA pub key from ssh-agent. Here is a simple terraform module to reproduce the case:

resource "null_resource" "copy" {
    connection {
        type = "ssh"
        host = "<some coreos host>"
        user = "core"
        timeout = "10m"
        agent = true
    }
    
    provisioner "file" {
        content = "blabla"
        destination = "ttt"
    }
}

when running “terraform init” and “terraform apply”, the apply hangs for 10min and then dies. It works for any other host, which is not CoreOS.
When setting the TF_LOG to TRACE, there is a related message:

SSH authentication failed (core@k1.local.vlan:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Terraform version is 0.13 or 0.14, I have the issue with both.

Is there anything possibly wrong on my side or is this a bug?

I have some more logs when the login failure occurs. This is logged in the syslog of the coreos host when I try to login via the terraform connector, as described above:

userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth]
USER_ERR pid=4191 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=192.168.56.1 addr=192.168.56.1 terminal=ssh res=failed'

So, ssh-rsa is indeed not in the PubKeyAcceptedKeyTypes, as I confirmed with “sudo sshd -T | grep -i PubkeyAcceptedKeyTypes”. The output is

pubkeyacceptedkeytypes ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com

But it seems I cannot configure this value in /etc/ssh/sshd_config, when I put the line

PubkeyAcceptedKeyTypes +ssh-rsa

in /etc/ssh/sshd_config and restart the sshd, the value is still not displayed in the sshd config and it doesn’t solve anything. Thats weird, but I don’t know if it would solve the issue anyway if ssh-rsa would be on that list, because a simple ssh’ing with the same key works also without ssh-rsa being on that list.

Furthermore it seems that the issue is related to how ssh is invoked on the terraform side, or a combination of this invocation and the coreos sshd configuration. Apparently I must open an issue with terraform, too…

Make sure the permissions on the file are 0600.

See also Fedora CoreOS Frequently Asked Questions :: Fedora Docs and specifically for Terraform: F33: updated crypto defaults: SSH tooling may fail · Issue #699 · coreos/fedora-coreos-tracker · GitHub.

Reading the comments there, it seems to be exactly the problem: tighter ssh security settings on fedora 33 and a missing fix related to this in the golang library, which beats through to terraform. Looks like I just have to wait for the golang/terraform update and must work around that issue in the meantime with a different key.