It starts, but I cannot login

Project Discussion
1

I have decided to give CoreOS a try on libvirt, using the image fedora-coreos-31.20200323.3.2-qemu.x86_64.qcow2 and the instructions using the virt-install method.

I’m using f32 as host os and after a few unsuccessful tries, I setenforce 0 to get SELinux out for the moment.

However, until now CoreOS boots but I not able to login (neither on the login prompt nor with ssh).

This is my fcc file:

variant: fcos
version: 1.0.0
passwd:
  users:
    - name: core
      password_hash: $6$s...W$G....s0
      ssh_authorized_keys:
        - ssh-ed25519 AA...d aanno@linux.fritz.box
# allow passwd login
# https://discussion.fedoraproject.org/t/best-way-to-enable-password-auth-on-ssh/17731/4
systemd:
  units:
    - name: sshd.service
      dropins:
      - name: allowpasswordauth.conf
        contents: |
          [Service]
          Environment=OPTIONS='-oPasswordAuthentication=yes'

When I try to login with ssh, I could connect to the CoreOS sshd server, but login is unsuccessful:

$ ssh core@192.168.122.221
core@192.168.122.221: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

On the CoreOS (login) screen, I see the folowing:

localhost login: [ 1060.587819] audit: type=2404 audit(1586700155.788:178): pid=3201 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:57:8a:b4:b9:03:95:68:7e:99:97:bb:4c:41:e0:88:c2:4f:d8:70:15:df:e0:c0:1f:e1:8a:20:e9:fd:1c:14:5e direction=? spid=3201 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
[ 1060.591641] audit: type=2407 audit(1586700155.793:179): pid=3200 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-server cipher=aes256-gcm@openssh.com ksize=256 mac=<implicit> pfs=curve25519-sha256 spid=3201 suid=74 rport=35690 laddr=192.168.122.221 lport=22  exe="/usr/sbin/sshd" hostname=? addr=192.168.122.1 terminal=? res=success'
[ 1060.595264] audit: type=2407 audit(1586700155.796:180): pid=3200 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-client cipher=aes256-gcm@openssh.com ksize=256 mac=<implicit> pfs=curve25519-sha256 spid=3201 suid=74 rport=35690 laddr=192.168.122.221 lport=22  exe="/usr/sbin/sshd" hostname=? addr=192.168.122.1 terminal=? res=success'
[ 1060.618321] audit: type=1100 audit(1586700155.819:181): pid=3200 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=pubkey acct="core" exe="/usr/sbin/sshd" hostname=? addr=192.168.122.1 terminal=ssh res=failed'
[ 1060.625982] audit: type=1100 audit(1586700155.827:182): pid=3200 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=pubkey acct="core" exe="/usr/sbin/sshd" hostname=? addr=192.168.122.1 terminal=ssh res=failed'
[ 1060.634746] audit: type=2404 audit(1586700155.835:183): pid=3200 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=both spid=3201 suid=74 rport=35690 laddr=192.168.122.221 lport=22  exe="/usr/sbin/sshd" hostname=? addr=192.168.122.1 terminal=? res=success'
[ 1060.638336] audit: type=2404 audit(1586700155.839:184): pid=3200 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:57:8a:b4:b9:03:95:68:7e:99:97:bb:4c:41:e0:88:c2:4f:d8:70:15:df:e0:c0:1f:e1:8a:20:e9:fd:1c:14:5e direction=? spid=3201 suid=74  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
[ 1060.641617] audit: type=1109 audit(1586700155.839:185): pid=3200 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=192.168.122.1 addr=192.168.122.1 terminal=ssh res=failed'
[ 1060.644073] audit: type=2404 audit(1586700155.839:186): pid=3200 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:57:8a:b4:b9:03:95:68:7e:99:97:bb:4c:41:e0:88:c2:4f:d8:70:15:df:e0:c0:1f:e1:8a:20:e9:fd:1c:14:5e direction=? spid=3200 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
[ 1060.647509] audit: type=1112 audit(1586700155.839:187): pid=3200 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="core" exe="/usr/sbin/sshd" hostname=? addr=192.168.122.1 terminal=ssh res=failed'

From the above, I suspect that my ignition file is still not honour (otherwise I would expect at least a passwort prompt when using ssh). However, I have no idea why.

How coud I debug the problem?

Kind regards,

aanno

2

Check the console log on the VM’s serial port 0 (or on its VGA console, if you delete console=ttyS0,115200 from the kernel command-line arguments). Ignition should be reporting what it’s doing.

1 Like
3

Well, I’m not sure how to change the kernel parameters (is it possible with coreos-install and if yes how?). However, I was able to redirect all output of the virt-install to a file.

But the output is a bit too long to post it here (and an upload is only allowed for image formats). I have looked a bit at the file but I have not found anything that has drawn my attention. Is there some keyword to look for?

4

I escaped to using the iso image and doing a ‘raw’ install on virt-manager/qemu. This is not the solution but it circumvents the problem. I still feels strange to me that nobody else encounters this problem…

5

Hmm, are you also unable to login using your private key? That’s definitely odd.

In the logs early on, you should be able to see Ignition dropping the allowpasswordauth.conf file and the authorized keys.

If this is all local, one other way to debug this is to follow these steps so you can log into the system over the console and poke around.

6

@aanno - were you able to figure out the problem?

7

I had the same issue.
Solution for password: hash must be sha-256.
I used
mkpasswd --method=sha-256.
Solution for public key auth: it worked with 2048 bit RSA key. With my old 1024 bit RSA key it didn’t.

Cheers
Thomas

1 Like
8

Hey, I had a similar issue. In my case it was a file system specified in my ignition file to mount at /var. Due to that I was not able to login using my private key but instead had to use a password.

9

I’ve been trying to install on bare metal for several days. I think that I have discovered the reason we can’t log in, but not how to fix it. Apparently, there was a change in how SSH accesses the the keys and where FCOS/Ignition put them. As I understand it FCOS/Ignition writes writes SSH keys to .ssh/authorized_keys.d/ignition file in the user’s folder and SSH is looking for them in ~/.ssh/authorized_keys file. This is a software killing flaw that seems to have been around since about October of last year. I’ve spend literally days on this building Ignition files and reinstalling, thinking it was something wrong I was doing.

10

sshd looks in ~/.ssh/authorized_keys.d/ignition on FCOS, and always has. OKD, running on FCOS, switches sshd back to ~/.ssh/authorized_keys to improve consistency with OpenShift on RHCOS. But if you’re not running OKD, this shouldn’t affect you.