Fedora CoreOS next stream rebasing to Fedora Linux 38

Project Discussion
1

Update: see Fedora CoreOS next stream rebasing to Fedora Linux 38 - #16 by bgilbert

Cross-posting from this coreos-status email.

Fedora Linux 38 Beta will be released next week [1]. Starting March 14, newly provisioned Fedora CoreOS nodes on the next stream will be based on Fedora Linux 38. On March 16, existing Fedora CoreOS nodes on the next stream will begin to update.

For more information about Fedora 38, see the Fedora Project’s list of official Changes [2] and the Fedora CoreOS analysis of each Change [3].

One Change related to OpenSSH [4] deserves special mention. In Fedora 38, SSH host keys are migrating to a new set of permissions to better match upstream defaults. This migration is performed automatically and we do not expect any issues. However, a failed migration may prevent SSH access to the machine. If you have console access to your machines and wish to be especially cautious, consider setting a password for the core user.

Please test out the next stream over the coming month and report any issues in our issue tracker [5].

Thank you to everyone helping find issues by running the next stream!

Jonathan Lebon, for the Fedora CoreOS team

  1. F38 Beta is GO - devel-announce - Fedora Mailing-Lists ↩︎

  2. Releases/38/ChangeSet - Fedora Project Wiki ↩︎

  3. tracker: Fedora 38 changes considerations · Issue #1357 · coreos/fedora-coreos-tracker · GitHub ↩︎

  4. Changes/SSHKeySignSuidBit - Fedora Project Wiki ↩︎

  5. Issues · coreos/fedora-coreos-tracker · GitHub ↩︎

4 Likes
2

Reading your post make me aware my lost of ssh after rebase Silverblie 37 to Services 38.

3

Hi,

Out of curiosity, when did you do the rebase? Was it before or after https://bodhi.fedoraproject.org/updates/FEDORA-2023-fdf3721184 was pushed to the repos? If before, updating to the latest compose should automatically fix the issue if it hasn’t been fixed manually yet.

4

It should be 1 day before my post.

As the push to stable is 4 days ago, I think my rebase is after that.

Not sure how much delay internet mirror sync is adding.

== Update ==

  1. In a VM, I installed Silverblue 37
$ rpm-ostree status
State: busy
Transaction: rebase fedora:fedora/38/x86_64/sericea 
  Initiator: client(id:cli dbus:1.213 unit:vte-spawn-7e3b0af5-4607-4cfa-ae98-72a9905adaab.scope uid:0)
Deployments:
● fedora:fedora/37/x86_64/silverblue
                  Version: 37.1.7 (2022-11-05T06:01:00Z)
                   Commit: bfe9de223c9a4ba4a793d3e01f6b09024c919685ee73c896af767958725cac79
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A

  1. Enabled sshd, tested remote access OK.

  2. Then immediately rebase to Sericea 38 -

State: idle
Deployments:
● fedora:fedora/38/x86_64/sericea
                  Version: 38.20230313.n.0 (2023-03-13T08:02:37Z)
                   Commit: 215b7e48549f0b1cb521dec6e5a08bda68c9d9e507386032a3c53c139bf45790
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464

  fedora:fedora/37/x86_64/silverblue
                  Version: 37.1.7 (2022-11-05T06:01:00Z)
                   Commit: bfe9de223c9a4ba4a793d3e01f6b09024c919685ee73c896af767958725cac79
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A

  1. After reboot, remote ssh got “Connection refused”

  2. sudo systemctl status sshd “code-exited, status=1/FAILURE”

  3. sudo journalctl -u sshd (Update: Fixed typo of Permission)

...
WARNING: UNPROTECTED PRIVATE KEY FILE!
...
Permission 0640 for `/etc/ssh/ssh_host_rsa_key` are too open.

For all the keys: rsa, ecdsa, ed25519

  1. ls -l /etc/ssh/*key in SB37:
    -rw-r-----.

  2. ls -l /etc/ssh/*key in Sericea 38:
    -rw-r-----.

  3. Reboot - same result

  4. sudo rpm-ostree update - “No upgrade available.”

  5. sshd start OK after sudo chmod 0600 /etc/ssh/*key

5

Interesting… I guess the fear we had over in https://github.com/coreos/fedora-coreos-tracker/issues/1394#issuecomment-1460302563 was legitimate.

What language to you have set on your system?

6

As it is a quick test to reproduce the issue, I change nothing. English is selected during installation.

7

Can you run the following commands and share the output?

  • sudo sshd -T
  • sudo LANG=c sshd -T
8

This is run:

  • Terminal under Workstation 38
  • ssh into the VM running Sericea 38
$sudo sshd -T
port 22
addressfamily any
listenaddress [::]:22
listenaddress 0.0.0.0:22
usepam yes
logingracetime 120
x11displayoffset 10
x11maxdisplays 1000
maxauthtries 6
maxsessions 10
clientaliveinterval 0
clientalivecountmax 3
requiredrsasize 2048
streamlocalbindmask 0177
permitrootlogin without-password
ignorerhosts yes
ignoreuserknownhosts no
hostbasedauthentication no
hostbasedusesnamefrompacketonly no
pubkeyauthentication yes
kerberosauthentication no
kerberosorlocalpasswd yes
kerberosticketcleanup yes
kerberosuniqueccache no
kerberosusekuserok yes
gssapienablek5users no
gssapiauthentication yes
gssapicleanupcredentials no
gssapikeyexchange no
gssapistrictacceptorcheck yes
gssapistorecredentialsonrekey no
gssapikexalgorithms gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-
passwordauthentication yes
kbdinteractiveauthentication no
printmotd no
printlastlog yes
x11forwarding yes
x11uselocalhost yes
permittty yes
permituserrc yes
strictmodes yes
tcpkeepalive yes
permitemptypasswords no
compression yes
gatewayports no
usedns no
allowtcpforwarding yes
allowagentforwarding yes
disableforwarding no
allowstreamlocalforwarding yes
streamlocalbindunlink no
fingerprinthash SHA256
exposeauthinfo no
pidfile /var/run/sshd.pid
modulifile /etc/ssh/moduli
xauthlocation /usr/bin/xauth
ciphers aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr
macs hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
banner none
forcecommand none
chrootdirectory none
trustedusercakeys none
revokedkeys none
securitykeyprovider internal
authorizedprincipalsfile none
versionaddendum none
authorizedkeyscommand none
authorizedkeyscommanduser none
authorizedprincipalscommand none
authorizedprincipalscommanduser none
hostkeyagent none
kexalgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
casignaturealgorithms ecdsa-sha2-nistp256,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-256,rsa-sha2-512
hostbasedacceptedalgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com
hostkeyalgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com
pubkeyacceptedalgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com
loglevel INFO
syslogfacility AUTHPRIV
authorizedkeysfile .ssh/authorized_keys
hostkey /etc/ssh/ssh_host_rsa_key
hostkey /etc/ssh/ssh_host_ecdsa_key
hostkey /etc/ssh/ssh_host_ed25519_key
authenticationmethods any
subsystem sftp /usr/libexec/openssh/sftp-server
maxstartups 10:30:100
persourcemaxstartups none
persourcenetblocksize 32:128
permittunnel no
ipqos af21 cs1
rekeylimit 0 0
permitopen any
permitlisten any
permituserenvironment no
pubkeyauthoptions none


$ sudo LANG=c sshd -T
port 22
addressfamily any
listenaddress [::]:22
listenaddress 0.0.0.0:22
usepam yes
logingracetime 120
x11displayoffset 10
x11maxdisplays 1000
maxauthtries 6
maxsessions 10
clientaliveinterval 0
clientalivecountmax 3
requiredrsasize 2048
streamlocalbindmask 0177
permitrootlogin without-password
ignorerhosts yes
ignoreuserknownhosts no
hostbasedauthentication no
hostbasedusesnamefrompacketonly no
pubkeyauthentication yes
kerberosauthentication no
kerberosorlocalpasswd yes
kerberosticketcleanup yes
kerberosuniqueccache no
kerberosusekuserok yes
gssapienablek5users no
gssapiauthentication yes
gssapicleanupcredentials no
gssapikeyexchange no
gssapistrictacceptorcheck yes
gssapistorecredentialsonrekey no
gssapikexalgorithms gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-
passwordauthentication yes
kbdinteractiveauthentication no
printmotd no
printlastlog yes
x11forwarding yes
x11uselocalhost yes
permittty yes
permituserrc yes
strictmodes yes
tcpkeepalive yes
permitemptypasswords no
compression yes
gatewayports no
usedns no
allowtcpforwarding yes
allowagentforwarding yes
disableforwarding no
allowstreamlocalforwarding yes
streamlocalbindunlink no
fingerprinthash SHA256
exposeauthinfo no
pidfile /var/run/sshd.pid
modulifile /etc/ssh/moduli
xauthlocation /usr/bin/xauth
ciphers aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr
macs hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
banner none
forcecommand none
chrootdirectory none
trustedusercakeys none
revokedkeys none
securitykeyprovider internal
authorizedprincipalsfile none
versionaddendum none
authorizedkeyscommand none
authorizedkeyscommanduser none
authorizedprincipalscommand none
authorizedprincipalscommanduser none
hostkeyagent none
kexalgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
casignaturealgorithms ecdsa-sha2-nistp256,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-256,rsa-sha2-512
hostbasedacceptedalgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com
hostkeyalgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com
pubkeyacceptedalgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com
loglevel INFO
syslogfacility AUTHPRIV
authorizedkeysfile .ssh/authorized_keys
hostkey /etc/ssh/ssh_host_rsa_key
hostkey /etc/ssh/ssh_host_ecdsa_key
hostkey /etc/ssh/ssh_host_ed25519_key
authenticationmethods any
subsystem sftp /usr/libexec/openssh/sftp-server
maxstartups 10:30:100
persourcemaxstartups none
persourcenetblocksize 32:128
permittunnel no
ipqos af21 cs1
rekeylimit 0 0
permitopen any
permitlisten any
permituserenvironment no
pubkeyauthoptions none
9

hmm. I don’t see any errors there. If you reboot your host and run the following commands what do you see?

  • sudo systemctl status sshd
  • sudo journalctl -b0 -u sshd
10

I fixed my key permission to 0600 already. Do you want me to change it back to the same as fresh SB37?

Below logs before changing key permissions:

r 14 09:03:47 fedora systemd[1]: Starting sshd.service - OpenSSH server daemon...
Mar 14 09:03:47 fedora sshd[770]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:03:47 fedora sshd[770]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:03:47 fedora sshd[770]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:03:47 fedora sshd[770]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
Mar 14 09:03:47 fedora sshd[770]: It is required that your private key files are NOT accessible by others.
Mar 14 09:03:47 fedora sshd[770]: This private key will be ignored.
Mar 14 09:03:47 fedora sshd[770]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:03:47 fedora sshd[770]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:03:47 fedora sshd[770]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:03:47 fedora sshd[770]: Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
Mar 14 09:03:47 fedora sshd[770]: It is required that your private key files are NOT accessible by others.
Mar 14 09:03:47 fedora sshd[770]: This private key will be ignored.
Mar 14 09:03:47 fedora sshd[770]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:03:47 fedora sshd[770]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:03:47 fedora sshd[770]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:03:47 fedora sshd[770]: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Mar 14 09:03:47 fedora sshd[770]: It is required that your private key files are NOT accessible by others.
Mar 14 09:03:47 fedora sshd[770]: This private key will be ignored.
Mar 14 09:03:47 fedora sshd[770]: sshd: no hostkeys available -- exiting.
Mar 14 09:03:47 fedora systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Mar 14 09:03:47 fedora systemd[1]: sshd.service: Failed with result 'exit-code'.
Mar 14 09:03:47 fedora systemd[1]: Failed to start sshd.service - OpenSSH server daemon.
Mar 14 09:04:29 fedora systemd[1]: sshd.service: Scheduled restart job, restart counter is at 1.
Mar 14 09:04:29 fedora systemd[1]: Stopped sshd.service - OpenSSH server daemon.
Mar 14 09:04:29 fedora systemd[1]: Starting sshd.service - OpenSSH server daemon...
Mar 14 09:04:29 fedora sshd[1202]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:04:29 fedora sshd[1202]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:04:29 fedora sshd[1202]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:04:29 fedora sshd[1202]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
Mar 14 09:04:29 fedora sshd[1202]: It is required that your private key files are NOT accessible by others.
Mar 14 09:04:29 fedora sshd[1202]: This private key will be ignored.
Mar 14 09:04:29 fedora sshd[1202]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:04:29 fedora sshd[1202]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:04:29 fedora sshd[1202]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:04:29 fedora sshd[1202]: Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
Mar 14 09:04:29 fedora sshd[1202]: It is required that your private key files are NOT accessible by others.
Mar 14 09:04:29 fedora sshd[1202]: This private key will be ignored.
Mar 14 09:04:29 fedora sshd[1202]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:04:29 fedora sshd[1202]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:04:29 fedora sshd[1202]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:04:29 fedora sshd[1202]: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Mar 14 09:04:29 fedora sshd[1202]: It is required that your private key files are NOT accessible by others.
Mar 14 09:04:29 fedora sshd[1202]: This private key will be ignored.
Mar 14 09:04:29 fedora sshd[1202]: sshd: no hostkeys available -- exiting.
Mar 14 09:04:29 fedora systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Mar 14 09:04:29 fedora systemd[1]: sshd.service: Failed with result 'exit-code'.
Mar 14 09:04:29 fedora systemd[1]: Failed to start sshd.service - OpenSSH server daemon.
Mar 14 09:05:11 fedora systemd[1]: sshd.service: Scheduled restart job, restart counter is at 2.
Mar 14 09:05:11 fedora systemd[1]: Stopped sshd.service - OpenSSH server daemon.
Mar 14 09:05:11 fedora systemd[1]: Starting sshd.service - OpenSSH server daemon...
Mar 14 09:05:11 fedora sshd[1207]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:11 fedora sshd[1207]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:05:11 fedora sshd[1207]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:11 fedora sshd[1207]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
Mar 14 09:05:11 fedora sshd[1207]: It is required that your private key files are NOT accessible by others.
Mar 14 09:05:11 fedora sshd[1207]: This private key will be ignored.
Mar 14 09:05:11 fedora sshd[1207]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:11 fedora sshd[1207]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:05:11 fedora sshd[1207]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:11 fedora sshd[1207]: Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
Mar 14 09:05:11 fedora sshd[1207]: It is required that your private key files are NOT accessible by others.
Mar 14 09:05:11 fedora sshd[1207]: This private key will be ignored.
Mar 14 09:05:11 fedora sshd[1207]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:11 fedora sshd[1207]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:05:11 fedora sshd[1207]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:11 fedora sshd[1207]: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Mar 14 09:05:11 fedora sshd[1207]: It is required that your private key files are NOT accessible by others.
Mar 14 09:05:11 fedora sshd[1207]: This private key will be ignored.
Mar 14 09:05:11 fedora sshd[1207]: sshd: no hostkeys available -- exiting.
Mar 14 09:05:11 fedora systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Mar 14 09:05:11 fedora systemd[1]: sshd.service: Failed with result 'exit-code'.
Mar 14 09:05:11 fedora systemd[1]: Failed to start sshd.service - OpenSSH server daemon.
Mar 14 09:05:53 fedora systemd[1]: sshd.service: Scheduled restart job, restart counter is at 3.
Mar 14 09:05:53 fedora systemd[1]: Stopped sshd.service - OpenSSH server daemon.
Mar 14 09:05:53 fedora systemd[1]: Starting sshd.service - OpenSSH server daemon...
Mar 14 09:05:53 fedora sshd[1210]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:53 fedora sshd[1210]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:05:53 fedora sshd[1210]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:53 fedora sshd[1210]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
Mar 14 09:05:53 fedora sshd[1210]: It is required that your private key files are NOT accessible by others.
Mar 14 09:05:53 fedora sshd[1210]: This private key will be ignored.
Mar 14 09:05:53 fedora sshd[1210]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:53 fedora sshd[1210]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:05:53 fedora sshd[1210]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:53 fedora sshd[1210]: Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
Mar 14 09:05:53 fedora sshd[1210]: It is required that your private key files are NOT accessible by others.
Mar 14 09:05:53 fedora sshd[1210]: This private key will be ignored.
Mar 14 09:05:53 fedora sshd[1210]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:53 fedora sshd[1210]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:05:53 fedora sshd[1210]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:05:53 fedora sshd[1210]: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Mar 14 09:05:53 fedora sshd[1210]: It is required that your private key files are NOT accessible by others.
Mar 14 09:05:53 fedora sshd[1210]: This private key will be ignored.
Mar 14 09:05:53 fedora sshd[1210]: sshd: no hostkeys available -- exiting.
Mar 14 09:05:53 fedora systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Mar 14 09:05:53 fedora systemd[1]: sshd.service: Failed with result 'exit-code'.
Mar 14 09:05:53 fedora systemd[1]: Failed to start sshd.service - OpenSSH server daemon.
Mar 14 09:06:35 fedora systemd[1]: sshd.service: Scheduled restart job, restart counter is at 4.
Mar 14 09:06:35 fedora systemd[1]: Stopped sshd.service - OpenSSH server daemon.
Mar 14 09:06:35 fedora systemd[1]: Starting sshd.service - OpenSSH server daemon...
Mar 14 09:06:35 fedora sshd[1212]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:06:35 fedora sshd[1212]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:06:35 fedora sshd[1212]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:06:35 fedora sshd[1212]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
Mar 14 09:06:35 fedora sshd[1212]: It is required that your private key files are NOT accessible by others.
Mar 14 09:06:35 fedora sshd[1212]: This private key will be ignored.
Mar 14 09:06:35 fedora sshd[1212]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:06:35 fedora sshd[1212]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:06:35 fedora sshd[1212]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:06:35 fedora sshd[1212]: Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
Mar 14 09:06:35 fedora sshd[1212]: It is required that your private key files are NOT accessible by others.
Mar 14 09:06:35 fedora sshd[1212]: This private key will be ignored.
Mar 14 09:06:35 fedora sshd[1212]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:06:35 fedora sshd[1212]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:06:35 fedora sshd[1212]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:06:35 fedora sshd[1212]: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Mar 14 09:06:35 fedora sshd[1212]: It is required that your private key files are NOT accessible by others.
Mar 14 09:06:35 fedora sshd[1212]: This private key will be ignored.
Mar 14 09:06:35 fedora sshd[1212]: sshd: no hostkeys available -- exiting.
Mar 14 09:06:35 fedora systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Mar 14 09:06:35 fedora systemd[1]: sshd.service: Failed with result 'exit-code'.
Mar 14 09:06:35 fedora systemd[1]: Failed to start sshd.service - OpenSSH server daemon.
Mar 14 09:07:18 fedora systemd[1]: sshd.service: Scheduled restart job, restart counter is at 5.
Mar 14 09:07:18 fedora systemd[1]: Stopped sshd.service - OpenSSH server daemon.
Mar 14 09:07:18 fedora systemd[1]: Starting sshd.service - OpenSSH server daemon...
Mar 14 09:07:18 fedora sshd[1215]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:07:18 fedora sshd[1215]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:07:18 fedora sshd[1215]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:07:18 fedora sshd[1215]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
Mar 14 09:07:18 fedora sshd[1215]: It is required that your private key files are NOT accessible by others.
Mar 14 09:07:18 fedora sshd[1215]: This private key will be ignored.
Mar 14 09:07:18 fedora sshd[1215]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:07:18 fedora sshd[1215]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:07:18 fedora sshd[1215]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:07:18 fedora sshd[1215]: Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
Mar 14 09:07:18 fedora sshd[1215]: It is required that your private key files are NOT accessible by others.
Mar 14 09:07:18 fedora sshd[1215]: This private key will be ignored.
Mar 14 09:07:18 fedora sshd[1215]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:07:18 fedora sshd[1215]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:07:18 fedora sshd[1215]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:07:18 fedora sshd[1215]: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Mar 14 09:07:18 fedora sshd[1215]: It is required that your private key files are NOT accessible by others.
Mar 14 09:07:18 fedora sshd[1215]: This private key will be ignored.
Mar 14 09:07:18 fedora sshd[1215]: sshd: no hostkeys available -- exiting.
Mar 14 09:07:18 fedora systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Mar 14 09:07:18 fedora systemd[1]: sshd.service: Failed with result 'exit-code'.
Mar 14 09:07:18 fedora systemd[1]: Failed to start sshd.service - OpenSSH server daemon.
Mar 14 09:08:00 fedora systemd[1]: sshd.service: Scheduled restart job, restart counter is at 6.
Mar 14 09:08:00 fedora systemd[1]: Stopped sshd.service - OpenSSH server daemon.
Mar 14 09:08:00 fedora systemd[1]: Starting sshd.service - OpenSSH server daemon...
Mar 14 09:08:00 fedora sshd[1216]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:00 fedora sshd[1216]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:08:00 fedora sshd[1216]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:00 fedora sshd[1216]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
Mar 14 09:08:00 fedora sshd[1216]: It is required that your private key files are NOT accessible by others.
Mar 14 09:08:00 fedora sshd[1216]: This private key will be ignored.
Mar 14 09:08:00 fedora sshd[1216]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:00 fedora sshd[1216]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:08:00 fedora sshd[1216]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:00 fedora sshd[1216]: Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
Mar 14 09:08:00 fedora sshd[1216]: It is required that your private key files are NOT accessible by others.
Mar 14 09:08:00 fedora sshd[1216]: This private key will be ignored.
Mar 14 09:08:00 fedora sshd[1216]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:00 fedora sshd[1216]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:08:00 fedora sshd[1216]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:00 fedora sshd[1216]: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Mar 14 09:08:00 fedora sshd[1216]: It is required that your private key files are NOT accessible by others.
Mar 14 09:08:00 fedora sshd[1216]: This private key will be ignored.
Mar 14 09:08:00 fedora sshd[1216]: sshd: no hostkeys available -- exiting.
Mar 14 09:08:00 fedora systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Mar 14 09:08:00 fedora systemd[1]: sshd.service: Failed with result 'exit-code'.
Mar 14 09:08:00 fedora systemd[1]: Failed to start sshd.service - OpenSSH server daemon.
Mar 14 09:08:42 fedora systemd[1]: sshd.service: Scheduled restart job, restart counter is at 7.
Mar 14 09:08:42 fedora systemd[1]: Stopped sshd.service - OpenSSH server daemon.
Mar 14 09:08:42 fedora systemd[1]: Starting sshd.service - OpenSSH server daemon...
Mar 14 09:08:42 fedora sshd[1218]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:42 fedora sshd[1218]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:08:42 fedora sshd[1218]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:42 fedora sshd[1218]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
Mar 14 09:08:42 fedora sshd[1218]: It is required that your private key files are NOT accessible by others.
Mar 14 09:08:42 fedora sshd[1218]: This private key will be ignored.
Mar 14 09:08:42 fedora sshd[1218]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:42 fedora sshd[1218]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:08:42 fedora sshd[1218]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:42 fedora sshd[1218]: Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
Mar 14 09:08:42 fedora sshd[1218]: It is required that your private key files are NOT accessible by others.
Mar 14 09:08:42 fedora sshd[1218]: This private key will be ignored.
Mar 14 09:08:42 fedora sshd[1218]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:42 fedora sshd[1218]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Mar 14 09:08:42 fedora sshd[1218]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Mar 14 09:08:42 fedora sshd[1218]: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Mar 14 09:08:42 fedora sshd[1218]: It is required that your private key files are NOT accessible by others.
Mar 14 09:08:42 fedora sshd[1218]: This private key will be ignored.
Mar 14 09:08:42 fedora sshd[1218]: sshd: no hostkeys available -- exiting.
Mar 14 09:08:42 fedora systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Mar 14 09:08:42 fedora systemd[1]: sshd.service: Failed with result 'exit-code'.
Mar 14 09:08:42 fedora systemd[1]: Failed to start sshd.service - OpenSSH server daemon.

● sshd.service - OpenSSH server daemon
     Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; preset: disabled)
     Active: activating (auto-restart) (Result: exit-code) since Tue 2023-03-14 09:08:42 HKT; 32s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 1218 ExecStart=/usr/sbin/sshd -D $OPTIONS (code=exited, status=1/FAILURE)
   Main PID: 1218 (code=exited, status=1/FAILURE)
        CPU: 12ms
11

ok so maybe your paste from Fedora CoreOS next stream rebasing to Fedora Linux 38 - #4 by sampsonf was a typo where the word shown is Permissoin ? I thought it was a translation.

12

Yes, it is my typo. Sorry for the confusion caused.

13

Does your system show anything for sudo journalctl -u ssh-host-keys-migration.service?

14 
$ sudo journalctl  -u ssh-host-keys-migration.service
-- No entries --
$ sudo systemctl status  ssh-host-keys-migration.service
○ ssh-host-keys-migration.service - Update OpenSSH host key permissions
Loaded: loaded (/usr/lib/systemd/system/ssh-host-keys-migration.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: https://fedoraproject.org/wiki/Changes/SSHKeySignSuidBit
15

https://bugzilla.redhat.com/show_bug.cgi?id=2172956#c20

16

We have withdrawn Fedora CoreOS next 38.20230310.1.0, which was released on Tuesday, due to a boot failure affecting aarch64 systems. Existing nodes will not update to the withdrawn release, and stream metadata has been reverted to point to the previous release. We are working on a fix and will provide more details early next week.

17

I think sudo chmod 0600 /etc/ssh/*key and systemctl enable ssh-host-keys-migration helped.

I was getting “connection reset by peer” afterwards so I did a rpm-ostree rollback. It turns out our LAN system admins’ DarkTrace blocked my silverblue box accidentally. After our LAN system admins cleared my box from Darktrace, the “connection reset by peer” went away. DarkTrace blocked my box because I did the rebase on the weekend and DarkTrace raised that weekend rebase as a “unusual activity” flag for DarkTrace to block me; it’s a DarkTrace feature, not a bug. Note to self: don’t do rebases/upgrades on the weekend LOL. I figured I should mention it to others as well just in case.

1 Like