How to troubleshoot Cockpit webservice and PAM stack?

Ask Fedora
1

On a new server installation, I’ve added Cockpit. Started working fine out of the box for localhost. I worked to lock down access VIA one IP on my LAN, and ended up with troubles. Following the suggestion of another user here at FF (1), I removed the IP binding. Now I can see the login screen on both localhost and remote, but after login now I get a White Screen of Death.

journalctl shows an error with pam and the ssh keys. I added SSH keys in the Cockpit user settings, and I’m not sure if this has changed login. I’m not sure how to access SSH Cockpit User settings from the CLI to investigate this further. (2)

Here is the output from journalctl pointing to PAM and seemingly exonerating SELinux:

[root@[my-machine] [my-user]]# firewall-cmd --list-all | grep cockpit
  services: cockpit dhcpv6-client ssh

[root@[my-machine] [my-user]]# systemctl status cockpit.socket
● cockpit.socket - Cockpit Web Service Socket
   Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/cockpit.socket.d
           └─listen.conf
   Active: active (listening) since Thu 2019-11-14 08:33:51 EST; 21h ago
     Docs: man:cockpit-ws(8)
   Listen: [::]:9090 (Stream)
    Tasks: 0 (limit: 23256)
   Memory: 940.0K
   CGroup: /system.slice/cockpit.socket

Nov 14 08:33:51 [my-machine] systemd[1]: Starting Cockpit Web Service Socket.
Nov 14 08:33:51 [my-machine] systemd[1]: Listening on Cockpit Web Service Socket.

[root@[my-machine] [my-user]]# journalctl -b -u cockpit
Nov 14 08:31:14 [my-machine] systemd[1]: Starting Cockpit Web Service...
Nov 14 08:31:14 [my-machine] systemd[1]: Started Cockpit Web Service.
Nov 14 08:31:14 [my-machine] cockpit-ws[4291]: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert
Nov 14 08:31:36 [my-machine] cockpit-session[4344]: pam_ssh_add: Failed adding some keys
Nov 14 08:31:36 [my-machine] cockpit-session[4344]: pam_unix(cockpit:session): session opened for user [my-user] by (uid=0)
Nov 14 08:31:36 [my-machine] cockpit-ws[4291]: logged in user session
Nov 14 08:31:36 [my-machine] cockpit-ws[4291]: received request from bad Origin: http://localhost:9090
Nov 14 08:31:36 [my-machine] cockpit-ws[4291]: Received invalid handshake request from the client
Nov 14 08:31:36 [my-machine] cockpit-ws[4291]: WebSocket from ::1 for session closed
Nov 14 08:31:51 [my-machine] cockpit-ws[4291]: session timed out
Nov 14 08:33:55 [my-machine] systemd[1]: Starting Cockpit Web Service...
Nov 14 08:33:55 [my-machine] systemd[1]: Started Cockpit Web Service.
Nov 14 08:33:55 [my-machine] cockpit-ws[4557]: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert
Nov 14 08:34:06 [my-machine] cockpit-session[4564]: pam_ssh_add: Failed adding some keys
Nov 14 08:34:06 [my-machine] cockpit-session[4564]: pam_unix(cockpit:session): session opened for user [my-user] by (uid=0)
Nov 14 08:34:06 [my-machine] cockpit-ws[4557]: logged in user session
Nov 14 08:34:06 [my-machine] cockpit-ws[4557]: received request from bad Origin: http://localhost:9090
Nov 14 08:34:06 [my-machine] cockpit-ws[4557]: Received invalid handshake request from the client
Nov 14 08:34:06 [my-machine] cockpit-ws[4557]: WebSocket from ::1 for session closed
Nov 14 08:34:22 [my-machine] cockpit-ws[4557]: session timed out
Nov 14 09:52:01 [my-machine] systemd[1]: Starting Cockpit Web Service...
Nov 14 09:52:01 [my-machine] systemd[1]: Started Cockpit Web Service.
Nov 14 09:52:01 [my-machine] cockpit-ws[5488]: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert
Nov 14 09:52:01 [my-machine] cockpit-ws[5488]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 14 09:52:01 [my-machine] cockpit-ws[5488]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 14 09:52:01 [my-machine] cockpit-ws[5488]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 14 09:52:01 [my-machine] cockpit-ws[5488]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:01:55 [my-machine] systemd[1]: Starting Cockpit Web Service...
Nov 15 06:01:55 [my-machine] systemd[1]: Started Cockpit Web Service.
Nov 15 06:01:55 [my-machine] cockpit-ws[14409]: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert
Nov 15 06:01:55 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:01:55 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:01:59 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:01:59 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:01:59 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:01:59 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:01:59 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:02:07 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:02:07 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:02:07 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:02:07 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:02:19 [my-machine] cockpit-session[14416]: pam_ssh_add: Failed adding some keys
Nov 15 06:02:19 [my-machine] cockpit-session[14416]: pam_unix(cockpit:session): session opened for user [my-user] by (uid=0)
Nov 15 06:02:19 [my-machine] cockpit-ws[14409]: logged in user session
Nov 15 06:02:19 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:02:19 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:02:19 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:02:20 [my-machine] cockpit-ws[14409]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate
Nov 15 06:02:20 [my-machine] cockpit-ws[14409]: received request from bad Origin: https://192.168.0.5:9090
Nov 15 06:02:20 [my-machine] cockpit-ws[14409]: Received invalid handshake request from the client
Nov 15 06:02:20 [my-machine] cockpit-ws[14409]: WebSocket from 192.168.0.6 for session closed
Nov 15 06:02:35 [my-machine] cockpit-ws[14409]: session timed out

[root@[my-machine] [my-user]]# ausearch -m AVC -ts recent
<no matches>

How do I troubleshot Cockpit Authentication?

“By default the cockpit web service is installed on the base system and socket activated by systemd. In this setup access is controlled by a cockpit specific pam stack, generally located at /etc/pam.d/cockpit . By default this is configured to allow you to login with the username and password of any local account on the system…” (3)

One guide notes to run this command:

# ss -tunlp | grep cockpit

This does not produce output. Looking without the grep I see why,

tcp   LISTEN  0       128                    *:9090                *:*      users:(("systemd",pid=1,fd=201))

The example gives the additional tuple, users:(("cockpit-ws", pid=XX, fd=xx), ("systemd" ...

Also, this command is missing cockpit-ws line as well:

ps auxf | grep cockpit
root     25738  0.0  0.0  12112  1076 pts/0    S+   21:17   0:00                      \_ grep --color=auto cockpit

  1. HTTPS/ask.fedoraprojectDOTorg/t/is-there-a-non-obvious-effect-of-changing-network-from-dhcp-to-static-for-cockpit-service/4134

  2. HTTP/githubDOTcom/cockpit-project/cockpit/wiki/Config-format-for-known-machines-and-ssh-keys

  3. HTTPS/cockpit-projectDOTorg/guide/0.82/authentication.html