Ssh-keyring not really working at login

Ask Fedora
,
23

I don’t really know. It seems to be an issue of whatever is running ssh-add if that’s the case. Otherwise, it might indeed be an ssh-add issue.

The thing that puzzles me the most is the 100% CPU usage. If it:d just failed, then my key was a problem or something. But, when I scanned for it, I sometimes saw around eight processes at the same state.

This was a known issue in fedora 43 beta. Today, it just went away.

I’m sorry. It wasn’t able to pinpoint the exact cause.

24

No worries!

When you see these runaway ssh-add process the next time, you might try running ps -ef|grep -i ssh, then see what the parent process id (PPID) for these ssh-add invocations is, then ps -ef|grep PPID to learn what the process itself is. After that maybe we could figure out why they’re being started in the first place.

With that command, we can also learn whether ssh-agent itself is being run, and from there we can learn what is invoking it, as well.

25

Any time ssh-agent is involved its helpful to check env|grep SSH to see SSH_AGENT_PID in particular. Makes sense to re-check on first login, after ssh, etc. You might have competing agents running and SSH_AGENT_PID will tell you the one your session is currently using.

26

The gcr could be handling the keyring. If you run fuser $SSH_AUTH_SOCK you can get the pid of the process which handles the keyring. For example

$ fuser $SSH_AUTH_SOCK
/run/user/1000/keyring/ssh:  1428

$ ps 1428
    PID TTY      STAT   TIME COMMAND
   1428 ?        Sl     0:00 /usr/bin/gnome-keyring-daemon --daemonize --login

In this case, the ssh keyring is handled by gnome-keyring. On Gnome systems it could instead be handled by gcr.

1 Like
27

Unfortunately, this isn’t universally reliable. I don’t have any keys added to my ssh-agent via ssh-add:

$ ps -ef|grep -i ssh
jrredho     2983    2334  0 08:07 ?        00:00:00 /usr/bin/ssh-agent -D
jrredho    19206    5229  0 16:16 pts/2    00:00:00 grep --color=auto -i ssh
$ env|grep SSH
SSH_AUTH_SOCK=/run/user/1000/ssh-agent.socket
SSH_ASKPASS=/usr/bin/ksshaskpass
$ fuser $SSH_AUTH_SOCK
$

This is for a f43 Plasma Edition system.

28

In that case, you would need to use ssh-add to add the keys to the agent. Appearently, the kde environment doesn’t do that automatically like the gnome environment does.

29

Sure, but you don’t really need to do that unless you’re trying to connect to sites via commands that don’t allow terminal interaction, like doing git commands to github.com. For terminal interactions with ssh, you get prompted for your key’s password on the first connection, and the unlocked key gets automagically added to your ssh-agent.

For the purpose of this thread, I thought what we were trying to do was figure out where several, as yet unexplained, ssh-add commands were getting initiated and why they were hanging and sucking up cpu. If that’s some Gnome keyring manager, then, fair enough, let’s get to part two.

30

For anyone that is still stuck on this problem, ssh-add still hangs and takes up cpu even after updates, use ps -ef|grep -i ssh like jrredho says to check the SSH_AUTH_SOCK against your env
The socket the agent bound to:

image
image1917×165 105 KB

My env:
image
image688×131 53.2 KB

In my case the env was missing a dot before ssh, not sure why this happened because it was all fine before the update to 43

in my case all I needed to do now was to change the env to include that dot, pretty sure this is only band-aid but it works for now.

2 Likes
31

This is exactly the information we needed to see. It looks like a bug to me. I think you should report this as one on the Bugzilla site.

1 Like
32

That’s odd. My env doesn’t include the dot.

renich@desktop fast-kernel$ env | grep SSH
SSH_AUTH_SOCK=/run/user/1000/gcr/
sshSSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass

And I’m pretty sure I didn’t touch that.

Maybe it is fixed by an update…

1 Like
33

I tested a fully upgraded F43 VM with minimal GNOME setup deployed from netinstall, but I’m unable to reproduce the issue.

Apparently you have openssh-askpass installed.
This is a dependency for LXDE/LXQt/Xfce, but not for GNOME:

The default socket works fine in GNOME.
The hidden socket only appears after poking the agent by listing the added keys or attempting to connect to the server.
The variable SSH_ASKPASS is undefined.

> echo ${SSH_AUTH_SOCK}
/run/user/1000/gcr/ssh

> echo ${SSH_ASKPASS}


> ls -a -l ${SSH_AUTH_SOCK%/*}
total 0
drwx------.  2 vgaetera vgaetera  60 Dec  6 19:42 .
drwx------. 13 vgaetera vgaetera 460 Dec  6 19:42 ..
srw-rw-rw-.  1 vgaetera vgaetera   0 Dec  6 19:26 ssh

> ssh localhost exit

> ls -a -l ${SSH_AUTH_SOCK%/*}
total 0
drwx------.  2 vgaetera vgaetera  80 Dec  6 19:43 .
drwx------. 13 vgaetera vgaetera 460 Dec  6 19:43 ..
srw-------.  1 vgaetera vgaetera   0 Dec  6 19:43 .ssh
srw-rw-rw-.  1 vgaetera vgaetera   0 Dec  6 19:26 ssh
1 Like
34

Thanks for the reply,

Apparently you have openssh-askpass installed.

yes, I installed openssh-askpass but this was after the bug happened and I tried to fix it in various ways. Though I did uninstall openssh-askpass and update the system again for a sanity check but ssh-add was still sucking up cpu doing nothing anyways. (env doesn’t match)

The default socket works fine in GNOME.

It might have something to do with moving from 42 to 43, It might also be because I moved the entire /home from old 42 to new 43 (some corruption happened in the upgrade and I had to transplant /home to a new install).

This is exactly the information we needed to see. It looks like a bug to me. I think you should report this as one on the Bugzilla site.

That’s odd. My env doesn’t include the dot.

I’ll do some more investigations but my case might be a bit special and might be user error / very rare case. Thanks again!

35

I don’t know what to think about that.

Nonetheless, I see two things that seem odd to me:

  1. The permissions on the socket named ssh don’t look correct; and
  2. The fact that minimal Gnome seems to include ssh server by default.

My Plasma system without the server running gives:

$ ssh localhost exit
ssh: connect to host localhost port 22: Connection refused
1 Like
37

Is it not still a fact that the socket that is used by both ssh-agent and ssh-add has to be the same. If SSH_AUTH_SOCK is not set to the .ssh socket when the add gets called, then it won’t work correctly. Maybe @ranto1304 is setting this themselves somewhere?

That said, I really did like how you displayed your wizardry in that post via the shell variable expansion/pattern matching thing. :slight_smile:

38

It appears to work in GNOME without exposing the hidden socket to the user’s environment and it matches the agent’s environment:

> sudo awk -v 'RS=\0' -e '/^SSH/' /proc/$(pidof ssh-agent)/environ
SSH_AUTH_SOCK=/run/user/1000/gcr/ssh

Perhaps this is different for users affected by the issue, or there’s a race condition between the keyring components:

> grep -h -e '^Exec=' /etc/xdg/autostart/gnome-keyring-*
Exec=/usr/bin/gnome-keyring-daemon --start --components=pkcs11
Exec=/usr/bin/gnome-keyring-daemon --start --components=secrets
Exec=/usr/bin/gnome-keyring-daemon --start --components=ssh

> pgrep -f -a 'gnome-keyring|gcr'
18948 /usr/bin/gnome-keyring-daemon --daemonize --login
19863 /usr/bin/gnome-keyring-daemon --start --foreground --components=secrets
20222 /usr/libexec/gcr-ssh-agent --base-dir /run/user/1000/gcr
20226 /usr/bin/ssh-agent -D -a /run/user/1000/gcr/.ssh

I guess it should work like this:

  • gcr-ssh-agent is exposed to the user via /run/user/1000/gcr/ssh.
  • ssh-agent is exposed to gcr-ssh-agent via /run/user/1000/gcr/.ssh.
  • gcr-ssh-agent works as a proxy between the user and ssh-agent.
2 Likes
39

In KDE things works differently. It you don’t attempt to run gcr on KDE, everything works after you use the ssh-add command to add the keys. The SSH_AUTH_SOCK should refer to /run/user/1000/ssh-agent.socket.

40

Maybe I’m misremembering this situation, but weren’t the ssh-add’s that were hanging being run at system startup or user login time?

42

Apparently, SSH agent does not directly autostart in GNOME:

  • gnome-session@gnome.target wants gcr-ssh-agent.socket.
  • gcr-ssh-agent.socket triggers gcr-ssh-agent.service.

This expects socket activation by the user and does not load any keys at login.

See also: Socket Activation - systemd for Developers - 0pointer.de

3 Likes
43

Hi, I had the same problem upgrading Fedora from 42 to 43. Searching up the problem I found this solution https://bbs.archlinux.org/viewtopic.php?id=303064

secret-tool store --label="PW id_ed25519" unique ssh-store:$HOME/.ssh/id_ed25519

So it tells gcr-ssh to use the private key.

2 Likes
44

I updated gcr and gcr-libs today gcr-0:4.4.0.1-2.fc43.x86_64 → gcr-0:4.4.0.1-4.fc43.x86_64 and gcr is working normally again and ssh no longer hangs because of it, and without any band-aids like using secret-tool or changing env manually. Anyone that has this problem should update and check if they still have the problem.

1 Like