SSH "agent refused operation" with non-resident PIN-protected key

I’m trying to configure my Yubikey to work with SSH, but my key seems to be giving it problems. I generated my key with ssh-keygen -t ed25519-sk -O verify-required, replacing my old key so that it requires a PIN every use. However, now every time I try and use the key, it throws an error:

sign_and_send_pubkey: signing failed for ED25519-SK "/var/home/morgan/.ssh/id_ed25519_sk" from agent: agent refused operation

This was happening when I tried using a resident key as well, but it did Not happen when I used a non-resident key that didn’t require a PIN. And I double-checked, my permissions are set properly on the key and .ssh directory.I also tried adding IdentityFile ~/.ssh/id_ed25519_sk to .ssh/config. While that changed nothing, explicitly passing -i .ssh/id_ed25519_sk to ssh does work, but still say agent refused operation beforehand.

I have somewhat of a workaround: after running eval $(ssh-agent -s), the key will work properly for that session, prompting for the PIN and all. However, this only lasts to the end of the shell session, and it doesn’t help at all with GUI applications (e.g. VSCode git integration), so its utility is limited. I could find a way to run this on startup every time, but this feels like a hacky solution; I want to properly fix the issue. I’d also prefer not to go back to allowing access without a PIN, but we’ll see if I have to.

I’m running Fedora Silverblue 38, with OpenSSH9.0. The same error occurs on Workstation as well, I checked.

I don’t think you need or should use an ssh-agent with Yubikey based SSH keypairs.

You can try it with:

$ ssh ..... -o IdentityAgent=none