I think the best solution is to remove the package opensc, which creates conflicts.
I have the following in my .gnupg/scdaemon.conf now:
card-timeout 300
disable-ccid
reader-port Yubico Yubi
That last line is maybe not needed. The first line is more of a preference. The disable-ccid is what matters most. I think that’s my most stable solution. Let me know if this works for you, if not I’ll see if more things are needed.