Fedora 35. Plasma 5.23.3. Nvidia video. All up to date.
In a X11 session the task manager works as expected but in Wayland it takes ages to appear after login and when it does it also takes ages to respond to clicks. The “file system unavailable” (or similar) message keeps showing. The applications behavior is normal, like Firefox, Terminal and others.
Wayland support for Nvidia is still immature, most of the needed stuff is still missing.
There is a beta drive 495.44, you could try that one and see if it will fix the issues you are having or just stay on X11 until Nvidia support for Wayland can become better.
Does this happen only when you open Task Manager or any other applicaton?
Maybe this could be a KDE bug in Wayland as well.