Vfio GPU passthrough => Wayland tries to repeatedly load the Nvidia driver

I run Fedora 37 on my Laptop (Acer Aspire V 15 Nitro Black Edition - i7 7700HQ + GTX 1060) and in stock configuration it runs fine.

The problem starts when I want to passthrough the GTX 1060 to my VMs. I use


in /etc/modules-load.d/vfio.conf and

options vfio-pci ids=10de:1c20,10de:10f1

in /etc/modprobe.d/vfio.conf. To activate this configuration I run dracut --force. This way my system starts up with vfio-pci sealing off the Nvidia PCIe devices. Wayland can start up as it still has the integrated Intel GPU.

What’s new now in Fedora 37 (did not happen in F36) is that every time I start an application in Wayland (e.g. konsole in KDE Plasma) the startup takes several seconds and in the journal I see that something tries to load the Nvidia driver. Of course as the Nvidia GPU is sealed with the vfio-pci driver, the driver initialization fails and this whole process seems to block the application startup for a couple of seconds - hence the delay.
If I remove the Nvidia driver from the (host) system the problem disappears.
Does anyone have any idea what is happening and why?
Why and how does Wayland try to load the Nvidia driver at every application startup?
Any idea where I can start searching?