Virt-manager install experience on Silverblue is lousy

Have you tried Install Virtual Machine Manager on Linux | Flathub with the QEMU extension? (see: GitHub - flathub/org.virt_manager.virt-manager)

Hum, if you need access to USB devices then you indeed need to either layer the packages or use the systemd system extension (sysext). See Overlaying libvirt on Silverblue / Kinoite / Sericea / Onyx and CoreOS for a list of packages to layer.