Note that virt-manager is essentially a libvirt client.
The server and client can run on different hosts.
I install libvirt from scratch like this:
sudo dnf install \
libvirt-daemon-driver-network \
libvirt-daemon-driver-nodedev \
libvirt-daemon-driver-qemu \
libvirt-daemon-driver-storage-core \
qemu-audio-spice \
qemu-char-spice \
qemu-device-display-qxl \
qemu-device-display-virtio-gpu \
qemu-device-display-virtio-vga \
qemu-device-usb-redirect \
qemu-system-x86-core
sudo systemctl --now enable virtnetworkd.service
sudo systemctl --now enable virtqemud.service
That should be the bare minimum you need.