The Details
I am attempting to perform GPU PCI Pass-through to a QEMU (KVM) Virtual Machine. I did this once on Fedora Workstation, but like all first attempts, my docs were sparse and I didn’t want to mess with it. Fast-forward, now I’m on Fedora Silverblue, and I want to run the VM again and do the PCI Pass-through, but I think there may be some parts I’ll need help with.
I am following this Blog from the last time I did this:
- Blog: mrzk.io - Fedroa 32 and GPU Passthrough VFIO
- Blog: n00bsecurity - Easy GPU Passthrough using KVM on Fedora
The mrzk.io calls for some edits to /etc/modprobe.d/vfio.conf and then a dracut command. However, I don’t think that would “jive” with Silverblue. I also do not want to deviate too far and make too many custom modifications to my install as Silverblue’s ease of updates and upgrades is what makes it appealing to me.
The Question
For Silverblue, how would one handle this? Has anyone else got GPU Pass-through working on Silverblue?
Here is what I have currently…
GPU PCI-Passthrough on Silverblue
References
- ArchLinux Wiki: PCI Passthrough via OVMF
- GitHub: BigOakley/gpu-passthrough
- mrzk.io Blog: Fedora 32 and GPU Passthrough VFIO
- N00b security Blog: Easy GPU Passthrough using KVM on Fedora
Hardware
Primary GPU for Host
Secondary GPU for VM
- Zotac GTX 970 Dual Fan - 145w
Preliminary Checks
Run this grep command to find if your CPU has virtualization enabled.
sudo grep --color --regexp vmx --regexp svm /proc/cpuinfo
If there is no result, make sure to enable VT-d for Intel or AMD-V for AMD based motherboards. Consult your hardware’s instructions on how to do that.
Steps for AMD CPU
Ensure CPU has IOMMU enabled
Run the following command.
dmesg | grep -i -e DMAR -e IOMMU
This should already be enabled for AMD CPUs. Either way, you’ll want to do this.
sudo rpm-ostree kargs \
--append-if-missing="amd_iommu=on" \
--append-if-missing="iommu=pt" \
--append-if-missing="rd.driver.pre=vfio_pci"
Check PCI Bus Groups
#!/usr/bin/env bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;
It should spit out stuff that looks like this…
➜ ~
> bash ./02-check-pci-bus-groups.sh
IOMMU Group 0:
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
[...lots of output...]
IOMMU Group 32:
0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M] [1002:73df] (rev c1)
IOMMU Group 33:
0d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
IOMMU Group 34:
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
0e:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
[...more output...]
You can also get the device IDs using lspci. In this case, I’m looking for my NVIDIA card to pass-through.
➜ ~
> lspci -vnn | grep -i --regexp NVIDIA
0e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) (prog-if 00 [VGA controller])
0e:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
➜ ~
> lspci -vnn | grep -i --regexp Radeon
0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M] [1002:73df] (rev c1) (prog-if 00 [VGA controller])