Dual-GPU Passthrough (Linux host & Windows 10 VM)

On this page I'm describing the process of setting up a Windows 10 VM with a dual-GPU passthrough on a linux machine (which worked for my hardware setup).


Steps:

  1. IOMMU Setup
  2. GPU Isolation
  3. Isolation check and final configs
  4. Win10 VM Creation & Optimization (WIP)

Used hardware:


Used software:

QEMU/KVM and virt-manager packages:

sudo apt install qemu-kvm virt-manager virtinst libvirt-clients bridge-utils libvirt-daemon-system


Resources:

1. IOMMU Setup

Enabling IOMMU

First please make sure that you have IOMMU and Virtualization features ("VT-d","AMD-Vi" or "Virtualization") enabled in your BIOS (and that your motherboard supports it). Once enabled you can reboot and return to your linux host OS.


Back in your linux host you have to enable IOMMU in GRUB too:

sudo nano /etc/default/grub

Open the grub file with your preferred editor (in this example 'nano') as a superuser and change the line GRUB_CMDLINE_LINUX_DEFAULT:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=1 amd_iommu=on splash"

For AMD CPU add iommu=1 and amd_iommu=on.


Save the grub file and update GRUB:

sudo update-grub


Reboot your machine and after that check with sudo dmesg | grep IOMMU that IOMMU was correctly enabled. If all went well, you should see a message in the logs that IOMMU was initialized:

AMD-Vi: AMD IOMMUv2 loaded and initialized

IOMMU Grouping

Paste the below script to your terminal:

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;

It will output a list of IOMMU groups with your PCI hardware (if it doesn't return anything either you have IOMMU not enabled or your hardware doesn't support it)

- Example of an IOMMU groups list (.txt)


In the list of IOMMU groups check your VGA and Audio devices:

IOMMU Group 20:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Polaris 20 XL [Radeon RX 580 2048SP] [1002:6fdf] (rev ef)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]

IOMMU Group 25:
0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] [1002:73ff] (rev c7)

IOMMU Group 26:
0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]


Very important in this step is to check if both VGA and Audio devices are either both or each of them in a separate IOMMU group, i.e. they are not grouped together with other PCI devices like USB controllers, PCI bridge etc.)

If VGA and Audio devices are grouped with other PCI devices then it's not possible to do a GPU passthrough without additional kernel modifications (ACS override patch).

(Even if an ACS override might 'fix' the grouping issue, it's still not recommended as it might damage the hardware.)


Once you've found a suitable VGA/Audio pair to pass through, note down their hardware IDs: 1002:73ff,1002:ab28 - I'll be using my RX 6600 (group 25) and its HDMI audio device (group 26).

2. GPU isolation

In the following steps it's required to create/edit a few necessary files through the terminal for the GPU isolation process.


INITRAMFS-TOOLS (Ubuntu based)

Run the next command to edit the /etc/initramfs-tools/modules file:

sudo nano /etc/initramfs-tools/modules

Then add the following lines and replace the IDs in bold with your own IDs which you've noted down before:

softdep amdgpu pre: vfio vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=1002:73ff,1002:ab28
vfio_pci ids=1002:73ff,1002:ab28
vfio_pci
amdgpu

Save the the file and exit.


Open the next file:

sudo nano /etc/modules

Add the following lines (again replace the bold IDs with your own IDs:

vfio
vfio_iommu_type1
vfio_pci ids=1002:73ff,1002:ab28

Save the the file and exit.


MODPROBE.D

Create the next file (amdgpu.conf) in the modprobe.d folder:

sudo nano /etc/modprobe.d/amdgpu.conf

Add the following lines to the file:

softdep amdgpu pre: vfio vfio_pci


At last, create the next file for VFIO (vfio_pci.conf) in the same modprobe.d folder:

sudo nano /etc/modprobe.d/vfio_pci.conf

Add the following lines to the file - replace the IDs here as well with your own:

options vfio_pci ids=1002:73ff,1002:ab28


Finally, update/rebuild your kernel (initramfs-tools) with the next command:

sudo update-initramfs -u

Reboot your machine.

3. ISOLATION CHECK & FINAL CONFIGS

To check if your GPU (VGA & HDMI Audio) are properly isolated, run the command:

lspci -vnn

In case of a successful isolation both devices should have Kernel driver in use: vfio-pci.

(Note: It doesn't matter if the line below says: Kernel modules: amdgpu)


My output (trimmed down only to important lines):
0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] [1002:73ff] (rev c7) (prog-if 00 [VGA controller])
[...]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu


If the lspci / GPU isolation check doesn't output vfio-pci but instead outputs Kernel driver in use: amdgpu, then you need to re-check if all steps until now were done correctly (check for possible typos). Additionally other issues can prevent VFIO from loading (which would require further investigation).


Only proceed with the next steps (VM creation & optimization) if the GPU isolation check passed.


APPARMOR CONFIG

In some cases Ubuntu's build-in kernel security feature - AppArmor - can prevent a QEMU based virtual machine from working properly. To remove such restrictions it's required to edit the following file:

sudo nano /etc/apparmor.d/abstractions/libvirt-qemu

Modify the following lines under # for usb access:

/dev/bus/usb/** rw,
/etc/udev/udev.conf r,
/sys/bus/ r,
/sys/class/ r,
/run/udev/data/* rw,

Scroll down to the end of the file and add the next lines (which will be required to setup the Looking Glass virtual display).

# Looking Glass
/dev/shm/looking-glass rw,


Save the the file and restart apparmor:

sudo systemctl restart apparmor