Nested Virtualization

Our Neighbors Asahi Linux Asahi General
,
1

Hi,
I recently got my hands on an M2-series mac and installed Asahi Linux on it. So far I’ve been really impressed and would like to say thanks to everyone that has made this possible!

I got the M2 because I’d read that it actually supports nested virtualization, so I’ve been playing around to see if it works. It’s early days but I believe I have something working so I thought I’d share the steps I’ve taken in case anyone else is interested.

  1. Make sure you’re running Linux kernel 6.16+. There are some talks from Marc Zyngier about a major arm64 kvm refactor that was merged into 6.16; you’ll need this change.
  2. You need an M2 series mac (ARMv8.6-A); I don’t know if the changes in the 6.16 kernel support nested virtualization in the M1 series (ARMV8.4-A?).
  3. Download and build the latest version of QEMU (10.2.0 at this time). The default version I was able to get through dnf is 9.2.4, which seems to be missing support for some arm64 kvm virtualization.
  4. You need to add an argument (kvm-arm.mode=nested) to the kernel commandline. I’m not a grub expert but I did this by updating GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, then running sudo grub2-mkconfig -o /etc/grub2.cfg.
  5. Reboot and you should see something like this in dmesg: kvm [1]: VHE+NV2 mode initialized successfully
  6. Update your qemu commandline to include -M virt,accel=kvm,virtualization=on

I ran this with an Ubuntu 25.10 vm (kernel 6.17.0) and got this in the vm’s dmesg:

kvm [1]: nv: 567 coarse grained trap handlers
kvm [1]: nv: 664 fine grained trap handlers
kvm [1]: PAGE_SIZE not supported at Stage-2, giving up

I assume this has something to do with 16k page sizes but haven’t dug into it yet. I also tried running a Windows 11 arm64 vm with nested virtualization but it hung at bootmgr.

1 Like
2

Updating the Ubuntu 25.10 kernel to use 16k pages seems to have worked! Here’s the dmesg output for kvm now:

[    0.019049] kvm [1]: nv: 567 coarse grained trap handlers
[    0.019096] kvm [1]: nv: 664 fine grained trap handlers
[    0.019127] kvm [1]: IPA Size Limit: 42 bits
[    0.019166] kvm [1]: GICv3: no GICV resource entry
[    0.019182] kvm [1]: disabling GICv2 emulation
[    0.019199] kvm [1]: GIC system register CPU interface enabled
[    0.019213] kvm [1]: vgic interrupt IRQ9
[    0.019232] kvm [1]: VHE mode initialized successfully

The rough steps I took to do this (all in the Ubuntu 25.10 vm):

  1. Get the source for current kernel
  2. Copy the current kernel config into the kernel source directory (eg cp /boot/config-6.17.0-8-generic .config)
  3. Modify the config to use 16k pages. I diff’d the current config against the (Ubuntu) Asahi Linux kernel config to figure out what needed changing.
  4. Remove some Ubuntu-specific config options (CONFIG_SYSTEM_TRUSTED_KEYS and CONFIG_SYSTEM_REVOCATION_KEYS)
  5. Direct the build to use the existing config present in the directory (make olddefconfig)
  6. Install many build deps (see one of the guides for building kernel source for the full list)
  7. Build the kernel (eg make -j2 olddefconfig bindeb-pkg LOCALVERSION=-stock)
  8. Once building has finished, install the new kernel build (eg cd .. && sudo dpkg -i linux-headers-6.17.2-stock_6.17.2-10_arm64.deb linux-libc-dev_6.17.2-10_arm64.deb linux-image-6.17.2-stock_6.17.2-10_arm64.deb). I believe that adds a new boot option but ran sudo update-grub anyway.
  9. Reboot

Here’s the (massaged) diff for what I changed in the config (new vs old, with new first):

< CONFIG_ARM64_CONT_PTE_SHIFT=7
< CONFIG_ARM64_CONT_PMD_SHIFT=5
< CONFIG_ARCH_MMAP_RND_BITS_MIN=16
< CONFIG_ARCH_MMAP_RND_BITS_MAX=31
< CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=9
< # CONFIG_ARM64_4K_PAGES is not set
< CONFIG_ARM64_16K_PAGES=y
< CONFIG_ARCH_FORCE_MAX_ORDER=11
< CONFIG_ARCH_MMAP_RND_BITS=16
< CONFIG_HAVE_PAGE_SIZE_16KB=y
< CONFIG_PAGE_SIZE_16KB=y
< CONFIG_PAGE_SHIFT=14
< CONFIG_PAGE_BLOCK_MAX_ORDER=11
---
> CONFIG_ARM64_CONT_PTE_SHIFT=4
> CONFIG_ARM64_CONT_PMD_SHIFT=4
> CONFIG_ARCH_MMAP_RND_BITS_MIN=18
> CONFIG_ARCH_MMAP_RND_BITS_MAX=33
> CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=11
> CONFIG_ARM64_4K_PAGES=y
> # CONFIG_ARM64_16K_PAGES is not set
> CONFIG_ARCH_FORCE_MAX_ORDER=13
> CONFIG_ARCH_MMAP_RND_BITS=33
> CONFIG_HAVE_PAGE_SIZE_4KB=y
> CONFIG_PAGE_SIZE_4KB=y
> CONFIG_PAGE_SHIFT=12
> CONFIG_PAGE_BLOCK_MAX_ORDER=13

This answer was handy for building kernel source.

3

Hello Justin,

Thank you very much for your detailed document on this topic. I was trying to do the same thing and came across your post. I have some questions regarding your setup though.

The machine I’m working on doesn’t have an Apple M2 CPU, instead it has an Nvidia Grace (with Neoverse core).

I have updated the machine to use Kernel 6.16, and added kvm-arm.mode=nested to its boot command. I also updated the machine to use QEMU 10.1.2 which should have support for nested KVM in ARM processor (QEMU version 10.1.0 released - QEMU)

The dmesg log on my box shows the following when booting:

[    1.598967] kvm [1]: nv: 566 coarse grained trap handlers
[    1.599025] kvm [1]: IPA Size Limit: 48 bits
[    1.605379] kvm [1]: GICv4 support disabled
[    1.605380] kvm [1]: GICv3: no GICV resource entry
[    1.605381] kvm [1]: disabling GICv2 emulation
[    1.614095] kvm [1]: GIC system register CPU interface enabled
[    1.614111] kvm [1]: vgic interrupt IRQ9
[    1.640248] kvm [1]: VHE+NV2 mode initialized successfully

Note that we have the VHE+NV2 mode initialized successfully, same as what you had.

However, when I tried to run qemu-system-aarch64 with virtualization=on, the command would hang. Searching around, it seems this line GICv3: no GICV resource entry might be the reason.

I saw that you have that line as well, but it’s from dmesg of the L1 host instead. Can you share the dmesg of your host (ie. the L0 machine) so I can check if GICv3: no GICV resource entry is the cause or not?

Thank you very much in advance

4

Interesting! Here are the dmesg logs from my fedora (ie the bare metal) host:

[    0.054886] kvm [1]: nv: 567 coarse grained trap handlers
[    0.054950] kvm [1]: nv: 664 fine grained trap handlers
[    0.054995] kvm [1]: IPA Size Limit: 42 bits
[    0.055003] kvm [1]: Non-architectural vgic, tainting kernel
[    0.055004] kvm [1]: GICv3: no GICV resource entry
[    0.055005] kvm [1]: disabling GICv2 emulation
[    0.055005] kvm [1]: GICv3 with broken locally generated SEI
[    0.055006] kvm [1]: GICv3 sysreg trapping enabled ([G0G1D], reduced performance)
[    0.055034] kvm [1]: GIC system register CPU interface enabled
[    0.055038] kvm [1]: vgic interrupt IRQ33
[    0.055060] kvm [1]: VHE+NV2 mode initialized successfully

In case it’s useful, here’s my QEMU commandline:

qemu-system-aarch64 \
    -display sdl,gl=on \
    -cpu host \
    -M virt,accel=kvm,virtualization=on \
    -enable-kvm \
    -m 8G \
    -smp 2 \
    -hda ubuntu25.qcow2 \
    -device qemu-xhci \
    -device ramfb \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -audio driver=pipewire,model=virtio \
    -device usb-kbd \
    -device usb-tablet \
    -bios /usr/share/edk2/aarch64/QEMU_EFI.fd \
    -nic user,model=virtio-net-pci
5

Thank you very much for the speedy reply Justin.

This is enlightening. For your case some dmesg lines are different, but you do also have GICv3: no GICV resource entry so I think that’s not the reason.

I’ll try your qemu-system-aarch64 command and report back if that works

Cheers

6

FYI @juzzkidd , I tried the QEMU command that you gave but unfortunately that didn’t work for me.

Will be experimenting more on this front and will share any knowledges that I learn in case other people ever find this thread.

Cheers