QEMU/Virt-manager, no Internet when host vpn (Mullvad) is used

Fedora Workstation 40
Mullvad 2024.3
QEMU/virt-manager

I use Mullvad vpn. Normally, once “local network sharing” is enabled in the app, I could keep Mullvad connected when I used a virtual machine for my work. The VM guest traffic was properly routed through Mullvad, which is only installed on the host machine.

Just in the past couple of weeks, I need to disconnect Mullvad in order to have Internet connectivity in my VM guest. I would like to get things back to the previous behavior.

I’ve reached out to Mullvad also, but no solution there yet.

Thanks for any assistance.

go to Settings --Network


check VPN (your Mullvad) :gear: it is check box to use other users

Thanks for the idea, but this is already checked. It was working fine just until recently, when I-don’t-know-what happened.

How do you connect the VM ? Via libvirt NAT or bridge?

Via NAT, libvirt takes care of forwarding and masquerading, so packets should go out along the at that moment highest priority default gateway, VPN. Should just work.

With bridge, you go from the firewalld zone assigned to bridge to the zone assigned to VPN,
so it’s your responsibiliy to setup forwarding policies and masquerading.
So question in that case is whether you have changed something in firewalld config with this as side effect?

A way to start is to set firewall-config runtime LOG_DENIED=ALL, follow syslog with “journalctl -f”, start connection from VM and check for firewall related messages.

Thanks for the reply.
Connection is the default libvirt NAT. It did “just work”, for the last couple years, but now it’s not.
I have not touched my firewalld settings in recent times.
Unfortunately, the troubleshooting step you outlined is a bit beyond me.

Can you send the ouput of

sudo firewall-cmd --get-active-zones

with VPN on. To check whether the virbr0 interface is bound to the correct zone.

Policies for forwarding and NAT are still defined by iptables (nft):
Excerpt:

sudo /sbin/iptables -nL -v

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  276 20252 LIBVIRT_FWX  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
  276 20252 LIBVIRT_FWI  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
  138  9616 LIBVIRT_FWO  0    --  *      *       0.0.0.0/0            0.0.0.0/0           


Chain LIBVIRT_FWI (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   39  2972 ACCEPT     0    --  *      virbr0  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
    0     0 REJECT     0    --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   39  2972 ACCEPT     0    --  virbr0 *       192.168.122.0/24     0.0.0.0/0           
    0     0 REJECT     0    --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject

This means that traffic from virbr0 with source address 192.168.122.0/24 is allowed to be forwarded, and related/established traffic to virbr0 and to address 192.168.122.0/24 is allowed.
Everything else is blocked, so you have a safe firewall between the VM and rest of the world, including LAN.

Excerpt:

sudo iptables -t nat -n -L -v

Chain POSTROUTING (policy ACCEPT 486 packets, 64770 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  642 75754 LIBVIRT_PRT  0    --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_PRT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     0    --  *      *       192.168.122.0/24     224.0.0.0/24        
    0     0 RETURN     0    --  *      *       192.168.122.0/24     255.255.255.255     
    0     0 MASQUERADE  6    --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
   56  4256 MASQUERADE  17   --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
    1    84 MASQUERADE  0    --  *      *       192.168.122.0/24    !192.168.122.0/24

Everything with source address in the 192.168.122.0/24 range will get the source address of the outgoing interface, wherever it may go.

In this configuration, I’ve a VM with Protonvpn up, and a traceroute shows that it works as desired.
May be you can try the two sudo iptables commands and see whether you get similar output in the listing.

In addition, you can check routing:

#Without VPN
ip route get 1.1.1.1 from 192.168.122.1
1.1.1.1 from 192.168.122.1 via 192.168.2.254 dev bridge0 uid 0 
    cache 
#With VPN
ip route get 1.1.1.1 from 192.168.122.1
1.1.1.1 from 192.168.122.1 via 10.96.0.1 dev tun0 uid 0 
    cache

Here’s the output of sudo firewall-cmd --get-active-zones

libvirt
interfaces: virbr0
public (default)
interfaces: wlo1

The rest of what you said, I simply don’t understand.

That’s fine. So the virbr0 interface is connected to the correct zone. Then sorry, I do not know how to debug this further by forum. May be someone else has idea’s.

On my F40 system, a VM connected via NAT takes the normal way without VPN and the VPN route with a (free) ProtonVPN connection enabled.

Let’s make sure the problem is not related to DNS.
Check from the guest before and after connecting the VPN:

nslookup example.org
ping 1.1.1.1
ping 8.8.8.8

Reboot the guest to flush its DNS cache before the second test.

The nslookup and ping results are functioning , whether the vpn is on or not. Still there is no Internet connectivity via a browser or trying to check for upgrade on guest system, etc.

Good idea to check DNS. There is one thing which worries me. The libvirt starts the “dnsmasq” program which is a DNS cache and DHCP server. The cache implies that if you lookup only example.com, once found it will always find example.com until cache expires.

It would be interesting to know what a real life internet connection gives as error message,
so with VPN on e.g. “sudo dnf --refresh update” in a terminal.

The counter just reached zero: in my configuration with my ISP’s DNS servers,
example.com has a lifetime of currently 3266 seconds in cache.

With the host vpn on, a vm guest “dnf --refresh update” simply cannot connect to the servers.

So ping to 1.1.1.1 and 8.8.8.8 works, so there is a sign of connectivity. If DNS fails, one would expect messages that servers cannot be found, from your response I assume dnf just hangs.
Alternative e.g. “curl https://google.com” should output some html code.
What is the protocol used? I’ve seen Mullvad uses Wireguard with Linux. I’ve tested only OpenVPN now, but in theory this should not make difference. If Wireguard, what is the output of command
“sudo wg” with public key an peer id obfuscated?

Just a wild guess: are there Mullvad extensions active like multihop which could confuse the NAT connection tracking?

Edit: should not happen, For normal connection too any packet is identified by source address/port and destination address/port.
I assume internet from VM host is working flawless via VPN.

Go to Settings --Network – VPN – your connection – open :gear: check it is ipv6 method set to automatic if yes disable it save and try to use internet to your vm

Yes, dnf just hangs.

curl https://google.com also hangs

I’m using Wireguard.

Multihop is not enabled

Update as I write this

I just checked the Mullvad wireguard “obfuscation” setting. If automatic or off, I have no Internet in my VM. If on, I do have Internet in my VM.

I’ll attempt to repeat my results and will inform Mullvad.

Here is Mullvad’s note on obfuscation:

WireGuard out of the box works only over UDP. This can cause problems because UDP is blocked on many public networks like in cafes and on trains. If that happens then you can set Obfuscation to On (UDP-over-TCP).

Note that using WireGuard Obfuscation (UDP-over-TCP) can significantly increase the latency and also slow down the Internet speed. So if you don’t need it then do not turn it on.

So, I have no idea why my VM internet was working fine until recently, but it seems at least that we’re closing in on the cause.

If I chose IPv4 exclusively, it makes no difference, only the “obfuscation” setting I mentioned.

Thank you all for your help with the troubleshooting! I didn’t know if it was an issue with QEMU or Mullvad but at least I have a workaround and a point of discussion with Mullvad.

Interesting, please let know what’s Mullvad response is. Wireguard is UDP only, so some tunnel mechanism should be used to encapsulate this in TCP. OpenVPN has this option natively. But at the very end, one would expect the same output on the Wireguard interface, so why this interference with the libvirt NAT ???
ping works, so this extra encapsulation could be a MTU, maximum transmission unit, issue but why only on the VM?

I had a similar problem with hanging TCP connections on libvirt guests when using own VPN on the VPS and solved it with TCP MSS clamping.

Based on your active zones, it should look like this:

sudo firewall-cmd --permanent --zone=libvirt \
    --add-rich-rule="rule tcp-mss-clamp value=pmtu"
sudo firewall-cmd --reload

Something to study. The only thing I can say is that I have no problem with VM and openvpn/tcp to protonvpn. Unfortunately, they only support Wireguard/tcp on android.

The wireguard developers explicitely state to support only udp because tcp in tcp can lead to exploding timeouts. But VPN providers like to mimic the VPN as HTTPS traffic.

The tcp-mss-clamp rich rule lands in a forward rule, so apparently this is only to fix some device behind the Linux system including a VM in this case.

And I’ve to check why much bigger MSS TCP packets are in Wireshark while ethool shows offloading off. Unless the driver fools me by assembling bigger packets from mtu size ones.