Cannot connect to computers over Anyconnect VPN with fedora 33

Ask Fedora
,
1

Per suggestion from mcatanzaro I’m starting another after posting in another thread this response:

The summary of the above is that I cannot connect to any servers at my company when I am logged into the VPN. I do believe that DNS queries are working since the correct IPs are returned. But if I try to ping any thing in my work domain it doesn’t work. Same for ssh to a work domain server. Even mail and chat servers which are available outside the VPN are not accessible once I log into the Anyconnect VPN.

I’m using Cisco Anyconnect (not by choice) and I am wondering if there is an incompatibility between Anyconnect and systemd-resolved. I would like to know if anyone has been successful using Anyconnect with F33.

mcatanzaro suggested using resolv.conf mode: foreign. I’m not sure how to do this is this setting
dns=default in NetworkManager.conf? Or something else? I did try this and it didn’t make any difference in results. Is there a systemd configuration that I need to change as well?

Mike

2 
resolvectl query target.domain; tracepath -b target.domain; ip route show
3

I think you have an IP routing issue, which is not related to DNS or systemd-resolved. Vladislav’s suggested debugging commands will lead you away from investigating DNS and towards investigating the routing problem.

No, I suggest you stop using resolv.conf mode: foreign. That’s to blame for your strange configuration where you have both global and link-specific DNS servers configured. They are both used for every lookup, and whichever one finishes first wins, which is not what you want. Make sure /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf, as it is by default, and remove anything you added to NetworkManager.conf to get back to default behavior. That said, fixing your DNS configuration will not help fix your routing problem, which is separate.

1 Like
4

Thanks for the suggestions. Here’s the output of the above commands:

$ resolvectl query work.com
work.com: (correct IP address)                     -- link: cscotun0

$ tracepath -b work.com
1?: [LOCALHOST]                      pmtu 1200
1:  send failed
     Resume: pmtu 1200 

$ ip route show
default dev cscotun0 proto unspec scope link 
default via 192.168.0.1 dev enp4s0 proto dhcp metric 100 
4.31.13.10 via 192.168.0.1 dev enp4s0 proto unspec 
192.168.0.0/24 dev cscotun0 proto unspec scope link 
192.168.0.0/24 dev enp4s0 proto kernel scope link src 192.168.0.106 metric 100 
192.168.0.1 dev enp4s0 proto unspec scope link 
192.168.70.0/23 dev cscotun0 proto kernel scope link src 192.168.70.2 
192.168.122.0/24 dev cscotun0 proto unspec scope link 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

This is, of course, logged into the VPN. I’ll reiterate that I’m not a networking expert but the tracepath output clearly indicates a problem.

1 Like
5

No, I suggest you stop using resolv.conf mode: foreign.

Ah, well that is the default configuration at least for my system. I have experimented with different settings but I have reverted back to the default configuration.

Here is the output of resolvectl status before I connect to the VPN:

$ resolvectl status
Global
       Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub                                                 

Link 2 (enp4s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6                                   
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.0.1                                                 
       DNS Servers: 192.168.0.1                                                 

Link 3 (wlp3s0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 5 (virbr0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 6 (virbr0-nic)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

And here it is after connecting to the VPN:

$ resolvectl status
Global
       Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: foreign                                              
Current DNS Server: 192.168.100.1                                        
       DNS Servers: 192.168.100.1 192.168.110.5                          
        DNS Domain: work.com                                            

Link 2 (enp4s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6                                   
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.0.1                                                 
       DNS Servers: 192.168.0.1                                                 

Link 3 (wlp3s0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 5 (virbr0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 6 (virbr0-nic)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 17 (cscotun0)
Current Scopes: LLMNR/IPv4                                                  
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Another strange behavior is that after I disconnect from the VPN. I still cannot connect to work.com public servers (mail, www, etc.). In order to restore this connectivity, I have to restart systemd-resolved.

Here is resolvectl status before I restart systemd-resolved and after disconnecting from the VPN (only showing pertinent interfaces):

$ resolvectl status
Global                                                                                    
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub                                                  
Current DNS Server: 192.168.110.5                                         
       DNS Servers: 192.168.100.1 192.168.110.5                          
        DNS Domain: work.com                                            
                                             
Link 2 (enp4s0)                            
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6                                    
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported        
Current DNS Server: 192.168.0.1                                                                                                                                                     
       DNS Servers: 192.168.0.1

More information: when I am connected to the VPN. Anyconnect overwrites /etc/resolv.conf to be this:

$ cat /etc/resolv.conf
domain work.com
nameserver 192.168.100.1
nameserver 192.168.110.5
search work.com

And I cannot change it because it appears to be constantly overwritten by Anyconnect. After I disconnect from the VPN, the link to /run/systemd/resolve/stub-resolv.conf is restored.

Finally, one other bit of odd/new behavior with Fedora 33. With Fedora 32 I was able to connect to the VPN and also connect to the VPN with my work provided Windows laptop. Since upgrading to Fedora 33, if I am connected to the VPN with my work laptop and I attempt to connect to the VPN with my home Fedora 33 computer, it kills the laptop’s connection to the VPN.

Please let me know if you want to see any of my configuration files.

Thanks for all the help,
Mike

6

The output of tracepath -b work.com was incorrect here is the correct output:

$ tracepath -b work.com
 1?: [LOCALHOST]                      pmtu 1200 
 1:  no reply
 2:  no reply
 3:  no reply
 4:  no reply
 5:  no reply
...
26:  no reply
27:  no reply
28:  no reply
29:  no reply
30:  no reply
     Too many hops: pmtu 1200
     Resume: pmtu 1200
7

And I cannot change it because it appears to be constantly overwritten by Anyconnect. After I disconnect from the VPN, the link to /run/systemd/resolve/stub-resolv.conf is restored.

OK, software that writes to /etc/resolv.conf is super broken, that’s why you’re getting stuck in foreign mode. Is there some reason that you absolutely cannot use Fedora’s built-in Anyconnect support instead (System Settings → Network → VPN → openconnect)? It should hopefully work the same and would certainly not have this problem. If you intend to stick with the proprietary version of Anyconnect, then I would disable DNS on your ethernet link entirely (System Settings → Network → select enp4s0 → toggle off Automatic DNS on the IPv4 and IPv6 tabs) to ensure your DNS goes only to the VPN.

Another strange behavior is that after I disconnect from the VPN. I still cannot connect to work.com public servers (mail, www, etc.). In order to restore this connectivity, I have to restart systemd-resolved.

Huh, OK then you must have some DNS-related problem causing the connectivity issue indeed. But that contradicts your previous evidence:

I do believe that DNS queries are working since the correct IPs are returned. But if I try to ping any thing in my work domain it doesn’t work. Same for ssh to a work domain server.

This pointed us to a routing problem rather than a DNS problem, which is why I said that your issue was surely unrelated to systemd-resolved. But if restarting systemd-resolved has any effect on your problem, you must have a DNS problem. Hence, I doubt it is always returning the correct IPs? Can you confirm that resolvectl query work.com returns the same result before connecting to the VPN, when the VPN is connected, and again after having disconnected from the VPN?

1 Like
8

What subnet does this IP address correspond to?
Typically, it should be among the listed routes.

9

OK, software that writes to /etc/resolv.conf is super broken,…

I assume you are referring to Anyconnect, right? I’ve suspected that Anyconnect doesn’t know how to work with systemd-resolved.

built-in Anyconnect support instead (System Settings → Network → VPN → openconnect)

I don’t see VPN settings although openconnect and NetworkManager-openconnect plugin are installed according to dnf. I am using KDE. Could there be differences?

Can you confirm that resolvectl query work.com returns the same result before connecting to the VPN, when the VPN is connected, and again after having disconnected from the VPN?

Before connecting to the VPN:

$ resolvectl query work.com
work.com: 104.196.xxx.yyy                     -- link: enp4s0

-- Information acquired via protocol DNS in 33.4ms.
-- Data is authenticated: no

While connected to the VPN:

$ resolvectl query work.com
work.com: 104.196.xxx.yyy                     -- link: cscotun0

-- Information acquired via protocol DNS in 132.2ms.
-- Data is authenticated: no

[I will add that while connected to the VPN, ‘ping’ of the above address (not work.com but IP address) fails.]

After disconnecting from the VPN:

$ resolvectl query work.com
work.com: resolve call failed: All attempts to contact name servers or networks failed

Since I performed an upgraded from Fedora 32 to 33 with ‘dnf system-upgrade’ I wonder if something got messed up? Would it be worth installing 33 from an install image?

Thanks again,
Mike

10

What subnet does this IP address correspond to?
Typically, it should be among the listed routes.

Hmm, not sure what you are asking. tracepath does not show this subnet.
My route is through the gateway 192.168.0.1 which is my router. So,

$ ip route show
default via 192.168.0.1 dev enp4s0 proto dhcp metric 100 
192.168.0.0/24 dev enp4s0 proto kernel scope link src 192.168.0.106 metric 100 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

I don’t understand the purpose of the virtual interface virbr0 is there an issue with it?

Thanks,
Mike

11

quote

On my system, when I start the vpn, the metric for the default route similar to what I bolded above is reset to 20600. That ensures all traffic is first sent to the tunnel and only if the tunnel totally fails will it use the local router.

Secondly, it appears the tunnel is assigned 192.168.70.2 and the remote end is 192.168.70.1 I am not sure why the device cscotun0 is assigned routing for 192.168.122.0/24 and 192.168.0.0/24 but routing is definitely your issue with both dns and connecting. It should not take either of those addresses, but appears to be grabbing them and may be the source of the routing problem.
Compare your routing above with mine while my tunnel is active.

$ ip route
0.0.0.0/1 via 10.203.6.1 dev tun0 
default via 192.168.2.1 dev wlp4s0 proto dhcp metric 20600 
10.203.6.0/24 dev tun0 proto kernel scope link src 10.203.6.98 
128.0.0.0/1 via 10.203.6.1 dev tun0 
192.168.2.0/24 dev wlp4s0 proto kernel scope link src 192.168.2.111 metric 600 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown 
212.102.41.32 via 192.168.2.1 dev wlp4s0

There are only 3 entries for the tunnel, and the final entry at the bottom is a host entry for establishing the tunnel. In the process of establishing the tunnel the default route via my router is given an enormously high metric so it is only considered as very last resort for this machine.

This is my routing when the tunnel is disconnected.

$ ip route
default via 192.168.2.1 dev wlp4s0 proto dhcp metric 20600 
192.168.2.0/24 dev wlp4s0 proto kernel scope link src 192.168.2.111 metric 600 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

My suggestion is to figure out what in your vpn config file is causing the vpn to glom onto the local machine addresses and routing things incorrectly. Once the routing is fixed then everything else should work. (the high metric for the default route remains but is never an issue since it is the only default route entry)

12

I figured out how to run Openconnect and resolvectl status after connection is different. It is now this:

$ resolvectl status
Global
       Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub                                                 

Link 2 (enp4s0)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6                                       
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp3s0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 5 (virbr0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 6 (virbr0-nic)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 27 (vpn0)
    Current Scopes: DNS LLMNR/IPv4                                              
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.100.1                                               
       DNS Servers: 192.168.100.1 192.168.110.5                                 
        DNS Domain: work.com

No more foreign mode.

Unfortunately, I still cannot connect to any work.com servers. On the plus side, after closing openconnect I do not have to restart systemd-resolved to restore connectivity to public servers of work.com. Kind of a win, I guess.

13

I don’t have a vpn config file that I can find and I don’t see an option to change the metric. Can you provide additional guidance?

Mike

14

If I run openconnect from the command line the output of resolvectl status is a little different:

$ resolvectl status
Global
       Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub                                                 

Link 2 (enp4s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6                                   
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.0.1                                                 
       DNS Servers: 192.168.0.1                                                 

Link 3 (wlp3s0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 5 (virbr0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 6 (virbr0-nic)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 27 (vpn0)
Current Scopes: none                                                        
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 30 (tun0)
    Current Scopes: DNS LLMNR/IPv4                                              
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.100.1                                               
       DNS Servers: 192.168.100.1 192.168.110.5                                 
        DNS Domain: work.com

Note the DNS Server(s) for enp4s0.

-Mike

15

The IP addresses that the target domain resolves to.

Anyway, this should solve your routing conflicts:

sudo dnf install libvirt-client
sudo virsh net-dumpxml default \
| sed -e "s/192\.168\.122\./192.168.222./g" > /tmp/default.xml
sudo virsh net-undefine default
sudo virsh net-destroy default
sudo virsh net-define /tmp/default.xml
sudo virsh net-autostart default
sudo virsh net-start default
1 Like
16

I don’t know how the KDE VPN settings work, sorry. You could try using nm-connection-editor, which should work.

[I will add that while connected to the VPN, ‘ping’ of the above address (not [work.com](http://work.com/) but IP address) fails.]

After disconnecting from the VPN:

OK, so you have two separate poblems, which is why this was so confusing:

  • You have an IP routing problem when the VPN is active. Vladislav has proposed a workaround for this, which is good, because I would not have known how to help with this. This problem is not related to systemd-resolved.
  • You also had a DNS configuration problem after disconnecting from the VPN. This was an incompatibility between the proprietary Anyconnect client and systemd-resolved. I see you have already resolved this problem by switching to NetworkManager-openconnect, which was the best solution. You’ll now want to ensure /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf to ensure you’re back to Fedora’s default configuration. Notice also that your DNS configuration now only has link-specific DNS servers and no longer has any global servers, meaning you’ll no longer send two redundant racing DNS queries for each hostname to resolve; instead, you’ll send one query to the right place. So it looks like your DNS woes are fully… resolved.

And it looks like Vladislav has found you a workaround for your routing issue. It’s actually surprisingly not uncommon for VPNs to conflict with private addresses. I actually have a routing conflict between my public VPN and certain internal corporate domains. I’ve never seen a conflict with libvirt before, but it’s just bad luck I guess. I don’t know why it only started after upgrade to F33, but it’s not related to systemd-resolved at all. Manual hacks like Vladislav’s proposed workaround are likely the best you can hope for to resolve a routing conflict. (The proper solution to routing conflicts is IPv6, where the address space is so large that conflicts should just never happen. But I guess it will be a while before we can comfortably disable IPv4!)

1 Like
17

Unfortunately this did not work. I am still unable to access any *.work.com servers when I’m connected to the VPN.

18

I’ve been wondering about this conflict since internal my network @home uses the 192.168.0 domain which partially overlaps with the domain at work for their internal domain. I haven’t considered this seriously as an issue because I have an older laptop which still has F29 and it hasn’t been an issue with that computer or with F30-32. If you’re sure this isn’t a systemd-resolved issue, then what else has changed from F32 to F33 that could cause this?

I could change my internal network to 10.0.0. but it’s kind of a pain to ensure all devices reconnect. Plus it hasn’t been a problem with fedora < 33. What are your thoughts on that?

Thanks,
Mike

19

Post the updated routing diagnostics:

ip route show; ip route get target.domain.ip.address
20

If you’re sure this isn’t a systemd-resolved issue, then what else has changed from F32 to F33 that could cause this?

I have no clue what changed, but it’s clearly an IP routing issue, which is not at all related to DNS.

1 Like