Fedora 40 - Split DNS Resolution(Internal/External) Issue - Favortizing External Resolution

Ask Fedora
, , , , , ,
1

I have multiple DNS entries defined both on my internal (LAN) and external (Hosted) DNS servers to allow ability to use certain devices/application irrelevant of what network I’m connected to; running both IPv4 and IPv6. This works 100% as anticipated both on windows and macs; yet the experience seems to be different with Fedora and I cannot figure out why.

All devices hit the same internal (LAN) DNS servers and the mac and apple devices return the appropriate internal DNS resolution and all connections work fine.

However, it seems fedora prioritizes the external resolution as any time I ping, ssh, etc. it attempts to use the external resolution.

When I run resolvectl status
Current DNS Server, DNS Servers, and DNS domain all present correctly to the internal definitions.

When I run nslookup for the devices where external resolution is being picked up as the priority the external CNAME or external IP entries show up AFTER the internal IP I would expect to be used.

My network configuration on the Fedora machine is nothing customized, followed the WiFi wizard and connected to my WiFi.

Any guidance would be much appreciated.

2

Added dns, f40, ipv4, ipv6, network, networking

3

How have you configure DNS entires for your internal network?
Did you edit any of the /etc config files to set this up?
Are you relying on the DHCP to set the DNS settings?

What is the output of resolvectl status?

1 Like
4

Yes. As mentioned all DNS entries internal and external function and everything that uses them from both a mac and win machines works as expected. Yes, relying on DHCP to set them and it’s defining as expected. It’s just how fedora seems to be resolving is the issue.

resolvectl status output (scrubbed actual IP and domain):

Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 3 (wlp9s0f0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: INTERNALIPV6_IP
       DNS Servers: INTERNALIPV4_IP INTERNALIPV6_IP
        DNS Domain: mydomain.com
5

All the names you resolve come from your internal DNS server?
That DNS server is programmed to ask external for names it does not know?

You can test this two ways.
Using the resolvectl query command to see how a DNS lookup is resolved.
Using the dig command to directly query your DNS server.
To query from a specific server use the dig @server name syntax.

What do you find out with these queries?

1 Like
6

Yes, resolve is going against my internal DNS (Unifi gateway) and this is why it’s baffling me that it works correctly on other devices (MACs & PCs), except fedora. All the other devices when I issue a dig or nslookup do not provide back the external DNS entries and only the internal.

dig example(IP Obfuscated):

dig servername

; <<>> DiG 9.18.28 <<>> servername
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53409
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;servername. IN A

;; ANSWER SECTION:
servername 30 IN A 10.XXX.XXX.XXX

;; Query time: 129 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sat Aug 03 14:41:53 EDT 2024
;; MSG SIZE rcvd: 58

nslookup servername
(IPs Obfuscated)

nslookup servername
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: servername.domain.com
Address: 10.XXX.XXX.XXX
Name: servername.domain.com
Address: 2606:XXXX:XXXXX
Name: servername.domain.com
Address: 2606:XXXX:XXXXX

The two 2606 addresses are external IPv6 addresses not the internal IPv6 addresses

resovlectl query servername

resolvectl query servername
homebridge: 10.XXX.XXX.XXX – link: wlp9s0f0
2606:XXXX:XXXX – link: wlp9s0f0
2606:XXX:XXXX – link: wlp9s0f0
(servername.domain.com)

– Information acquired via protocol DNS in 25.3ms.
– Data is authenticated: no; Data was acquired via local or encrypted transport: no
– Data from: network

10.XXX is internal DNS
2606 enteries are external DNS

8

My apologies, for missing that element.

Here are the results against the primary internal DNS server provided via DHCP.

dig @primary_lan_dnsip servername

; <<>> DiG 9.18.28 <<>> @primary_lan_dnsip servername
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28123
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;homebridge. IN A

;; ANSWER SECTION:
servername. 0 IN A 10.XXX.XXX.XXX

;; Query time: 9 msec
;; SERVER: LANDNSIP#53(LANDNSIP) (UDP)
;; WHEN: Sat Aug 03 17:08:28 EDT 2024
;; MSG SIZE rcvd: 55

Resolves back to the expected server LAN IP.

10

Both of those are the same internal IP for the dns server. 10.0.10.1

12

I’m a bit confused, but it looks like the internal DNS correctly resolves the name to the IPv4 address, but for the IPv6 address, it queries the external DNS, and the external IPv6 address is returned. It looks like that, no?

And maybe, Windows and Mac OS prioritizes IPv4 while Fedora or Linux in general prioritizes IPv6, so ping and ssh attempts to use the IPv6 address, that is only external.

Maybe I wrong.

13

I was thinking the same thing but the odd thing is I do have one host which is only IPv4 both internal and external but sees the same thing and accepts the external cname ip instead of local lan ip.

1 Like
14

Note that dig queries only A records by default, while you need to explicitly check for AAAA, so using nslookup is a bit easier.

If you get a positive reply, then override it with own custom AAAA records.

By the way, you can collect a detailed log for systemd-resolved like this:

HOST="example.org"; \
sudo resolvectl log-level debug; \
{ resolvectl flush-caches; \
resolvectl reset-server-features; \
resolvectl query ${HOST}; } 2>&1 \
| systemd-cat -t systemd-resolved; \
journalctl -S -1m -t systemd-resolved \
> resolved.log
15

So when I ran this, it gets even more interesting… it looks like it completely ignores my local A record and returns back the external DNS CNAME record as the only option. All IPv4 in this scenario. When I perform nslookup against the same internal DNS server I see both the internal IPv4 IP and the external CNAME record.

Any way I can send the file to you to see if you see something sticks out without posting to all here?

Thank you for all your help so far!

16

You could record the network traffic at your DNS server and see exactly which requests and replies are being made. Then you can compare Windows, macOS and Fedora.

Do you know how to do this?

1 Like
17

You need to tell us exactly what you are doing.
Please replace “it”’ with what you are referring to so I do not have to guess.
What command do you run to do the test?

I have Windows, macOS and Fedora systems that also all use a local DNS server that has internal names and forwards on to the internet for the external names.

18

I did what was suggested. Even tho nslookup shows the internal lan resolution and the external cname the below seems to have shown only the external cname resolution as an option. Both resolving against the same internal DNS server.

HOST=“example.org”;
sudo resolvectl log-level debug;
{ resolvectl flush-caches;
resolvectl reset-server-features;
resolvectl query ${HOST}; } 2>&1
| systemd-cat -t systemd-resolved;
journalctl -S -1m -t systemd-resolved \

resolved.log

19

Where is the output of the commands you ran?

1 Like
20

Overriding upstream AAAA with just local A is problematic, because resolving a hostname queries both record types, and IPv6 is always preferred as long as it is routable.

This task generally requires to create an AAAA record on your local DNS server using IPv6 ULA or LLA.

As a workaround, you can force using a local address resolved with LLMNR by adding a dot to the hostname:

# Server
sudo tee /etc/systemd/resolved.conf << EOF > /dev/null
[Resolve]
LLMNR=yes
EOF
sudo systemctl restart systemd-resolved.service
sudo firewall-cmd --permanent --add-service=llmnr
sudo firewall-cmd --reload

# Client
ssh server.
nautilus sftp://server.
21 
;homebridge. IN A

;; ANSWER SECTION:
servername. 0 IN A 10.XXX.XXX.XXX

Use some domain suffix for resolved name. Single label names are sent only to LLMNR, if that is enabled. If you have homebridge.mydomain.com, it should work with that. But just homebridge. is not sent to your DNS servers at all by systemd-resolved, unless you either disable LLMNR=no or enable single label queries by ResolveUnicastSingleLabel=yes in /etc/systemd/resolved.conf. It is tried only by multicast link-local protocol in default configuration.

dig by default does not apply +search option to searched name. try host servername $INTERNALIPV6_IP and host servername $INTERNALIPV4_IP. Or use dig +search @$INTERNALIPV4_IP servername.

It should help if you name your host with fully qualified domain name at unifi gateway. Ie. write servername.mydomain.com as their hostname, not just name without dot. It is failure of resolved to try find them by default IMO, but not likely to change soon.

Ensure you try to query both servername and servername.mydomain.com. Under normal circumstances with domain mydomain.com set in /etc/resolv.conf, bare name with domain appendend is tried first. Without systemd-resolved it will try also servername. resolved in normal DNS, but systemd people think that leaks names supposed to be local only.

Also I think dnsmasq is used at unifi gateway as DNS. It may serve IPv4 addresses automatically assigned via DHCP, but maybe DHCPv6 is not used at the same time. That would leave AAAA queries for servername.mydomain.com forwarded to external servers, which probably cannot know internal address. dnsmasq configuration option local=/mydomain.com/ would ensure nothing from mydomain.com is forwarder further. Whatever in that zone is not known by local dnsmasq, then it does not exist. I am not sure it can be set by unifi configuration interface. host command prints less information, but will query A, AAAA and MX types for you, unless query type is specified explicitly.

I understand your wish for privacy, but sharing just partial responses and incomplete configuration makes helping you effectively much harder.

It is also not clear to me, why you try to have internal and external IPv6 addresses. IPv6 has enough bits to use only public addresses, I thought that is its advantage.

22

@pemensik - Thank you there is some excellent info in your write up.

To answer your IPv6 addressing question. My provider gives me /56 which I am then subnet-ing via /64 and using DHCPv6. This is allows me to define VLANS and fw rules similar to what I am doing with my IPv4 IP space.

Let’s simplify my testing and pull IPv6 out of the equation and focus solely on a device with an internal and external DNS IPv4 resolution only.

I have a device that is only defined with IPv4 IP internally, and has an external DNS CNAME which is pointed to an IPv4 IP.

By doing what you mentioned and referencing the server as servername. [period] it’s consistently working as expected and I can ping,ssh, etc. However, the moment I reference it via it’s FQDN (of which on the local DNS is also defined with full domainname.) it reverts to favoring the external CNAME IP. Even if I perform servername.domainname.com. [period] it still goes to the external

Is there a way to get both the servername and servername.domainname.com prioritized internally without having to leverage a different connotation with the trailing . [period] ?

I also want to take a moment and thank everyone else as well so far on trying to help me figure this out.

23

Please compare A queries sent to different servers: HOST=servername.domainname.com; for NS in 127.0.0.53 $INTERNALIPV4_IP $INTERNALIPV6_IP; do dig -t A @$NS $HOST; host $HOST $NS; done

Are there any significant differences in their responses among different servers? does response directed at local resolved 127.0.0.53 address match what response it gets on internal DNS addresses? Do IPv4 and IPv6 internal servers respond with the same addresses? Order of addresses might change if multiple records, but the set should be the same.

My friend has unifi based network. I have tested that network, it defines names only for name used in the UI. When he used just server1 in UI, then server1. name is offered only by its DNS. It does not register server1.example.com on that network, unless the machine is named with FQDN in unifi UI. Then it again does not register server1., but search example.com makes that work, unless final dot is present in name. It might be also your case.

Ensure all servers offered by the network know internal names. Relying on order of nameservers is usually unreliable with any cache except dnsmasq with special options.