Local DNS resolution breaks when cache enabled

Hi, I have a weird issue, that’s visible only on Fedora (not endavouros, not on manjaro, not on android nor macOS).

I have DHCP which is sending internal DNS server address. Then under this address I am running coredns which is returning some internal reolutions for domain: xxxx.lan and forwarding other requests.

When I turn on laptop with fresh install of fedora I can resolve to local DNS addresses (like host.domain.lan) only for a moment. After a while I’m getting unknown host error. After restarting/flushing dns everything starts to work for a minute/two again.

I was experimenting with /etc/systemd/resolved.conf config. It seems that if I introduce Cache=no setting everything works fine.

Do you have any idea why this is required?

Please replace that screen shot with a copy-n-paste of the text.
You can format nicely using the </> button (that maybe in the :gear: menu on a small screen.

For example:

```
pre-formatted text
example
```
alke@k2so:~$ dig caracal.fennec.lan

; <<>> DiG 9.18.24 <<>> caracal.fennec.lan
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42698
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;caracal.fennec.lan.            IN      A

;; ANSWER SECTION:
caracal.fennec.lan.     44      IN      A       192.168.0.4

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Mar 05 20:44:07 CET 2024
;; MSG SIZE  rcvd: 63

wait 3-5 minutes

alke@k2so:~$ dig caracal.fennec.lan

; <<>> DiG 9.18.24 <<>> caracal.fennec.lan
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 14202
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;caracal.fennec.lan.            IN      A

;; AUTHORITY SECTION:
.                       7086    IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2024030501 1800 900 604800 86400

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Mar 05 20:48:28 CET 2024
;; MSG SIZE  rcvd: 122

alke@k2so:~$ dig @192.168.0.107 caracal.fennec.lan

; <<>> DiG 9.18.24 <<>> @192.168.0.107 caracal.fennec.lan
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10815
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 7f7cee465615e974 (echoed)
;; QUESTION SECTION:
;caracal.fennec.lan.            IN      A

;; ANSWER SECTION:
caracal.fennec.lan.     60      IN      A       192.168.0.4

;; Query time: 4 msec
;; SERVER: 192.168.0.107#53(192.168.0.107) (UDP)
;; WHEN: Tue Mar 05 20:49:58 CET 2024
;; MSG SIZE  rcvd: 93

alke@k2so:~$ resolvectl flush-caches
alke@k2so:~$ dig caracal.fennec.lan

; <<>> DiG 9.18.24 <<>> caracal.fennec.lan
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33584
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;caracal.fennec.lan.            IN      A

;; ANSWER SECTION:
caracal.fennec.lan.     60      IN      A       192.168.0.4

;; Query time: 5 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Mar 05 20:51:44 CET 2024
;; MSG SIZE  rcvd: 63

alke@k2so:~$ ping caracal.fennec.lan
ping: caracal.fennec.lan: Name or service not known
alke@k2so:~$ resolvectl --no-pager status
Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (wlp0s20f3)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.0.107
       DNS Servers: 192.168.0.107
        DNS Domain: fennec.lan
alke@k2so:~$ resolvectl --no-pager query caracal.fennec.lan
caracal.fennec.lan: Name 'caracal.fennec.lan' not found
alke@k2so:~$ journalctl --no-pager -b -u systemd-resolved.service
Mar 05 22:52:20 fedora systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
Mar 05 22:52:20 fedora systemd-resolved[1781]: Positive Trust Anchors:
Mar 05 22:52:20 fedora systemd-resolved[1781]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Mar 05 22:52:20 fedora systemd-resolved[1781]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Mar 05 22:52:20 fedora systemd-resolved[1781]: Using system hostname 'fedora'.
Mar 05 22:52:20 fedora systemd[1]: Started systemd-resolved.service - Network Name Resolution.
Mar 05 22:52:25 fedora systemd-resolved[1781]: wlp0s20f3: Bus client set search domain list to: fennec.lan
Mar 05 22:52:25 fedora systemd-resolved[1781]: wlp0s20f3: Bus client set default route setting: yes
Mar 05 22:52:25 fedora systemd-resolved[1781]: wlp0s20f3: Bus client set DNS server list to: 192.168.0.107
Mar 05 22:52:25 k2so systemd-resolved[1781]: System hostname changed to 'k2so.fennec.lan'.

When cache=no, everything works as expected (???)

alke@k2so:~$ sudo cat /etc/systemd/resolved.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the resolved.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.

[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
#DNS=
#FallbackDNS=
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=no
#LLMNR=no
Cache=no
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
#StaleRetentionSec=0
alke@k2so:~$ resolvectl --no-pager query caracal.fennec.lan
caracal.fennec.lan: 192.168.0.4                -- link: wlp0s20f3

-- Information acquired via protocol DNS in 34.7ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

It depends what dig @192.168.0.107 caracal.fennec.lan will respond. I would just guess, you have something like dnsmasq at 192.168.0.107. First server knows *.lan names only, but dnsmasq has also other servers. Like Google’s 8.8.8.8 or Cloudflare’s 1.1.1.1. That does not know lan. domain or any your local machines. Try watching ttl in dig caracal.fennec.lan output, the one in ANSWER section after the name. When it reaches 0, make a new query (repeat dig command). Repeat that few times. Does it reliably respond with the same name? Problem might be lan non-existence might be cached, but for quite long.

Fix would be ensuring lan is forwarded only to servers that know it. But just guessing :slight_smile: