Local DNS resolution breaks when cache enabled

Hi, I have a weird issue, that’s visible only on Fedora (not endavouros, not on manjaro, not on android nor macOS).

I have DHCP which is sending internal DNS server address. Then under this address I am running coredns which is returning some internal reolutions for domain: xxxx.lan and forwarding other requests.

When I turn on laptop with fresh install of fedora I can resolve to local DNS addresses (like host.domain.lan) only for a moment. After a while I’m getting unknown host error. After restarting/flushing dns everything starts to work for a minute/two again.

I was experimenting with /etc/systemd/resolved.conf config. It seems that if I introduce Cache=no setting everything works fine.

Do you have any idea why this is required?

Please replace that screen shot with a copy-n-paste of the text.
You can format nicely using the </> button (that maybe in the :gear: menu on a small screen.

For example:

```
pre-formatted text
example
```
alke@k2so:~$ dig caracal.fennec.lan

; <<>> DiG 9.18.24 <<>> caracal.fennec.lan
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42698
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;caracal.fennec.lan.            IN      A

;; ANSWER SECTION:
caracal.fennec.lan.     44      IN      A       192.168.0.4

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Mar 05 20:44:07 CET 2024
;; MSG SIZE  rcvd: 63

wait 3-5 minutes

alke@k2so:~$ dig caracal.fennec.lan

; <<>> DiG 9.18.24 <<>> caracal.fennec.lan
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 14202
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;caracal.fennec.lan.            IN      A

;; AUTHORITY SECTION:
.                       7086    IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2024030501 1800 900 604800 86400

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Mar 05 20:48:28 CET 2024
;; MSG SIZE  rcvd: 122

alke@k2so:~$ dig @192.168.0.107 caracal.fennec.lan

; <<>> DiG 9.18.24 <<>> @192.168.0.107 caracal.fennec.lan
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10815
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 7f7cee465615e974 (echoed)
;; QUESTION SECTION:
;caracal.fennec.lan.            IN      A

;; ANSWER SECTION:
caracal.fennec.lan.     60      IN      A       192.168.0.4

;; Query time: 4 msec
;; SERVER: 192.168.0.107#53(192.168.0.107) (UDP)
;; WHEN: Tue Mar 05 20:49:58 CET 2024
;; MSG SIZE  rcvd: 93

alke@k2so:~$ resolvectl flush-caches
alke@k2so:~$ dig caracal.fennec.lan

; <<>> DiG 9.18.24 <<>> caracal.fennec.lan
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33584
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;caracal.fennec.lan.            IN      A

;; ANSWER SECTION:
caracal.fennec.lan.     60      IN      A       192.168.0.4

;; Query time: 5 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Mar 05 20:51:44 CET 2024
;; MSG SIZE  rcvd: 63

alke@k2so:~$ ping caracal.fennec.lan
ping: caracal.fennec.lan: Name or service not known
alke@k2so:~$ resolvectl --no-pager status
Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (wlp0s20f3)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.0.107
       DNS Servers: 192.168.0.107
        DNS Domain: fennec.lan
alke@k2so:~$ resolvectl --no-pager query caracal.fennec.lan
caracal.fennec.lan: Name 'caracal.fennec.lan' not found
alke@k2so:~$ journalctl --no-pager -b -u systemd-resolved.service
Mar 05 22:52:20 fedora systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
Mar 05 22:52:20 fedora systemd-resolved[1781]: Positive Trust Anchors:
Mar 05 22:52:20 fedora systemd-resolved[1781]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Mar 05 22:52:20 fedora systemd-resolved[1781]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Mar 05 22:52:20 fedora systemd-resolved[1781]: Using system hostname 'fedora'.
Mar 05 22:52:20 fedora systemd[1]: Started systemd-resolved.service - Network Name Resolution.
Mar 05 22:52:25 fedora systemd-resolved[1781]: wlp0s20f3: Bus client set search domain list to: fennec.lan
Mar 05 22:52:25 fedora systemd-resolved[1781]: wlp0s20f3: Bus client set default route setting: yes
Mar 05 22:52:25 fedora systemd-resolved[1781]: wlp0s20f3: Bus client set DNS server list to: 192.168.0.107
Mar 05 22:52:25 k2so systemd-resolved[1781]: System hostname changed to 'k2so.fennec.lan'.

When cache=no, everything works as expected (???)

alke@k2so:~$ sudo cat /etc/systemd/resolved.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the resolved.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.

[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
#DNS=
#FallbackDNS=
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=no
#LLMNR=no
Cache=no
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
#StaleRetentionSec=0
alke@k2so:~$ resolvectl --no-pager query caracal.fennec.lan
caracal.fennec.lan: 192.168.0.4                -- link: wlp0s20f3

-- Information acquired via protocol DNS in 34.7ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network