Systemd-resolved duplicate entries

Hi,
I have a problem with systemd-resolved and NetworkManager.

I didn’t set any global DNS settings, and only have one interface with 2 DNS nameserver, but resolvectl says that somehow, I have 2 global and 2 interface related setting (which is the same as global), and, because of that, there are duplicate entries in the systemd-resolved generated. resolv.conf file.
(So at the moment I have 4 entries in systemd-resolved generated resolv.conf instead of 2).

The Rancher created RKE cluster uses this generated file, and I get a lot of DnsConfigForming event.
So, there are any idea about how to configure NetworkManager or systemd-resolved and make the generated resolv.conf usable again?

]$ cat /etc/os-release
NAME=Fedora
VERSION="34.20210518.3.0 (CoreOS)"
ID=fedora
VERSION_ID=34
VERSION_CODENAME=""
PLATFORM_ID="platform:f34"
PRETTY_NAME="Fedora CoreOS 34.20210518.3.0"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:34"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=34
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=34
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='34.20210518.3.0'
DEFAULT_HOSTNAME=localhost
$ cat /run/systemd/resolve/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 8.8.8.8
# Too many DNS servers configured, the following entries may be ignored.
nameserver 8.8.4.4
search .
# /etc/systemd/resolved.conf file not modified, doesn't contain any settings.
$ resolvectl dns
Global: 8.8.8.8 8.8.4.4
Link 2 (ens2): 8.8.8.8 8.8.4.4
Link 3 (docker0):
Link 4 (calic26bd4debcf):
Link 5 (calif54f842fb17):
Link 6 (cali2530e2739b1):
Link 7 (tunl0):

Is it possible those entries are getting set from two different sources (i.e. one set on the system and one set via DHCP?).

What nmconnection files exist on your system (in /etc/NetworkManager/system-connections/)? What are there contents?

What are the logs for NetworkManager and systemd-resolved for the current boot?

journalctl -u NetworkManager -u systemd-resolved -b0

Have only one interface with static IP and DNS settings:

$ sudo cat /etc/NetworkManager/system-connections/ens2.nmconnection
[connection]
type=ethernet
interface-name=ens2

[ipv4]
method=manual
addresses=10.142.0.35/24
gateway=10.142.0.254
dns=8.8.8.8;8.8.4.4
dns-search=

Output of $ journalctl -u NetworkManager -u systemd-resolved -b0:

I tried some settings, that’s why contains somw warnings, but I pasted the current nmconnections configuration.

It looks like your systemd-resolved is complaining and restarting itself alot:

Jul 16 21:57:43 k8s-worker-5 systemd[1]: Starting Network Name Resolution...
Jul 16 21:57:43 k8s-worker-5 systemd-resolved[81832]: Positive Trust Anchors:
Jul 16 21:57:43 k8s-worker-5 systemd-resolved[81832]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jul 16 21:57:43 k8s-worker-5 systemd-resolved[81832]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-add
Jul 16 21:57:43 k8s-worker-5 systemd-resolved[81832]: /etc/systemd/resolved.conf.d/global-dns.conf:1: Assignment outside of section. Ignoring.
Jul 16 21:57:43 k8s-worker-5 systemd-resolved[81832]: Using system hostname 'k8s-worker-5'.
Jul 16 21:57:43 k8s-worker-5 systemd[1]: Started Network Name Resolution.
Jul 16 21:58:04 k8s-worker-5 systemd-resolved[81832]: Failed to emit notification about changed property CurrentDNSServer: Transport endpoint is not connected
Jul 16 21:58:04 k8s-worker-5 systemd[1]: Stopping Network Name Resolution...

What’s in /etc/systemd/resolved.conf.d/global-dns.conf ?

If you can reproduce this easily it might be better for you to provide a minimal Butane config that shows the problem. Then I can try to reproduce myself in my local environment.

Also… can you try using 208.67.222.222 and 208.67.220.220 (the opendns nameservers) instead of 8.8.8.8 and 8.8.4.4. This will at least let us know if the two entries are both coming from your static network config or if one of them is coming from somewhere else.

The whole /etc/systemd/resolved.conf.d/ directory is empty for me.
Changed the DNS entries in ens2.nmconnection, restarted NetworkManager, and called resolvectl dns:

Global: 8.8.8.8 8.8.4.4
Link 2 (ens2): 208.67.220.220 208.67.222.222
Link 3 (docker0):
Link 4 (calic26bd4debcf):
Link 5 (calif54f842fb17):
Link 6 (cali2530e2739b1):
Link 7 (tunl0):

After that I searched everywhere the global entries, but I didn’t found them, so I restarted the systemd-resolved, and it’s changed the global entries too:

Global: 208.67.220.220 208.67.222.222
Link 2 (ens2): 208.67.220.220 208.67.222.222
Link 3 (docker0):
Link 4 (calic26bd4debcf):
Link 5 (calif54f842fb17):
Link 6 (cali2530e2739b1):
Link 7 (tunl0):

So, at the moment, I don’t know why the systemd-resolved picks up the interface entries as global entries (and generates the resolv.conf file incorrectly).

That’s weird. The log mentioned specifically /etc/systemd/resolved.conf.d/global-dns.conf:

Jul 16 21:57:43 k8s-worker-5 systemd-resolved[81832]: /etc/systemd/resolved.conf.d/global-dns.conf:1: Assignment outside of section. Ignoring.

I saw that, but it’s full of emptiness.

# ls -lah /etc/systemd/resolved.conf.d/
total 4.0K
drwxr-xr-x  2 root root    6 Jul 16 21:59 .
drwxr-xr-x. 7 root root 4.0K Jul 16 21:56 ..

1 Like

As I see, Fedora guys will fix the full integration of systemd-resolved in F35 based images, but it will come maybe in september. :
https://github.com/coreos/fedora-coreos-tracker/issues/834

@dustymabe Do you have any idea how to resolve this, or continue the journey? :slight_smile:

Hey @mecseid - I’m sorry but I really am not sure what is causing that system to get multiple entries in the resolv.conf. You linked to fully enabling systemd-resolved · Issue #834 · coreos/fedora-coreos-tracker · GitHub but I don’t know if that would really fix your problem, though, yes, there would be just the stub listener entry in resolv.conf like:

nameserver 127.0.0.53
options edns0 trust-ad
search .

Would you want to try out one of our rawhide images (purely for testing) just to see what the behavior is for you there? These images already have the systemd-resolved enablement bits in them since they are on f35 Fedora CoreOS Build Browser