I am running FCOS on Bare Metal with minimal ignition setup (basically it just sets a key for core user, nothing else).
I am trying to setup bond configs, based on interface names, but at times, after reboot, the interfaces making up the bond, are being renamed. Additionally, I understand FCOS should be using only “Predictable Naming Interfaces” yet, my ifconfig output shows both old-style (eth4 for example and ens2f2) new style naming. Is this a bug?
EXAMPLE:
BEFORE REBOOT:
5: ens2f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether a0:36:9f:46:5f:9f brd ff:ff:ff:ff:ff:ff
altname enp2s0f3
6: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1e:67:a4:31:2e brd ff:ff:ff:ff:ff:ff
altname enp6s0f0
AFTER REBOOT:
5: ens2f3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond state UP group default qlen 1000
link/ether ba:7a:f5:b8:71:d1 brd ff:ff:ff:ff:ff:ff permaddr a0:36:9f:46:5f:9f
altname enp2s0f3
6: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1e:67:a4:31:2e brd ff:ff:ff:ff:ff:ff
altname enp6s0f0
Since interfaces are part of a strict bond setup, it breaks all connectivity when these are randomly renamed. What is causing this and how can it be fixed?
yes. i used ifconfig just to see interfaces. ip addr and ip link show exactly the same thing.
i want to clarify that the bond is created with nmcli. All devices and connections are managed by network manager since networkd is not available.
Is there a way to stop NetManager from messing with these or can we move to the networkd network-scripts instead?
With NetManager disabled, this issue is not happening.
@florinb make sure this is a fresh installation from a very recent FCOS image (i.e. not an old upgraded node). Older images were not using persistent naming due to a bug.
its difficult to reproduce this as we lose connectivity each time the mac is changed, due to the bond config (this is a bare metal server)
what I noticed, is that the Software MAC is moved to HW MAC:
5: ens2f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether a0:36:9f:46:5f:9f brd ff:ff:ff:ff:ff:ff
altname enp2s0f3
AFTER REBOOT:
5: ens2f3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond state UP group default qlen 1000
link/ether ba:7a:f5:b8:71:d1 brd ff:ff:ff:ff:ff:ff permaddr a0:36:9f:46:5f:9f
altname enp2s0f3
Notice how link/ether address was moved to permaddr after reboot.
Additionally, because /usr is read-only, I cannot configure set naming/mac conventions inside /usr/lib/udev/rules.d/60-net.rules. @siosm how can i set Udev rules, after OS installation?
Yes, you can write a Udev rule in /etc/ and they should take precedence over the one in /usr. You might have to be careful with naming and ordering but I don’t know the exact rules for Udev.
The udev rules are read from the files located in the system rules
directories /usr/lib/udev/rules.d and /usr/local/lib/udev/rules.d,
the volatile runtime directory /run/udev/rules.d and the local
administration directory /etc/udev/rules.d. All rules files are
collectively sorted and processed in lexical order, regardless of the
directories in which they live. However, files with identical
filenames replace each other. Files in /etc have the highest
priority, files in /run take precedence over files with the same name
under /usr. This can be used to override a system-supplied rules file
with a local file if needed; a symlink in /etc with the same name as
a rules file in /usr/lib, pointing to /dev/null, disables the rules
file entirely. Rule files must have the extension .rules; other
extensions are ignored.
Thank you for the help. Will try that as soon as possible.
In the meantime, I have narrowed this down to predictive renaming failure. Running dmesg and grepping for eth (the eth4 and eth5 interfaces are being renamed, but only 1 can be renamed):
system-connections]# dmesg | grep eth
[ 9.338010] systemd-udevd[713]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[ 9.403278] igb 0000:02:00.0: added PHC on eth0
[ 9.421333] igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) a0:36:9f:63:34:28
[ 9.430422] igb 0000:02:00.0: eth0: PBA No: H47822-001
[ 9.559340] igb 0000:02:00.1: added PHC on eth1
[ 9.576747] igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) a0:36:9f:63:34:29
[ 9.585816] igb 0000:02:00.1: eth1: PBA No: H47822-001
[ 9.659581] igb 0000:02:00.2: added PHC on eth2
[ 9.676902] igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) a0:36:9f:63:34:2a
[ 9.685999] igb 0000:02:00.2: eth2: PBA No: H47822-001
[ 9.759337] igb 0000:02:00.3: added PHC on eth3
[ 9.776574] igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) a0:36:9f:63:34:2b
[ 9.785363] igb 0000:02:00.3: eth3: PBA No: H47822-001
[ 9.873908] igb 0000:06:00.0: added PHC on eth4
[ 9.890087] igb 0000:06:00.0: eth4: (PCIe:5.0Gb/s:Width x4) 00:1e:67:a4:17:39
[ 9.898202] igb 0000:06:00.0: eth4: PBA No: 100000-000
[ 9.969289] igb 0000:06:00.1: added PHC on eth5
[ 9.984736] igb 0000:06:00.1: eth5: (PCIe:5.0Gb/s:Width x4) 00:1e:67:a4:17:3a
[ 9.992598] igb 0000:06:00.1: eth5: PBA No: 100000-000
[ 10.012258] igb 0000:02:00.0 ens2f0: renamed from eth0
[ 10.058972] igb 0000:02:00.3 ens2f3: renamed from eth3
[ 10.065791] systemd-udevd[674]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[ 10.086384] systemd-udevd[683]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[ 10.100794] systemd-udevd[713]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[ 10.103811] igb 0000:02:00.2 ens2f2: renamed from eth2
[ 10.116035] systemd-udevd[667]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[ 10.147340] systemd-udevd[669]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[ 10.163547] systemd-udevd[707]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[ 10.164753] igb 0000:06:00.0 eno1: renamed from eth4
[ 10.194637] igb 0000:02:00.1 ens2f1: renamed from eth1
[ 10.214933] systemd-udevd[707]: eth5: Failed to rename network interface 7 from 'eth5' to 'eno1': File exists
[ 10.232496] systemd-udevd[707]: eth5: Failed to process device, ignoring: File exists
# for nic in ens2f0 ens2f1 ens2f2 ens2f3 eno1 eth5; do udevadm test-builtin net_id /sys/class/net/$nic; done
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_MAC=enxa0369f633428
ID_OUI_FROM_DATABASE=Intel Corporate
ID_NET_NAME_PATH=enp2s0f0
ID_NET_NAME_SLOT=ens2f0
Unload module index
Unloaded link configuration context.
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_MAC=enxa0369f633429
ID_OUI_FROM_DATABASE=Intel Corporate
ID_NET_NAME_PATH=enp2s0f1
ID_NET_NAME_SLOT=ens2f1
Unload module index
Unloaded link configuration context.
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_MAC=enxa0369f63342a
ID_OUI_FROM_DATABASE=Intel Corporate
ID_NET_NAME_PATH=enp2s0f2
ID_NET_NAME_SLOT=ens2f2
Unload module index
Unloaded link configuration context.
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_PATH=enp2s0f3
ID_NET_NAME_SLOT=ens2f3
Unload module index
Unloaded link configuration context.
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_ONBOARD=eno1
ID_NET_LABEL_ONBOARD=PowerVille
ID_NET_NAME_PATH=enp6s0f0
Unload module index
Unloaded link configuration context.
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_MAC=enx001e67a4173a
ID_OUI_FROM_DATABASE=Intel Corporate
ID_NET_NAME_ONBOARD=eno1
ID_NET_LABEL_ONBOARD=PowerVille
ID_NET_NAME_PATH=enp6s0f1
Unload module index
Unloaded link configuration context.
I am going to try static naming for the two NICs that are being interchanged after reboot, eno1 (eth4) and eth5. Before doing so, @siosm could you confirm if indeed the name for my udev rule file should be this:
I am not changing anything, I just want to statically name the NICs as they are now, with their HW MAC the same, so my existing bond connection is not broken.
Added the rules and rebooted. Unfortunately we lost connectivity, likely due to the bond already existing in place. We will check again and see what changed.
In the meantime, let me know what you see amiss in the output above @dustymabe
Honestly I think you might be right about it being a bug with the hardware/BIOS/firmware. You can see ID_NET_NAME_ONBOARD=eno1 in the last two entries in the output. See this very similar issue where the problem got resolved with a BIOS update.
Any chance there is an outstanding update for the firmware of the server that might fix things?
yes, static renaming worked.
however, eno1 cannot be used. Renaming is done twice, once at boot, then once /etc/udev/rules.d are read, a second time. And if the first renaming process used eno1, the second time won’t work because the config file for eno1 exists.
Using eno2 and eno3, eth4 and eth5 or any other naming convention but eno1 works properly.
Just FYI, there is a net.ifnames=0 kernel parameter that can be used to disable that first automatic renaming process. I’m not sure if that would help in your situation, but maybe. It is not guaranteed, but depending on your circumstances and hardware, just leaving the old ethX names without any renaming might be more stable.
no, definitely not. I want predictive naming because the server has more than 1 card and one with more than 1 port. Without it, names would be all over the place.