#3

yes. i used ifconfig just to see interfaces. ip addr and ip link show exactly the same thing.
i want to clarify that the bond is created with nmcli. All devices and connections are managed by network manager since networkd is not available.

Is there a way to stop NetManager from messing with these or can we move to the networkd network-scripts instead?

With NetManager disabled, this issue is not happening.

#4

Can you open a new issue on the Fedora CoreOS tracker with all the details? Thanks!

#5

i can, but in the meantime, can i just name interfaces statically, using udev, such as explained here:

or will this work only at installation time?

or, can i just set /etc/udev/rules.d/70-persistent-net.rules with static if names, such as:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:b1:05:63:31", ATTR{type}=="1", KERNEL=="*", NAME="myeth2"

will it work on fcos?

#6

@florinb make sure this is a fresh installation from a very recent FCOS image (i.e. not an old upgraded node). Older images were not using persistent naming due to a bug.

#7

we are using the latest version of FCOS. The predictive naming is used, it just messes up the names after reboots at times.

#8

You should be able to use both the systemd Link or direct Udev rules to set static names to interfaces.

#9

I’d be very interested in a bug report with more details. Hopefully you can easily reproduce and we can find and fix the bug!

#10

its difficult to reproduce this as we lose connectivity each time the mac is changed, due to the bond config (this is a bare metal server)
what I noticed, is that the Software MAC is moved to HW MAC:

5: ens2f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a0:36:9f:46:5f:9f brd ff:ff:ff:ff:ff:ff
    altname enp2s0f3

AFTER REBOOT:

5: ens2f3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond state UP group default qlen 1000
    link/ether ba:7a:f5:b8:71:d1 brd ff:ff:ff:ff:ff:ff permaddr a0:36:9f:46:5f:9f
    altname enp2s0f3

Notice how link/ether address was moved to permaddr after reboot.

Additionally, because /usr is read-only, I cannot configure set naming/mac conventions inside /usr/lib/udev/rules.d/60-net.rules. @Siosm how can i set Udev rules, after OS installation?

#11

Yes, you can write a Udev rule in /etc/ and they should take precedence over the one in /usr. You might have to be careful with naming and ordering but I don’t know the exact rules for Udev.

From the man page:

The udev rules are read from the files located in the system rules
directories /usr/lib/udev/rules.d and /usr/local/lib/udev/rules.d,
the volatile runtime directory /run/udev/rules.d and the local
administration directory /etc/udev/rules.d. All rules files are
collectively sorted and processed in lexical order, regardless of the
directories in which they live. However, files with identical
filenames replace each other. Files in /etc have the highest
priority, files in /run take precedence over files with the same name
under /usr. This can be used to override a system-supplied rules file
with a local file if needed; a symlink in /etc with the same name as
a rules file in /usr/lib, pointing to /dev/null, disables the rules
file entirely. Rule files must have the extension .rules; other
extensions are ignored.
#12

Thank you for the help. Will try that as soon as possible.

In the meantime, I have narrowed this down to predictive renaming failure. Running dmesg and grepping for eth (the eth4 and eth5 interfaces are being renamed, but only 1 can be renamed):

system-connections]# dmesg | grep eth
[    9.338010] systemd-udevd[713]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[    9.403278] igb 0000:02:00.0: added PHC on eth0
[    9.421333] igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) a0:36:9f:63:34:28
[    9.430422] igb 0000:02:00.0: eth0: PBA No: H47822-001
[    9.559340] igb 0000:02:00.1: added PHC on eth1
[    9.576747] igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) a0:36:9f:63:34:29
[    9.585816] igb 0000:02:00.1: eth1: PBA No: H47822-001
[    9.659581] igb 0000:02:00.2: added PHC on eth2
[    9.676902] igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) a0:36:9f:63:34:2a
[    9.685999] igb 0000:02:00.2: eth2: PBA No: H47822-001
[    9.759337] igb 0000:02:00.3: added PHC on eth3
[    9.776574] igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) a0:36:9f:63:34:2b
[    9.785363] igb 0000:02:00.3: eth3: PBA No: H47822-001
[    9.873908] igb 0000:06:00.0: added PHC on eth4
[    9.890087] igb 0000:06:00.0: eth4: (PCIe:5.0Gb/s:Width x4) 00:1e:67:a4:17:39
[    9.898202] igb 0000:06:00.0: eth4: PBA No: 100000-000
[    9.969289] igb 0000:06:00.1: added PHC on eth5
[    9.984736] igb 0000:06:00.1: eth5: (PCIe:5.0Gb/s:Width x4) 00:1e:67:a4:17:3a
[    9.992598] igb 0000:06:00.1: eth5: PBA No: 100000-000
[   10.012258] igb 0000:02:00.0 ens2f0: renamed from eth0
[   10.058972] igb 0000:02:00.3 ens2f3: renamed from eth3
[   10.065791] systemd-udevd[674]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   10.086384] systemd-udevd[683]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   10.100794] systemd-udevd[713]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   10.103811] igb 0000:02:00.2 ens2f2: renamed from eth2
[   10.116035] systemd-udevd[667]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   10.147340] systemd-udevd[669]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   10.163547] systemd-udevd[707]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   10.164753] igb 0000:06:00.0 eno1: renamed from eth4
[   10.194637] igb 0000:02:00.1 ens2f1: renamed from eth1
[   10.214933] systemd-udevd[707]: eth5: Failed to rename network interface 7 from 'eth5' to 'eno1': File exists
[   10.232496] systemd-udevd[707]: eth5: Failed to process device, ignoring: File exists

I think this is a FW bug with the NICs…

#13

odd. What is printed out if you go through each NIC in your system and run

for nic in ens2f0 ens2f1 ens2f2 ens2f3 eno1 eth4; do
    udevadm test-builtin net_id /sys/class/net/$nic
done
#14

eth4 is eno1, eth5 can’t be renamed:

# for nic in ens2f0 ens2f1 ens2f2 ens2f3 eno1 eth5; do     udevadm test-builtin net_id /sys/class/net/$nic; done
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_MAC=enxa0369f633428
ID_OUI_FROM_DATABASE=Intel Corporate
ID_NET_NAME_PATH=enp2s0f0
ID_NET_NAME_SLOT=ens2f0
Unload module index
Unloaded link configuration context.
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_MAC=enxa0369f633429
ID_OUI_FROM_DATABASE=Intel Corporate
ID_NET_NAME_PATH=enp2s0f1
ID_NET_NAME_SLOT=ens2f1
Unload module index
Unloaded link configuration context.
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_MAC=enxa0369f63342a
ID_OUI_FROM_DATABASE=Intel Corporate
ID_NET_NAME_PATH=enp2s0f2
ID_NET_NAME_SLOT=ens2f2
Unload module index
Unloaded link configuration context.
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_PATH=enp2s0f3
ID_NET_NAME_SLOT=ens2f3
Unload module index
Unloaded link configuration context.
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_ONBOARD=eno1
ID_NET_LABEL_ONBOARD=PowerVille
ID_NET_NAME_PATH=enp6s0f0
Unload module index
Unloaded link configuration context.
Load module index
Parsed configuration file /usr/lib/systemd/network/99-default.link
Created link configuration context.
Using default interface naming scheme 'v245'.
ID_NET_NAMING_SCHEME=v245
ID_NET_NAME_MAC=enx001e67a4173a
ID_OUI_FROM_DATABASE=Intel Corporate
ID_NET_NAME_ONBOARD=eno1
ID_NET_LABEL_ONBOARD=PowerVille
ID_NET_NAME_PATH=enp6s0f1
Unload module index
Unloaded link configuration context.

I am going to try static naming for the two NICs that are being interchanged after reboot, eno1 (eth4) and eth5. Before doing so, @Siosm could you confirm if indeed the name for my udev rule file should be this:

/etc/udev/rules.d/80-ifname.rules

And the content will be something like this:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:xx:xx:xx:xx", ATTR{type}=="1", NAME="eno1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:1e:xx:xx:xx:xx", ATTR{type}=="1", NAME="eth5"

I am not changing anything, I just want to statically name the NICs as they are now, with their HW MAC the same, so my existing bond connection is not broken.

#15

The udev rule looks good to me.

#16

Added the rules and rebooted. Unfortunately we lost connectivity, likely due to the bond already existing in place. We will check again and see what changed.

In the meantime, let me know what you see amiss in the output above @dustymabe

#17

Honestly I think you might be right about it being a bug with the hardware/BIOS/firmware. You can see ID_NET_NAME_ONBOARD=eno1 in the last two entries in the output. See this very similar issue where the problem got resolved with a BIOS update.

Any chance there is an outstanding update for the firmware of the server that might fix things?

#18

yes, static renaming worked.
however, eno1 cannot be used. Renaming is done twice, once at boot, then once /etc/udev/rules.d are read, a second time. And if the first renaming process used eno1, the second time won’t work because the config file for eno1 exists.

Using eno2 and eno3, eth4 and eth5 or any other naming convention but eno1 works properly.

#19

Just FYI, there is a net.ifnames=0 kernel parameter that can be used to disable that first automatic renaming process. I’m not sure if that would help in your situation, but maybe. It is not guaranteed, but depending on your circumstances and hardware, just leaving the old ethX names without any renaming might be more stable.

#20

no, definitely not. I want predictive naming because the server has more than 1 card and one with more than 1 port. Without it, names would be all over the place.

#21

Did you by chance check to see if there was a bios/firmware update you could apply?

Also, if you really want to take it to the “predictable extreme” you could have your NICs named based on the mac address with something like:

storage:                                                                                                                                                                                                                                     
  files:                                                                                                                                                                                                                                     
    - path: /etc/systemd/network/99-default.link                                                                                                                                                                                             
      mode: 0644                                                                                                                                                                                                                             
      contents:                                                                                                                                                                                                                              
        inline: |                                                                                                                                                                                                                            
          [Link]                                                                                                                                                                                                                             
          NamePolicy=mac                                                                                                                                                                                                                     
          MACAddressPolicy=persistent

Gives you a NIC names based on the mac address like enx5254009110fa.

#22

there likely is, but these servers don’t have warranty or support and messing an upgrade for FW or BIOS could be a complete loss.