FCOS / Kubernetes (using docker) / Multus / CNI Bridge plugin - pods unable to access the additional network

I’m trying to migrate some container linux nodes to FCOS.

For my main pod to pod networking I use Calico and that continues to work fine on the new
FCOS nodes.

In addition I have Multus setup which allows me to attach additional network interfaces to
pods by creating network attachment definitions (which define the CNI plugin to use for
that network interface and its config) and then associating those with the few pods that
need them.

A NetworkAttachmentDefinition I have has the following config:

{
 "cniVersion": "0.3.1",
 "name": "lan",
 "type": "bridge",
 "bridge": "br-lan",
 "ipam": {
  "type": "static",
  "addresses": [
   { "address": "10.1.0.2/16" }
  ]
 }
}

Which really is as simple as it looks. Basically use the bridge CNI plugin to create a
veth pair so that the additional network interface in the container is added to the br-lan
bridge and configure the container side veth interface with the specified IP address.

If the pod that it annotated with this network attachment definition is moved onto an FCOS
node, its networking on that interface fails to work. From outside, if I tcpdump on the
FCOS node I see traffic traverse eno1 -> bond0 -> br-lan, but then it goes nowhere.
If the pod is on the container linux node that traffic additionally makes its way onto the
veth interface associated with the pod and everything works as expected.
In reverse I see the same thing. If I do something from the pod I see its traffic make it
to the veth interface on the host, but never onto the bridge, but on the CL node it makes
it to the bridge, through the other interfaces and out to the network. It’s almost
like the veth interface hasn’t been added to the bridge… But it has!

Like I said, this works totally fine on container linux. The networking is configured
exactly the same (other than via NetworkManager rather than systemd-networkd). So my
network interfaces are as follows:
eno1 -> bond0
eno2 -> bond0
bond0 -> br-lan
bond0.100 -> br-something
bond0.200 -> br-somethingelse

When the pod comes up, the bridge CNI plugin via Multus creates the veth interface and
adds the host side interface of the pair into br-lan. This output is from the FCOS node
(the ‘master br-lan’ part is apparently shows which devices are added to the
br-lan bridge if you haven’t got brctl available to see the output that most people
are probably used to seeing):
$ ip link | grep br-lan
7: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode
DEFAULT group default qlen 1000
11: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master
br-lan state UP mode DEFAULT group default qlen 1000
31: veth1f6cabd9@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master br-lan state UP mode DEFAULT group default

And this from the CL node, which matches:
$ ip link | grep br-lan
6: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode
DEFAULT group default qlen 1000
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master
br-lan state UP mode DEFAULT group default qlen 1000
218: vetha2c4788e@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master br-lan state UP mode DEFAULT group default

ipv4 ip_forward is enabled on both nodes
$ cat /proc/sys/net/ipv4/ip_forward
1

rp_filter for all of the above mentioned interfaces is set to 1 on the CL node and 2 on
the FCOS node. As far as I can tell 2 is the same as 1 but more loose about what it allows
so long as it can still route to the source of a packet somehow. So I can’t see it
being the problem

Examining the bridge config, the CL node has stp disabled. The FCOS node has stp enabled.
If I investigate further, both bond0 and the veth interface are in the forwarding state.
(3 seems to be the FORWARDING state)
cat /sys/devices/virtual/net/br-lan/brif/bond0/state 3 cat /sys/devices/virtual/net/br-lan/brif/veth1f6cabd9/state
3

If I disable stp on the bridge on the FCOS node it makes no difference

I even started comparing everything under /sys/devices/virtual/net/br-lan on both nodes,
but still can’t see anything out of the ordinary.

I really can’t believe that anything has changed too much in how any of this works
between CL and FCOS, but i’ve run out of ideas. I’ve been doing VMs in a similar
way for 10 years or so too, so as far as I can tell the networking itself is sound.

Any thoughts welcomed.

I finally managed to get traffic to work between the bridge and the veth interface. To do that I had to set /proc/sys/net/bridge/bridge-nf-call-iptables to 0.

This was set to 1, which matches the container linux node. So that’s odd.

As far as I can tell bridge-nf-call-iptables (at least according to stack exchange causes bridged packets to traverse the iptables rules. In my environment I think Calico is creating the iptables rules, so I would expect them to be the same.

A quick look however shows that on the CL node, the FORWARD chain policy is ACCEPT, but on the FCOS node it’s DROP. This looks to be the culprit. I can’t understand why FCOS itself would have anything to do with this, so it seems unlikely that FCOS itself is the problem.

I will report back if I find an answer to what’s actually causing it.

1 Like

Ok… enabling docker on the FCOS node and rebooting (I would guess starting it would achieve the same thing) is what’s adjusting the FORWARD chain policy. My guess is that docker on the CL nodes isn’t doing this due to being either older or configured differently in some way.

I guess my options are to work out how to stop docker doing this (or switch to a different container runtime). Although since everything else seems to work perhaps I should be working out a way to allow this traffic by adding iptables rules in some way.

Since my problem isn’t FCOS itself i’ll now leave this topic alone.

Thanks for the update @ashak I’m glad at least for now we’ve figured out where the traffic was getting stopped.