Br0 suddenly stopped passing traffic

As the title says, I’m struggling to solve a spontaneous new problem.

I’ve been running this Dell Poweredge T640 Fedora 36 server for ~7 months. I created a br0 bridge in cockpit webUI. I added 2 physical nics to the bridge with no addresses and add manually created libvirt VMs as well as Kubevirt VMs via multus to this bridge regularly. Traffic has been flowing across this bridge no issue the whole time, without issues across updates and reboots until today.

Today I installed updates for the first time in a month, rebooted the server, and now I’m in a bind.

The default (and only route) for the host is via the br0 ip. I can ssh to the server at it’s default route address on the br0 interface. From on the server, I can also ssh to VMs hosted on the server too.

The VMs cannot ping any IP on the LAN other than only the host IP. They cannot ping the default gateway, or anything and I can not reach them from any host on the LAN other than the server they are running on. Also, the two physical pics bridged by br0 are no-longer passing traffic. There are no vlans in play, only one subnet (192.168.1.0/24) this is a really rudimentary network setup. Firewalld is completely disabled. sysctl net forwarding is enabled.

[kc2admin@node1 ~]$ echo; ip a s br0; echo; ip a s enp137s0; echo; ip a s enp132s0f1; echo

10: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 32:98:e6:96:c7:2d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.21/24 brd 192.168.1.255 scope global noprefixroute br0
       valid_lft forever preferred_lft forever
    inet6 fe80::d636:cc13:82d0:645a/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

4: enp137s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether fc:34:97:1e:79:86 brd ff:ff:ff:ff:ff:ff

8: enp132s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether a0:36:9f:00:4e:06 brd ff:ff:ff:ff:ff:ff

I could use some troubleshooting ideas. Thank you in advance :slight_smile:

I think libvirt et al. might actually use some “lower-level” iptables/nftables commands to configure the routing/firewall rules. You might want to double-check that everything is truly disabled by running iptables --list-rules and/or nft list ruleset.

1 Like

Welcome to ask.:fedora:edora @usrbinkat

I also had problems to do a working bridge in libvirt. I tried to achieve it with the “Virtual Machine Manager” with a lot of headache.

Here the discussion where I found a solution:

Soooo, I’ve managed to re-create this on a totally different server.

Steps to reproduce:

  • install Fedora 36
  • install all updates
  • enable cockpit
  • create br0 bridge with manual static IP configuration
  • add physical Nic(s) to the bridge
  • observe host networking functions, host can reach up and downstream devices on the bridge
  • observe host bridge br0 does not pass traffic

I re-attempted this same setup again without cockpit, using NetworkManager-tui (nmtui), and observed the same results.

I’m really confused. This has been a tried and true method for me for years.

[kc2admin@honeycomb-lx2 ~]$ sudo iptables --list-rules
[sudo] password for kc2admin:
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
[kc2admin@honeycomb-lx2 ~]$ sudo nft list ruleset
table ip nat {
        chain KUBE-MARK-MASQ {
        }

        chain DOCKER {
                iifname "docker0" counter packets 0 bytes 0 return
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                oifname != "docker0" ip saddr 172.17.0.0/16 counter packets 0 bytes 0 masquerade
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                fib daddr type local counter packets 103 bytes 7480 jump DOCKER
        }

        chain OUTPUT {
                type nat hook output priority -100; policy accept;
                ip daddr != 127.0.0.0/8 fib daddr type local counter packets 0 bytes 0 jump DOCKER
        }
}
table ip6 nat {
        chain KUBE-MARK-MASQ {
        }
}
table ip filter {
        chain DOCKER {
        }

        chain DOCKER-ISOLATION-STAGE-1 {
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
                counter packets 11878 bytes 2737698 return
        }

        chain DOCKER-ISOLATION-STAGE-2 {
                oifname "docker0" counter packets 0 bytes 0 drop
                counter packets 0 bytes 0 return
        }

        chain FORWARD {
                type filter hook forward priority filter; policy drop;
                counter packets 11878 bytes 2737698 jump DOCKER-USER
                counter packets 11878 bytes 2737698 jump DOCKER-ISOLATION-STAGE-1
                oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
                oifname "docker0" counter packets 0 bytes 0 jump DOCKER
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
                iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
        }

        chain DOCKER-USER {
                counter packets 11878 bytes 2737698 return
        }
}

Yeah. There are some non-trivial forwarding rules being created by the tools there. Unfortunately, I’m not familiar enough with that sort of setup to be of much help. A few basic things to check though would be:

  1. Use brctl show to make sure that the right network interfaces are actually on the right bridges.
  2. Since you said this is using libvirt, make sure that there is a allow br0 line (or whatever your bridge name is) in /etc/qemu/bridge.conf.

If none of that works, I’d probably be down to trial-and-error experimenting with the forwarding rules. I don’t really use libvirt for anything anymore on my systems. I’ve mostly switched to using systemd-nspawn containers and using MACVLAN “bridges” to connect them to the external network. I also use systemd-networkd to configure everything manually so that I know exactly how things are wired and how they are supposed to work. I don’t like it when the tooling auto-configures a lot of stuff that I don’t know about and don’t know how to fix when things go wrong.

I lost track of the fact I was installing docker-ce. This post jogged my memory and a clean install without docker-ce resolved the issue.

The issue is resolved, docker-ce needs to fix their shiznits lol. Thank you!

I gave my selves the credit for the solution :slight_smile: but the thru hero is @jorti :

I’m happy to see that I’m not alone fighting with such things

Thanks @glb, of course mine was on virtbr0 :grin: as we used br0 in the topic on discussion.fp.org

Since I started with fedora, I observed all the changes about networking (F28) and it brings my brain cells to glow with all this changes.

If you have time and pleasure I would really appreciate if you could point out (in the discussion mentioned above) how it is on F36 and what exactly you have to turn off to make everything manually as you say you love it.

I’m not against new technologies, not at all. Just love to control them, and not that they control my workflow and interfere in that what i like to configure manually.

I know the feeling. :slightly_smiling_face:

I guess I could. It wouldn’t happen anytime soon though since I work as a sysadmin for a state university and the fall semester is approaching rapidly. I have to upgrade a bunch of servers and install a few CMS’s over the next few weeks. But once things settle down, I might. :slightly_smiling_face:

1 Like