I’ve just encountered an issue when trying to upgrade from 41.20241109.3.0 to 41.20250130.3.0.
We currently run Kubernetes 1.26.15 on top of containerd/podman running on stateless FCOS on both AWS EC2 instances and on-prem vSphere VMs (deployed via Terraform/Ignition IaC since 2017).
Following a successful major upgrade from FCOS 40 (eventually) we expected no issues with this non-major upgrade.
Everything appears fine with our on-prem upgrades but when we upgraded on AWS kiam2-agent went into CrashLoopBackoff with the following error message:
{"level":"info","msg":"configuring iptables","time":"2025-03-10T16:44:28Z"}
{"level":"error","msg":"error configuring iptables: running </sbin/iptables -t nat -C PREROUTING -p tcp -d 169.254.169.254 --dport 80 -j DNAT --to-destination 10.252.25.27:8181 -i cali<ins> --wait>: exit status 3: iptables v1.8.3 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\n","time":"2025-03-10T16:44:28Z"}
{"level":"fatal","msg":"fatal error: running </sbin/iptables -t nat -C PREROUTING -p tcp -d 169.254.169.254 --dport 80 -j DNAT --to-destination 10.252.25.27:8181 -i cali</ins> --wait>: exit status 3: iptables v1.8.3 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\n","time":"2025-03-10T16:44:28Z"}
Logging into the nodes and I could modprobe ip_tables successfully and kiam2-agent sprang back into life.
After much searching and testing various possible fixes (e.g. NetavarkNftablesDefault), none of which worked, I eventually was able to implement a workaround in our IaC to force-load ip_tables to by effectively doing the following:
echo ip_tables > /etc/modules-load.d/ip_tables.conf
Does anyone have any information as to what may have changed between these two FCOS releases that might have caused this please?