I updated my F38 desktop this weekend, and since then, some scripts I wrote a few years ago to help manage custom a firewalld ipset started failing. Now, both of the custom ipsets I created back then are missing. I’m just wondering if anyone knows why that happened. Does that sound familiar to anyone?
It works fine in my script using --add-entries-from-file.
Thanks, but my question is what happened to the ipset that I created? And, why did it get deleted?
Nothing unexpected should happen as long as you create and populate the IP sets properly using the relevant firewalld commands.
I created them with firewalld-cmd. So, again I ask, where did they go and why? They existed for several years. It’s not like my system crashed or anything, it was a normal dnf update. Something changed recently.
My setup is fine, so it doesn’t look like a common problem.
In any case, check your configs:
sudo firewall-cmd --permanent --get-ipsets
sudo ls -a -l -Z /etc/firewalld/ipsets
I really need help, so I guess we’ll continue. I don’t see what your situation has to do with mine, though.
-
It looks like the basic service I wrote to keep the ipset updated, has been disabled by…something. I see that the fine has been changed and includes this comment: “# this doesn’t work, not all chains are ready immediately after it starts”. So, that is part of why my ipset isn’t working, but not why it’s completely missing. This service was compliant a few years ago, so, I guess I’ll need to try to figure out how to fix it. How do you update your firewalld ipsets? Is there a better way to do that other than to write my own service?
-
firewall-cmd --permanent --get-ipsetsreturns an empty line -
contents of
/etc/firewalld/ipsets/
-rw-r--r--. 1 root root system_u:object_r:firewalld_etc_rw_t:s0 853 Oct 7 21:01 dshield-blocklist.xml
-rw-r--r--. 1 root root system_u:object_r:firewalld_etc_rw_t:s0 197 Oct 7 20:01 dshield-blocklist.xml.old
-rw-r--r--. 1 root root system_u:object_r:firewalld_etc_rw_t:s0 368 Jan 15 2020 local-blocklist.xml
-rw-r--r--. 1 root root system_u:object_r:firewalld_etc_rw_t:s0 339 Nov 16 2019 local-blocklist.xml.old
-rw-r--r--. 1 root root system_u:object_r:firewalld_etc_rw_t:s0 24064 Oct 8 01:01 misp6-blocklist.xml
-rw-r--r--. 1 root root system_u:object_r:firewalld_etc_rw_t:s0 24064 Oct 7 20:01 misp6-blocklist.xml.old
-rw-r--r--. 1 root root system_u:object_r:firewalld_etc_rw_t:s0 257 Sep 22 2019 PrivateBlockList.xml.old
I’m not sure what I’m looking at in this directory. I only used firewall-cmd to manage the ipsets; I didn’t know they where stored here. These all look correct, though. The automated update scripts broke on Oct 7, after a dnf update, just like I said. But, I don’t see any reason why these ipsets are not currently active. They where active, and now they are not.
So, what should I do? It looks like firewalld is complaining about dshield-blocklist. Should I reset these? Do I just remove them from this directory and reload? I can’t believe that firewalld broke so badly and I didn’t have any indication.
Okay, so I ran firewall-config and everything looked fine. All the IPsets where there, as I remember from when I set them up a few years ago. But, from there, I reloaded the firewall, and now, it’s much happier. firewall-cmd --permanent --get-ipsets now returns the three IPsets that firewall-config showed me. So, I’m guessing that whatever changed was not trivial, perhaps some file format change, and firewall-config was able to read the old configuration and wrote out a configuration in the proper new format? Something like that, anyway. Really don’t understand why such a breaking change was included in an update to a stable version. That sucks. If I didn’t actually read my logs like a good sysadmin, I might not have ever noticed.
So, I firewalld is working now. But, I need to know how to automatically update these ipsets. What do other people do for this? Should I try figure out whats wrong with the service I wrote and fix it? That’s the “proper” way, right?
This depends on what you want to populate the IP sets with.
Data sources like DNS, ASN, GeoIP require different implementation.
I use own small service to resolve domains and populate IP sets:
# Domain lists
sudo mkdir -p /etc/firewalld/domains
sudo tee /etc/firewalld/domains/fedora.list << EOF > /dev/null
bodhi.fedoraproject.org
dl.fedoraproject.org
fedoraproject.org
EOF
# Script populating firewalld IP sets
sudo tee /usr/local/sbin/firewall-ipset << "EOF" > /dev/null
#!/usr/bin/bash
for LIST in /etc/firewalld/domains/*.list
do for IPV in 4 6
do IPSET="$(basename "${LIST%.*}")${IPV%4}"
FILE="/tmp/ipset.${IPSET}"
case "${IPV}" in (4) QUERY="A" ;; (6) QUERY="AAAA" ;; esac
while read -r DOMAIN
do resolvectl --legend="no" --type="${QUERY}" query "${DOMAIN}" \
| sed -n -r -e "s|^.*\s(\S+)\s+--\s.*$|\1|p"
done < "${LIST}" > "${FILE}"
firewall-cmd --permanent --delete-ipset="${IPSET}"
firewall-cmd --permanent --new-ipset="${IPSET}" \
--type="hash:net" --family="inet${IPV%4}"
firewall-cmd --permanent --ipset="${IPSET}" \
--add-entries-from-file="${FILE}"
firewall-cmd --reload
rm -f "${FILE}"
done &> /dev/null
done
EOF
sudo chmod +x /usr/local/sbin/*
# Services and timers
sudo tee /etc/systemd/system/firewalld-ipset.service << EOF > /dev/null
[Unit]
Description=firewalld IP set service
After=firewalld.service
PartOf=firewalld.service
[Service]
Type=oneshot
ExecStart=firewall-ipset
[Install]
WantedBy=firewalld.service
Also=firewalld-ipset.timer
EOF
sudo tee /etc/systemd/system/firewalld-ipset.timer << EOF > /dev/null
[Unit]
Description=firewalld IP set timer
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable firewalld-ipset.service firewalld-ipset.timer
sudo systemctl restart firewalld-ipset.service firewalld-ipset.timer
Ah ha. Thank you, I can work with that. I didn’t know about the timers. That’s cool.
Any theory about what happened in my case? Why did these IPsets break?
The error message you posted above is about some missing chains, so it is most likely related to nftables or iptables.
It looks like your method to update the IP sets critically depends on the firewalld backend and the specifics of its low-level implementation, which actually does not provide a stable API that can be reliably utilized.
To determine the core cause of the issue requires to reproduce your setup and analyze the code you have been using.
Can you provide documentation about these API changes and which commands are “stable”?
My code looks a lot like yours, btw. new-ipset, swap set, delete old ipset. I guess I was altering the live IPsets, then saving them to the permanent set. You are changing the permanent set and then reloading. I don’t see either as obviously better than the other. They both seem consistent.
Changes that break the API are usually posted in the changelog and release notes:
The complete list of changes requires to analyze the history of commits in a specific branch for both upstream and downstream repositories:
If your code is similar to mine, then it is not clear where the error message is coming from.
It should not complain about the chains, unless that error message is a different issue.
Also note that a few years ago firewalld switched from iptables and standalone ipset implementation to nftables with built-in sets.
But this should not affect you, unless you rely on some low-level APIs to populate the IP sets.
Yes, I remember that. At the time, I was using iptables directly. But, that broke, and I read about firewalld, which is how we got here. Thanks for your help.
1 Like
Well, no joy. These rules still aren’t working correctly. firewalld is throwing this message:
1004: internal:0:0-0: Error: Could not process rule: Too many open files in system
Hmm, so, this broke, again, and I don’t understand what’s going on. So, the last time I started firewall-config, it seemed to be working, that is, showing me things that look familiar, including IPsets that I setup. Now, those ipset files are still present in /etc/firewalld/ipsets, but firewall-config isn’t happy. I cannot seem to find the IPsets anywhere there now. I cannot seem to click on anything in the “active bindings” pane. When I try to reload, it gives that error many times.
Hmm, okay, I now see that, the interface name changed, and now the list of interfaces in firewall are wrong. I’m now seeing a list of interfaces in the Active Bindings pane, so that’s better. And, I didn’t notice before, so I don’t know what was there, but it looks like these are just old interface names. But, I don’t see how to fix it. When I try to manually remove this old interface, firewall-config crashes. And when try to add a real interface, it gives an error.
This is best to report to the firewalld bug tracker as I cannot reproduce the problem without knowing your complete configuration.
Remove statically assigned interfaces from the firewalld configuration:
sudo sh -c 'sed -i -e "/<interface\s[^>]*>/d" /etc/firewalld/zones/*.xml'
sudo firewall-cmd --check-config
sudo firewall-cmd --complete-reload
Then assign connections to zones with NetworkManager and it will dynamically bind the correct interface names:
sudo nmcli connection show
sudo nmcli connection modify CONNECTION connection.zone ZONE
sudo nmcli connection up CONNECTION
Ah ha, that makes some sense. NM wasn’t running. I didn’t know FWd talked to NM; I read that in the docs for FWd and that got me going in the right direction. Once I fixed NM, FW showed the correct new interface, and after I save active to permanent and reloaded, the old, bogus IFs were gone. All my IPsets got deleted, but that’s fine, I need to rewrite the service anyway.
1 Like