Firewall
Wikimedia production servers use local firewall rules which get configured via Puppet. They are managed via the profile::firewall Puppet class. The firewall configurations configures a DROP policy for all incoming traffic, all outgoing and forwarded traffic is accepted.
For the first ten years Wikimedia servers use and continues to use iptables as the kernel firewall solution. Rules are managed via a tool called Ferm which simplifies rule management via macros and which also deals with atomic rule loading.
Since then, nftables support was added to our Puppet classes. It represents the next generation of Linux firewall support. When introducing nftables the Puppet classes for firewall setup was heavily refactored to move towards configuration primitives which are agnostic of the underlying firewall stack. The migration will be gradual and there are some services which in the foreseeable future will remain using iptables/ferm because they lack upstream support for nftables.
Basic nftables usage instructions
If rules are changed in Puppet, they get automatically reloaded via Puppet. If rules get edited manually for testing/debugging, it's enough to restart the nftables.service on a host.
The current rule list can be displayed using sudo nft list ruleset
Puppet configuration of firewall rules
- firewall::service configures incoming traffic for a service. It configures nftables or ferm rules depending on whether nftables or ferm is configured as the firewall provider.
- ferm::service configures incoming Ferm traffic for a service. This is the define which also gets called by firewall::service if ferm is configured, so if found directly within a role or a profile, it should be migrated to
- nftables::service configures incoming nftables traffic for a service. It should never be directly used (only via firewall::service)
Migrating a ferm::service define to a firewall::service
ferm::service uses Ferm-specific syntax, in order to fix a service to use syntax also compatible with nftables the following settings need to be adapted:
- The port needs to be passed as a single Stdlib::Port or an array of Stdlib::Port, not as a string. Ferm supports looking up service names from /etc/services (e.g. ssh), these need to be replaced with the numerical port numbers
- If the Ferm syntax used a port range like 1000:1200, this can now be passed via the $port_range parameter instead.
- Ferm resolved DNS names locally via the resolve() function, so many $srange or $drange parameters use it. If these ranges are not passed as a string, but as an array of hostnames or IP addresses, the names are not resolved on the Puppet server side before the resulting IP addressed are passed down to ferm or nftables
When a service is fully adapted, it can be renamed to firewall::service to clarify that it's now suitable for both ferm and nftables.
Selecting the firewall provider
The firewall provider defaults to ferm, to switch a role to using Nftables, the Hiera setting profile::firewall::provider needs to be set to nftables. Any host which switches from iptables/ferm to nftables strictly needs a reboot after the provider has been changed. Some of the kernel modules used by iptables cannot be unloaded at runtime without a reboot (I tried various -f hacks, but to no avail). If the old iptables kernel modules are still loaded the constants formerly defined by ferm still persist and this will cause hard to spot issues.
There is no fully supported transition path from nftables back to ferm, so make sure to first test with a canary host before migrating.
There are also a special cases which are running with a firewall configured (e.g. OpenStack Nova which sets up firewall rules internally), for these the Hiera setting should be configured to none.
Basic usage of nftables
The concepts used in nftables are:
- tables - The top-level container that holds chains, sets, etc.
- chains - A container for rules. As opposed to iptables there are no predefined chains, usually you explicitly create a "base" chain.
- families - IPv4, IPv6 or others. Most commonly used is "inet" which means both IPv4 and IPv6.
- sets - A set of items that can be reused elsewhere. Most commonly a set of IP addresses.
- rules - The actual rules, for example "if a source IP is in this set then drop the packet".
The actions to use on these are "list", "add", "delete" etc.
Most basic example to list everything: nft list ruleset.
Also see: man nft, quick reference
The config files generated by puppet can be found under /etc/nftables where 100_base_puppet.nft has the global rules every host gets from base::firewall and /etc/nftables/input/ is where you can find most custom rules for a specific role.
Throttling with nftables
Once a service has been switched to the nftables firewall provider you can use new features, such as throttling based on recent traffic.
To use this include profile::firewall::nftables_throttling in your role class and set some or all of the following keys in Hiera, based on role or a host name:
profile::firewall::nftables_throttling::ensure
The usual ensure parameter, default is present. This just means a new throttling chain will exist, not that it will actually drop any packets.
profile::firewall::nftables_throttling::max_connections
How many parallel connections are allow before throttling, default 32 concurrent connections.
profile::firewall::nftables_throttling::throttle_duration
How long should packets be dropped until new connections are allowed again, default 300 (seconds).
profile::firewall::nftables_throttling::nft_policy
Whether to accept or drop packets, default accept. So unless you actively set this to "drop" no traffic will be affected.
profile::firewall::nftables_throttling::nft_logging
Whether to log actions, default false. Be careful with this as it's relatively easy to fill up disk space.
profile::firewall::nftables_throttling::port
Which port to throttle, default 443.
A practical example where we use this for Gerrit, can be found at gerrit:10640252.
Detailed manual commands with explanation
In practice this is all puppetized, so no need to manually run any of these commands. This example is just here to explain the syntax in more detail.
# throttling with nftables
# for https://phabricator.wikimedia.org/T365259
#
# create a new CHAIN (nft add chain)
# of family "inet" (IPv4 + IPv6)
# in table "base"
# called "throttling"
# of the standard type "filter"
# "hook" it up to the networking stack
# on device "input"
# as a base chain with "priority" -5 (so that it comes before the default for "filter" - 0)
nft add chain inet base throttling { type filter hook input priority -5 \; }
# create 2 new SETs (nft add set) to hold a list of denied hosts, one for each protocol
# of family "inet" (IPv4 + IPv6)
# in table "base"
# called "DENYLIST"/"DENYLIST_v6" (capitalization seems standard)
# of type ipv4/ipv6
# with flags "dynamic" and "timeout"
# and a "timeout" value of 5 minutes
nft add set inet base DENYLIST { type ipv4_addr \; flags dynamic, timeout \; timeout 5m \; }
nft add set inet base DENYLIST_v6 { type ipv6_addr \; flags dynamic, timeout \; timeout 5m \; }
# create 2 new RULES (nft add rule)
# of family "inet" (IPv4 + IPv6)
# in table "base"
# in chain "throttling"
# that add hosts to the DENYLIST sets IF:
# they are NOT from PROD or CLOUD networks and
# they are TCP, destination port 443,... and over 30/minute..then
# add the source IP / source IP v6
nft add rule inet base throttling ip saddr != @PRODUCTION_NETWORKS_ipv4 ip saddr != @CLOUD_NETWORKS_ipv4 tcp dport 443 ct state new, untracked limit rate over 30/minute add @DENYLIST { ip saddr }
nft add rule inet base throttling ip6 saddr != @PRODUCTION_NETWORKS_ipv6 ip6 saddr != @CLOUD_NETWORKS_ipv6 tcp dport 443 ct state new, untracked limit rate over 30/minute add @DENYLIST_v6 { ip6 saddr }
# create 2 new RULES (nft add rule) that ACTUALLY DROP PACKAGES if they are in the DENYLIST sets
nft add rule inet base throttling ip saddr @DENYLIST drop
nft add rule inet base throttling ip6 saddr @DENYLIST_v6 drop
# remove rules nft delete chain inet base throttling nft delete set inet base DENYLIST nft delete set inet base DENYLIST_v6