Implementing application filtering with nftables and Suricata on Debian

05/02/2023 (updated on 05/10/2024) · 14 min read · cybersecurity networking firewall applicationfiltering nftables suricata debian  ·
Share on:

Translations

Overview

In this article, I invite you to discover or rediscover application filtering, with some help to set up application filtering with a netfilter firewall using the nftables utility and the Suricata tool on the Debian GNU/Linux operating system. The aim is to be able to detect and block flows whose behavior does not correspond to those expected for network protocols that are allowed on a firewall.

Introduction

Packet filtering was the first form of firewall introduced in the late 1980s, allowing data packets in transit over a network to be authorized or blocked according to certain criteria. In the early 1990s, an improvement was made to allow packets to be filtered according to established sessions, known as session filtering carried out by a stateful firewall. In the 2000s, a new kind of firewall came up massively the market as a next-generation firewall, in the sense that new security features were added on top of packet and session filtering. They are based on packet inspection on the upper layers of the OSI model, known as Deep Packet Inspection (DPI), where the payload of a packet is analyzed for additional specific criteria to support decision when filtering flows, anomalies, attempts to exploit vulnerabilities and many other things. Let’s then take a look at application filtering, already developed and proposed in the early 1990s, which identifies the type of application, considered here as a network protocol (i.e. FTP, HTTP, Facebook, YouTube, SSH, Telnet, BitTorent, NFS…) in order to filter more precisely than at the Transport level of the OSI model, where mainly protocol (TCP, UDP…) and port values are evaluated. We are going to see how to implement it on an open source platform based on netfilter and Suricata on Debian.

Application filtering

Application filtering is a very useful security feature to reduce the attack surface on networks. This is because authorized network flows are easy channels for attacks (e.g. vulnerability exploitation, worm replication, virus downloading, etc.). This even more true when the types of flows on the way may not correspond to the protocols authorized to flow. For example, a TCP flow on port 80 has been authorized, but SSH flows are flowing via TCP port 80 (theoretically, nothing prevents an administrator from running an SSH server on port 80!). More generally, this feature allows you to control which application flows are authorized to circulate on the network. The term “application” is regularly used here from the network firewall standpoint, and not of the software component installed on a server. We generally distinguish two main families of applications: those related to common known network services (and associated with port values below 1024: FTP, DNS, DHCP, HTTP…) and those related to software (and associated with port values greater than or equal to 1024).

Without application filtering, any application can transit over open services

With application filtering, onl authorized applications can transit over open services

Deployment with nftables and Suricata

Now I would like to show you step-by-step how to set up application filtering with a combination of open source solutions: netfilter via nftables, and Suricata.

nftables

Netfilter is a packet filtering project and software for Linux distributions. It goes along with iptables, a well-known utility for configuring filtering rules, whose successor, nftables, is more flexible and powerful.
iptables and nftables work with tables, which enable different types of operation. In iptables, there are predefined tables (for example, the filter table enables packet filtering, while the nat table enables address translation), whereas in nftables none exist by default, so you have to create them.
Within tables, chains group together all rules, including filtering rules in the case of the filter table. In iptables, there are predefined chains (for example, the input chain matches packets addresssed to the firewall, the output chain matches packets emitted by the firewall and the forward chain matches packets forwarded by the firewall), whereas in nftables there are none by default, so they must be created.

The first step is to install nftables: 

$ sudo -- sh -c 'apt-get update && apt-get install -y nftables'

You can then load the default configuration to create a table of type filter and chains of types input, output and forward. 
# nft -f /etc/nftables.conf

However, I strongly recommend changing the default policy (which is to accept all packets on all 3 chains) at least on the forward chain by blocking all packets by default: 
# nft 'add chain inet filter forward { type filter hook forward priority 0; policy drop; }'

With nftables (and iptables), packet filtering rules describe a given traffic to allow or block it. A session filtering function can be added to allow only packets associated with new or established sessions. In addition, another type of action can be taken: forwarding the characterized packet to another software program for further processing. More precisely, this involves sending the packet to a queue for processing in user space by a third-party application. User space contains all the code (and therefore software) running outside the operating system kernel. I invite you to consult this page to learn how to make rules with nftables.
On nftables, in order to indicate the use of queues for a given traffic, simply use the queue keyword as an action in filter rules. For example, if we want to send TCP traffic on destination port 80 (with session filtering: associated with new or existing sessions) and TCP traffic on source port 80 (with session filtering: associated with existing sessions as this is the return traffic) transmitted by the firewall to a queue, we can define the following rules: 
# nft add rule inet filter forward tcp dport 80 ct state new, established counter queue
# nft add rule inet filter forward tcp sport 80 ct state established counter queue

If we list all the rules from the filter table, here what we get: Si on liste les règles de la table filter, voici alors ce qu’on obtient: 
# nft -a list table inet filter
table inet filter { # handle 4
	chain input { # handle 1
		type filter hook input priority filter; policy accept;
	}

	chain forward { # handle 2
		type filter hook forward priority filter; policy drop;
		tcp dport 80 ct state established,new counter packets 0 bytes 0 queue num 0-3 # handle 5
		tcp sport 80 ct state established counter packets 0 bytes 0 queue num 0-3 # handle 6
	}

	chain output { # handle 3
		type filter hook output priority filter; policy accept;
	}
}

You can also specify in which queue(s) to distribute the traffic to, for instance: 
# nft add rule inet filter forward tcp dport 80 ct state new, established counter queue num 0-3
# nft add rule inet filter forward tcp sport 80 ct state established counter queue num 0-3

In addition, you can also ask for packets to be authorized if no third-party application is monitoring the queues (because it would be blocked by default in this case): 
# nft add rule inet filter forward tcp dport 80 ct state new, established counter queue num 0-3 bypass
# nft add rule inet filter forward tcp sport 80 ct state new, established counter queue num 0-3 bypass

We are now only missing the third-party component that will inspect the packets transmitted in the queues.

Suricata

Suricata is an Intrusion Detection System (IDS) which defines rules based on signatures (relatives to the content of packets’ payload) triggered on characterized traffic, with which actions (e.g. alert, block) are taken. As Suricata is able to read queues, packets can be transmitted from nftables for further processing by Suricata, which then complements session filtering with application filtering. The diagram below illustrates how this mechanism works.

nftables and Suricata can join forces

First, let’s install Suricata: 

$ sudo -- sh -c 'apt-get update && apt-get install -y suricata'

Suricata rules are stored in files located in a directory (by default /etc/suricata/rules/). 
# ls /etc/suricata/rules/
app-layer-events.rules  dhcp-events.rules  dns-events.rules  http-events.rules   kerberos-events.rules  nfs-events.rules  smb-events.rules   stream-events.rules
decoder-events.rules    dnp3-events.rules  files.rules       ipsec-events.rules  modbus-events.rules    ntp-events.rules  smtp-events.rules  tls-events.rules

You can update the predefined rules with the following command: 
# suricata-update

There are several ways of implementing application filtering with Suricata, as the tool is highly modular. The first is to use a default set of rules to detect anomalies in protocol behavior and block flows that do not correspond to what is expected at the application level.
The file we are interested in here is app-layer-events.rules. It contains several predefined rules for detecting application problems. 
# App layer event  rules
#
# SID's fall in the 2260000+ range. See http://doc.emergingthreats.net/bin/view/Main/SidAllocation
#
# These sigs fire at most once per connection.
#
# A flowint applayer.anomaly.count is incremented for each match. By default it will be 0.
#
alert ip any any -> any any (msg:"SURICATA Applayer Mismatch protocol both directions"; flow:established; app-layer-event:applayer_mismatch_protocol_both_directions; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260000; rev:1;)
alert ip any any -> any any (msg:"SURICATA Applayer Wrong direction first Data"; flow:established; app-layer-event:applayer_wrong_direction_first_data; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260001; rev:1;)
alert ip any any -> any any (msg:"SURICATA Applayer Detect protocol only one direction"; flow:established; app-layer-event:applayer_detect_protocol_only_one_direction; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260002; rev:1;)
alert ip any any -> any any (msg:"SURICATA Applayer Protocol detection skipped"; flow:established; app-layer-event:applayer_proto_detection_skipped; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260003; rev:1;)
# alert if STARTTLS was not followed by actual SSL/TLS
alert tcp any any -> any any (msg:"SURICATA Applayer No TLS after STARTTLS"; flow:established; app-layer-event:applayer_no_tls_after_starttls; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260004; rev:2;)
# unexpected protocol in protocol upgrade
alert tcp any any -> any any (msg:"SURICATA Applayer Unexpected protocol"; flow:established; app-layer-event:applayer_unexpected_protocol; flowint:applayer.anomaly.count,+,1; classtype:protocol-command-decode; sid:2260005; rev:1;)

#next sid is 2260006

It is the first rule we are particularly interested in, as it can detect whether flows in one direction (e.g. from an SSH client to a Web server) are not from the same application (protocol) as those in the other direction (e.g. from a Web server to an SSH client).
On the other hand, it is in detection mode, not blocking mode. The alert keyword should therefore be replaced by reject (if you want to discard the packet and send a TCP RST or ICMP Port Unreachable message back to the sender) or drop (if you just want to silently discard the packet). I invite you to consult this page to see all the actions available. 
# sed -s 's/alert/reject/g' /etc/suricata/rules/app-layer-events.rules

In addition to the generic detection and blocking of abnormal flows from an application point of view, I invite you to add detection rules specific to each application. The aim is to detect whether, for a given port, we are observing indeed the expected application.
For example, if you want to block flows on TCP port 80 that are not HTTP, you can create a rule like the following: 

reject tcp any any -> any 80 (msg:"SURICATA Port 80 but not HTTP"; flow:to_server; app-layer-protocol:!http; sid:2271002; rev:1;)

I invite you to consult this page to see how to build Suricata rules. However, here is a list of keywords that can currently be used for applications:

  • http
  • ftp
  • tls (this includes ssl)
  • smb
  • dns
  • dcerpc
  • ssh
  • smtp
  • imap
  • modbus (disabled by default)
  • dnp3 (disabled by default)
  • enip (disabled by default)
  • nfs
  • ikev2
  • krb5
  • ntp
  • dhcp
  • rfb
  • rdp
  • snmp
  • tftp
  • sip
  • http2

It is therefore possible to create a rule for each of these applications, selecting the ports you wish to associate as authorized.

Once the rule file is ready, you need to call it in the rules-files section of the Suricata configuration file (/etc/suricata/suricata.yaml): 

rules-files:
 - app-layer-events.rules

Eventually, you can start Suricata by specifying the number of queues to be monitored: 

# suricata -c /etc/suricata/suricata.yaml -q 0:3 &

Suricata alert logs are stored in a file located in a dedicated directory (by default /var/log/suricata/, which can be modified in the Suricata configuration file).
Here is an example of log when I tested SSH access from a client to an SSH server on TCP port 80: 

# tail -f /var/log/suricata/fast.log 
02/04/2023-08:59:19.150163  [Drop] [**] [1:2271002:1] SURICATA Port 80 but not HTTP [**] [Classification: (null)] [Priority: 3] {TCP} 192.168.1.1:47900 -> 193.167.10.13:80

Here is an example of log when I tested Web access from a client to an SSH server on TCP port 80: 
# tail -f /var/log/suricata/fast.log 
02/04/2023-08:59:38.823127  [Drop] [**] [1:2260000:1] SURICATA Applayer Mismatch protocol both directions [**] [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 193.167.10.13:80 -> 192.168.1.1:47902

The solution with nftables and Suricata that I have presented remains for learning purposes and is not intended for usage in a large production environment. In fact, the configuration remains rather cumbersome (two tools to configure, with rules containing many parameters to specify), while scalability and performance cannot be guaranteed, and the number of applications proposed is limited. That said, even with current products on the market, there are still major challenges to be overcome when it comes to large-scale application filtering.

The challenges of application filtering

There are several challenges to implementing application filtering in a production environment. In addition to expose the problems, I will try to point you towards solutions that will always depend on the context.

  • What to do when there is no application available on the firewall to control certain network protocols?
    It is generally possible to create customized application signatures from real traffic packet captures, or simply by characterizing the traffic as best as you can using fairly generic criteria: direction of connection, TCP timers, values of certain procotolar headers…

  • How do you manage application signature updates as the associated traffic evolves? For example, if you use a signature for an application related to a software component (e.g. a database server) whose network exchanges change following an update on these components: how do you ensure that the signature is updated correctly and on time on the firewall?
    It is generally recommended to secure certain flows by leaving underlying filtering rules based on services without application filtering, until you have confidence in the firewall ability to keep up to date. Also, when updating the firewall application signature base, for critical flows, temporarily cloning application filtering rules with services and without application filtering is a good practice to avoid impacts. If the cloned rule is still used, this is because there is a problem with the application updates used in the original rule.

  • How do you perform application filtering on encrypted flows?
    The only solution is to integrate decryption within the firewall, so that it can inspect the payload of the packets initially encrypted. As this can be quite cumbersome to implement, it is also necessary to accept not to perform application filtering in these cases, beyond filtering on the encryption protocol (TLS for example).

  • How to determine the ports associated with an application in a filtering rule?
    Since, in theory, an application flow can use any network port, it is preferable to associate one or several ports with an application in a filtering rule. This allow to combine the two functions: packet/session filtering and application filtering.

  • Application firewalls are sometimes forced to let the first exchanges pass in order to detect which application is in transit. Although algorithms and signatures are improving over time, this remains a form of vulnerability by design where an attacker could carry out an attack via these very first packets. How can this kind of vulnerability be overcome?
    First of all, this needs to be weighed up against the added value of application filtering compared with simple packet or session filtering. Secondly, a security solution is never alone in securing a communication. At the network level, this can be complemented by traffic analysis using probes such as IDS or Network Detection and Response (NDR) tools, or even enhanced network access controls with proxies, Network Access Control (NAC), network macrosegmentation and microsegmentation, etc.

Conclusion

We have looked at the added value of application filtering, and then at how it works with open source solutions netfilter (via nftables) and Suricata, using an example for educational purposes. Application filtering is now widespread on the firewall market, but is still quite demanding to implement in order to limit operational impact, and should therefore be carefully integrated into a clear strategy.

Sources

Translations