Introduction of IDS and WAF

This is a beginner's guide of IDS and WAF. It explains the problem of Firewall and how the IDS and WAF covers those problems. To understand the merit of WAF, let’s discuss Firewall and IDS first.

Firewall

To protect from attack/intrude/breach/compromise via a network, Firewall is a key element of network defense. Firewall provides network layer protection against malicious connections, mainly by inspecting the incoming packet's header, not by inspecting the payload.

A traditional Firewall is designed to control access to the network, and it operates at the network layer (L4 of the OSI model) by examining packet headers and determining whether to allow or block the traffic based on predefined rules. While Firewall can block some types of the application layer (L7 of the OSI model), it is not designed to protect web based application itself or prevent attacks that can pass through the allowed ports and protocols. And Firewall is not designed to detect attacks within the internal network. 

So there is a problem in relying on only Firewall. IDS and/or WAF can provide another layer of network protection. IDS and WAF can not only detect an attack that has bypassed Firewall defenses but it is also able to detect the attacks which are launched within the network (inside of the Firewalls). IDS and WAF are needed  to help mitigate the risks of cyber threats that can bypass Firewall defenses.

Protection against application layer attacks

Application layer attacks are indistinguishable from normal accesses at the network layer (L4) because it happens in the application layer (L7), thus being unable to protect by Firewalls. Therefore, to detect and defend against application layer attacks, it is necessary to analyze application layer data that is transmitted on packet payloads.

How can we detect application layer attacks that Firewalls cannot detect?

The first step is to analyze the packets by human: mirror all packets on the Web server or in its path and analyze with WireShark or tcpdump. However, the amount of communication over the internet has increased dramatically and applications have become more complex, it is impossible to analyze each packet one by one.

That is too hard so it needs some security tools to reduce the human’s workload by filtering out packets buried in normal accesses, logging them only when they appear to be attacked, and notifying the administrator of any attacks or denying requests that appear to be attacked before the administrator has a chance to respond. That is IDS and WAF.

IDS

An Intrusion Detection System (IDS) is an application that monitors and analyzes the network traffic and system activity for signs of unauthorized access, malicious behavior, or policy violations.

The primary purpose of an IDS is to detect potential security breaches and alert security personnel and/or system administrators so that they can investigate the packets and respond to the threats. IDS basically only warns the administrator and does not stop the attack by itself.

IDS can be classified into 2 types - one is Network Based Intrusion Detection System (NIDS): This type of IDS is deployed at key points on a network to monitor network traffic and detect suspicious activity. NIDS typically analyze packets of data flowing over a network and compares them against a set of preconfigured rules or signatures to identify possible security breaches. The most famous (and de-facto standard) IDS is Snort, which is open-source network-based IDS that can analyze network traffic and detect any network-based attacks. It uses a signature-based detection method, monitors all the packets, and compares network packets against signatures to identify potential threats (IDS can monitor both of the inbound and outbound traffics – monitoring outbound traffic is to monitor the information leakage, detect C&C by a malicious attacker, Jump host activities, or etc).

Another type is a Host-based Intrusion Detection System (HIDS): this type of IDS is deployed on individual hosts or servers to monitor activity and detect signs of malicious activity or policy violations.

IPS

Intrusion Prevention System (IPS) is a variation of IDS or a feature of IDS. IPS can not only detect potential security breaches but also prevent the breaches before they can cause harm. The primary function of an IPS is to block suspicious traffic or activity before it comes in. The detection mechanism of IPS is the same as IDS, and the key difference IPS is that it can actively prevent attacks by blocking malicious access. However, IPS can also generate false positives, which lead to legitimate traffic being blocked.

Detection methods of the application layer attack

How to detect the application layer attack which bypasses Firewall - there are mainly 2 methodologies. One is so-called signature-based detection, which does pattern matching with pre-defined signatures, and another is anomaly detection.

The signature-based detection method uses a database of the character strings which are known attack patterns, that is called "signature", to identify potential security breaches. This is a blacklist approach.

On the other hand, Anomaly detection uses a white list approach. Learn the normal (daily) behavior and set some parameters, and detects abnomalies that exceed such parameters in network and/or system activity, which may indicate malicious behavior.

Anomaly detection also uses a blacklist approach. For example, if there is a file that contains sensitive information and is supposed to not be accessed frequently, the rule of detecting a malicious attempt is to warn the administrator if the file is accessed by someone. Unlike the signature detection method, this method can handle unknown attacks, but it may not be able to detect attacks that mimic normal behavior and may have many false positives (false positives are detected as attacks when they are normal user behavior).

And one more detection method we need discuss recently: Machine Learning detection. It may be included in the Anomaly detection, but the main purpose of machine learning detection is to detect unknown attacks. Many commercial WAFs uses machine learning for detection. 

Web Application Firewall

Among all the protocols, HTTP communication accounts for a very large percentage of network traffic and it has been becoming complex. Although IDS can detect malicious packets on HTTP(S) traffic, another security solution that focuses on complexity on HTTP traffic was required. That is Web Application Firewall (WAF), which is designed to protect Web applications from various types of attacks, typically cross-site scripting (XSS), SQL injection, OS command injection, and other application layer attacks. WAF acts as a filter between a web application and the external network, examining all incoming and outgoing HTTP(S) traffic and blocking (like IPS) any requests that do not meet specific security policies.

Like IDS, WAF works by analyzing the payload of packets of incoming and outgoing traffic and uses both signature-based and anomaly detection methodology to detect a potential threat, block or limit traffic from suspicious sources, and restrict certain types of HTTP(S) requests. The most popular open-source WAF project is the ModSecurity module of Apache and NGNIX. Since ModSecurity is a module of web server, it needs Apache or NGINX as a proxy to use it. For commercial WAF solutions, F5 BigIP AWAF and NGINX App Protect are our products.

Differences

IDS and WAF differ in their approach to security and the scope of their protection. IDS's scope of protection is all the protocol. IDS can detect attacks against any system or application on the network, while WAF is designed specifically to protect web applications (Even if the application is not a web application, if that uses HTTP communication, WAF can protect it). And, while IDS just monitors the traffic and does not block the malicious traffic (If IPS is not activated), most of the WAF is designed to block the traffic when it detects malicious traffic (it depends on the configuration, though).

Since IDS covers all the protocols, it requires more customization than that of WAF. IDS signatures must be designed for the specific protocols. On the other hand, WAF can be deployed with a set of common signatures of well-known attacks on HTTP (if WAF is used for API protection, it needs more customization).