HTTP Brute Force Mitigation Playbook: Bot Profile for Brute Force Mitigations - Chapter 5

Bots traffic can be challenging to handle in general and bots that do brute force or credentials staffing In specific requires powerful bot detection. Advance WAF bot profile released in version 14.1. is a powerful bot manager for protecting your web application

As of version 14.1  Bot profile is a dedicated service that works in parallel to the ASM policy and to the Dos profile.

This article describes the configuration options of the bot profile in the Advance WAF version 14.1 and above when dealing with brute force or credential stuffing attacks. 

When to use it:

General use and visibility : 

Bot profile is a powerful on premises bot manager that allows control over the type of traffic accessing the web application. By classifying the bots traffic to known good or known bad it is easier to manage the bot traffic. 

Bot profile can be used before or during automated attack. The optimize approach is to set the bot profile in transparent mode (no traffic will be blocked) and understand the traffic arriving at the application. Only then make educated decisions on traffic SAP (site access policy)

The trusted bot allows search engines and monitoring tools to continue working as expected and the reporting provides valuable information on the traffic

Brute force

For brute force attacks bot profile can detect and prevent the automation of the attack agent by anomalies, user agent and other detection technique.

Pre configuration

Bot profile and logging profile

The new bot profile should be assign to the relevant virtual server. Under: Local Traffic  ->  Virtual Servers : Virtual Server List  ->  virtual_server_name  -> Bot Defense Profile -> from the defaults options choose “bot-defense”

The bot defense should also be assigned with log profile. Choose the local-bot-defense from the available and move it to the selected. While the default logging profile is good enough you can create a new logging profile for bot defense under Security  ->  Event Logs -> Logging Profiles -> create -> choose

Click update.

DNS resolver

Bot profile uses a revers DNS lookup check to identify known good search engines. Therefor a DNS server and a DNS resolver must be configured on the BIGIP.

DNS server is under System -> Configuration -> Device -> DNS

DNS resolver is under Network -> DNS Resolver -> DNS Resolver List

Caution : using bot profile to mitigate attacks and NOT configuring the DNS resolver may affect analytics data or monitoring crawlers such as google analytics etc .

Bot profile concept

Bot profile is the unification of several mechanism into one that is now called bot profile.

The new bot profile allows the automation of classifying request to bots types and according to it the applied prevention policy is done.

• Browser – classified as a browser, additional checks will be done such as browser capabilities, sources scoring and more 
• Mobile App - Mobile App with SDK, Mobile App
• Trusted bot - Signature categories: Search Engine
• Untrusted bot - Signature categories: Crawler, HTTP Library, Headless Browser, RSS Reader, Search Bot, Service Agent, Site Monitor, Social Media Agent, Web Downloader
• Suspicious Browser - Anomaly categories: Suspicious Browser Type, Suspicious Browser Extension
• Malicious Bot :
• Signature categories: DOS Tool, E-Mail Collector, Exploit Tool, Network Scanner, Spam Bot, Spyware, Vulnerability Scanner, Web Spider
• Anomaly categories: Browser Automation, Malicious Browser Extensions, Headless Browser Anomalies, Browser Masquerading, Mobile App Automation, Mobile App Masquerading, Illegal Mobile App, Search Engine Masquerading, OWASP Automated Threat Anomalies, Classification Evasion

Each class type is getting the prevention policy that was defined in the template.

Bot profile - General settings

Profile Template is defied once when the bot profile is created and includes several modes:

• Relaxed Mode – security policy that is not instructive because no java script is injected for source identification nor for device ID java script. The only blocked bots are the ones that are classified as malicious bots. The rest of the bots will be in alarm only.
• Balanced Mode – security policy that is more intrusive and includes java script injection to identify the sources and inject the java script to generate device ID. This mode is considered more intrusive because the java script is injected in the first response ( after access). The blocked by default bots classification are malicious bot, suspicious browser will get CATPCHA and unknown bots will get rate limit.
• Strict Mode – security policy that is most intrusive and inject the java script on the first request ( before access) ,. This approach is considered intrusive but much more accurate in detection. By default all classified bot types will be blocked except trusted bot that will be alarmed.

It is recommended to do the initial implementation with relaxed policy mode for visibility and false positive reduction so that when an attack starts the profile can be moved to blocking mode immediately.

When under brute force attack the setting should be changed to set to strict mode so that the attackers will be blocked from any bot types. Note that this includes blocking other bots that might be legitimate but during attack the priority is to prevent the “you got hacked” scenario.

In cases where the web application must have availably over confidentiality then the relax template should be used and gradually adding other policy elements that will tradeoff between the attack mitigation (confidentiality ) and the web application performance ( availability )

Mitigation settings tab

Bot profile classify the bots traffic into the following types:

Trusted Bot - Bots that are detected using search engine signatures. Note that there is no anomaly detection for Trusted Bots, It means that even if anomalies where found, we pass this request. 

Untrusted Bot - Bots that are detected using signatures for untrusted bots. This group is covered by bot signature category Benign (excluding the Search Engines).

Suspicious Browser - browser clients for which anomalies of Suspicious Browser category were detected.

Malicious Bot - Bots that are detected using malicious bot signatures or malicious behavioral anomalies.

Unknown - Bots that were not detected by neither bot signatures nor behavioral anomalies.

For example

TOR bowsers - should get a captcha to verify that it is not automated . but it will get blocking if the SAP defines that it is not allowed.

Suspicious HTTP Headers Presence or Order - A missing header can be indicative of bot activity

Browser verification tab

Allows to fine tune and override the template setting for the level of intrusiveness

Browser verification - defines the intrusive level of the browser classification process.

• Challenge free – means that no java script injection will be done and only passive detection will be done
• Verify after access – means the java script will be injected in the first response
• Verify before access – means that the java script will be injected in the first request

The grace period is the time between a successful CAPTCHA challenge solution and when another CAPTCHA challenge can be sent for a request.

Device ID mode – define when the java script for generating Device ID will be injected, after access or before access which is first request or first response corresponding.

Verification and Device-ID Challenges in Transparent Mode enables to allow challenges and JavaScript injects even though the profile is configured with Transparent Enforcement Mode and no mitigations will be done. The challenges will be logged.

Single Page Application if your website is a Single Page Application, meaning a web application that loads new content without triggering a full page-reload. The system will inject JavaScript code to every HTML response. This will allow handling browser verification challenges, Device ID challenges and CAPTCHA without requiring to reload the page.

Mobile Applications tab

In Mobile Applications setting you can define how requests from mobile application built with the Anti-Bot Mobile SDK clients are handled. This feature requires an Anti-Bot Mobile SDK license to be operational. [Click]

For applications that don’t use Anti-Bot Mobile SDK add an “Allowed Application” signature in Applications without Anti-Bot Mobile SDK

Note that this relies on analyzing the User-Agent header field only, which can be easily spoofed, and should be used with care.

Signature Enforcement tab

Bot signature are powerful first line of defense tool that should be enabled in transparent mode to understand the traffic arriving to the web application.

When under brute force it is recommended to enforce the bot signature to filter and eliminate traffic access to the web application.

Note that white listed sources including IP’s and search engines are allowed if configured properly as noted in the white list tab.

In Signature Enforcement setting you can change the enforcement setting.

If a signature is in staging, one of two states are shown:

• Ready to be enforced
• Waiting for more traffic samples

Use the filter to find one or more signatures from the list.

Bot defense signature pool

The lists of Bot Signatures and Signatures Categories are located under Security ›› Bot Defense : Bot Signatures.

A new signature for a specific bot can be created manually

This is covered in more details :  Writing Custom Attack Signatures

Bot signatures are being update periodically by F5 , this process is document : Updating Attack and Bot Signatures

Whitelist tab

The Whitelist setting is used to disable mitigation actions or browser verification and Device ID challenges.

It is recommended to white list known to be good to exempt them from being classified by the browser verifications process.

There is also an option to white list specific IP’s to specific URL’s

Each IP that is added to the white list can be exempt from :

mitigation action – the actual action done on the classified source e.g. alarm, CAPTCHA ,Rate Limit, Block , TCP Reset, Honeypot page , Redirect to Pool

browser verification and Device ID challenges - the classification process of injecting java script to classify the source with capabilities script or device ID injection.

Reporting

The Bot Traffic summary is located under Security ›› Event Logs : Bot Defense : Bot Traffic.

The graph display the bot types and traffic statistics which will indicate the percentages of human traffic versus bots traffic.

When under brute force attack this graph helps in investigating the offending sources.

The bot request page provides a list of requests that got caught on the bot profile.

Drilling down on requests allows the view by bot type or by incident as well as the mitigation type that was done on them

The list of bots is powerful tool to see the sources that are hitting the site and helps understand the type of bots used to execute the brute force attack.

The request log shows the reasons for the request classification in the Request details section ( click on all details )

The bot details describe the bot name, calls and category , in cases where the attack is done via browser the bot name will be the browser name.

For example: if a brute force is done via burp proxy and a Firefox the bot name will be :

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36

The bot request page allows various filters to be applied to help with better search of the offending brute force bot, the following filter options are helpful when tracking the brute force sources:

Specific URL – specify the login URL to see the type of bots access the login. One if those sources is an offending boter (under advance filter)

Time range – specify the time range that the brute force occur to examine the bots that access the web application during the attack. (under advance filter)

Source IP address – specify the suspicious source IP to see the type of bots arriving from this Ip.

more Resources: Bot defense configuration details - https://support.f5.com/csp/article/K42323285