ABOUT DEVCENTRAL

DevCentral News Technical Forum Technical Articles Technical CrowdSRC Community Guidelines DevCentral EULA Get a Developer Lab License Become a DevCentral MVP

RESOURCES

Product Documentation White Papers Glossary Customer Stories Webinars Free Online Courses F5 Certification LearnF5 Training

SUPPORT

Manage Subscriptions Professional Services Professional Services Create a Service Request Software Downloads Support Portal

PARTNERS

Find a Reseller Partner Technology Alliances Become an F5 Partner Login to Partner Central

---

©2024 F5, Inc. All rights reserved.

While estimates vary, it is believed that more than half of the Internet traffic is being generated by bots, out of which unwanted or malicious ones (like spam or malware bots) account for more than half of the traffic, the remaining traffic being generated by “good” bots (like crawlers or feed fetcher bots). It is therefore important to differentiate between different classes of bots and treat them according to site-specific security policies.

The Unified Bot Defense profiles, first released in TMOS version 14.1, package bot protection features like Bot Signatures and Proactive Bot Defense previously found in L7 DoS profiles and Web Scraping protection found in ASM policies.

Configuring Unified Bot Defense profiles through BIG-IQ ensures configuration consistency over the centralized managed BIG-IP estate and enhanced reporting capabilities.

This article will guide you through the configuration of Unified Bot Defense profiles using BIG-IQ CM User Interface.

It is assumed that the BIG-IP device where the Bot Defense profile will be deployed is currently managed by the BIG-IQ cluster, at least one BIG-IQ Logging Node / Data Collection Device is available and the Virtual Server to be protected is already configured (in the example below, VS_12BOX) - the configuration of these elements will not be part of this article.

This article covers:

• configuring the Shared Security / Application Security Event Logging Profile
• configuring the Bot Defense profile
• monitoring the Bot Defense profiles

Configuration of the Security Log Profile

1. Go to Configuration->LOCAL TRAFFIC->Pools, click Create and fill in the settings:

-Name: Pool_DCD

-Device: select the BIG-IP device

-Health monitors: gateway_icmp

-New member: - Select \"New Node\"

- Address: Type the Log Node / DCD IP address

- Port: 8514 (this is the port that Web Application Security Service is listening on the Logging Node / DCD)

Note: Ensure that the Logging Node / Data Collection Device has the Web Application Security Service activated and the managed BIG-IP has LTM, SSM and ASM services Discovered/Imported.

2. Go to Configuration->LOCAL TRAFFIC->Logs ->Log Destination, click Create and fill in the settings:

- Name: Log_dst_HSL_DCD

- Type: Remote High-Speed

- Device: select the BIG-IP device

- Pool: select /Common/Pool_DCD

3. Go to Configuration->LOCAL TRAFFIC->Logs ->Log Destination, click Create and fill in the settings:

- Name: Log_dst_Splunk_DCD

- Type: SPLUNK

- Forward to: select /Common/Log_dst_HSL_DCD

4. Go to Configuration->LOCAL TRAFFIC->Logs ->Log Publishers, click Create and fill in the settings:

- Name: Log_pub_DCD

- Log destinations: select /Common/Log_dst_Splunk_DCD

5. Go to Configuration->LOCAL TRAFFIC->Pinning Policies and select the BIG-IP device - Filter the available Local Traffic Manager (LTM) objects by selecting Log Publishers from the dropdown menu - Check Log_pub_DCD and click Add Selected button

6. Go to Configuration->SECURITY->Shared Security ->Logging Profiles, click Create and fill in the settings:

-Name: Log_bot_protect_demo

-Bot Defense:

-Status: Enabled

-Local Publisher: Enabled

-Remote Publisher: /Common/Log_pub_DCD 

Attach the Log_bot_protect_demo log profile to the protected Virtual Server (in this example, VS_12BOX VS)

1. Go to Configuration->SECURITY->Shared Security->Virtual Servers and click on VS_12BOX VS

2. Select the Log_bot_protect_demo log profile for Logging profiles

Deploy the configuration to the BIG-IP

1.    Go to Deployment->EVALUATE & DEPLOY-> Local Traffic & Network, create a new Deployment. Once the evaluation has finished, click on Deploy.

2. Go to Deployment->EVALUATE & DEPLOY-> Shared Security, create a new Deployment. Once the evaluation has finished, click on Deploy.

Configuration of the Bot Defense Profile

Go to Configuration->SECURITY->Shared Security ->Bot Defense-> Bot Profiles, click Create and fill in the settings:

-Name: bot_defense_demo

-Enforcement Mode: Blocking

-Profile Template: Strict

-Browser Verification:

-Browser Access: Allowed

-Browser Verification: Verify After Access (Blocking)

Note: As per K42323285: Overview of the unified Bot Defense profile the available options for the configuration elements used in this examples are:

Enforcement Mode: Select one of the following modes, depending on the readiness of your application environment and requirements:

• Transparent—The system logs traffic mitigation and verification actions, according to your logging profile settings, but does not provide the following:
• JavaScript-based verification.
• Device ID collection.
• CAPTCHA challenge.
• Blocking—The system performs traffic mitigation and verification, and logs them according to your logging profile settings.

Profile Template: The template you select determines the default values for mitigation and verification settings. However, you can customize these settings to meet your application security requirements.

After the system saves the profile, you can't change this setting.

The following list contains descriptions of the available templates:

• Relaxed—Performs basic verification of browsers and blocks malicious bots based on bot signatures.
• Balanced—This is the default selection. Performs advanced verification of browsers, including:
• CAPTCHA challenges for suspicious browsers.
• Anomaly detection algorithms and bot signatures to detect and block malicious bots.
• Limiting the total request rate for unknown bots.
• Strict—This is the strictest policy; it has settings that:
• Only allow browsers access if they pass proactive verification.
• Block all bots except trusted ones.

Browser Verification: Specifies what and when the system sends challenges.

• None—The system does not perform JavaScript and header-based verification. However, some anomaly detection (such as Session Opening) still occurs.
• Challenge-Free Verification—The default value when Profile Template is set to Relaxed. The system performs header-based verification but does not perform JavaScript verification.
• Verify Before Access—The default value when Profile Template is set to Strict. The system sends a white page with JavaScript to challenge the client. If the client fails the challenge, the system performs the configured mitigation action and reports the anomaly. If the client passes the challenge, the system forwards the request to the server.
• Verify After Access (Blocking)—The default value when Profile Template is set to Balanced. The system injects a JavaScript challenge in the server response prior to sending the response to the client. If the client fails the challenge, the system performs the configured mitigation action and reports the anomaly. If the client passes the challenge, the system forwards the request to the server.
• Verify After Access (Detection Only)—The system injects JavaScript challenge in the server response prior to sending the response to the client. If the client fails the challenge, the system only reports the anomaly but does not perform any mitigation action. If the client passes the challenge, the system forwards the request to the server.

Device ID Mode: A unique identifier that BIG-IP ASM creates by sending JavaScript to get information about the client device. The default value for this setting is determined by your selection in Profile Template (under General Settings). F5 recommends you use the default values set by the Profile Template you selected unless you have specific application requirements.

• None—The default value when Profile Template is set to Relaxed. The system does not send JavaScript to collect the device ID.
• Generate After Access—The default value when Profile Template is set to Balanced. The system injects the JavaScript in the server response before forwarding to the client. This is less intrusive and has less of a latency impact.
• Generate Before Access—The default value when Profile Template is set to Strict. The system sends the JavaScript challenge to the client before forwarding the client request to the server. This guarantees that every request that reaches the server has a device ID. This has more of a latency impact compared to the previous option. The system blocks bots that attempt to present themselves as browsers but are unable to execute the JavaScript challenge.

Attach the bot_defense_demo bot protect profile to the VS_12BOX VS

1. Go to Configuration->SECURITY->Shared Security->Virtual Servers and click on VS_12BOX VS

2. Select the bot_defense_demo profile for Bot Defense profile

Deploy the Bot Defense profile to the BIG-IP

Go to Deployment->EVALUATE & DEPLOY-> Shared Security, create a new Deployment. Once the evaluation has finished, click on Deploy

Monitoring Bot Defense Profiles

To monitor Bot Protection operation, check the Monitoring->DASHBOARDS->Bot Traffic Dashboard and Monitoring->EVENTS->Bot->Bot requests logs

While estimates vary, it is believed that more than half of the Internet traffic is being generated by bots, out of which unwanted or malicious ones (like spam or malware bots) account for more than half of the traffic, the remaining traffic being generated by “good” bots (like crawlers or feed fetcher bots). It is therefore important to differentiate between different classes of bots and treat them according to site-specific security policies.

The Unified Bot Defense profiles, first released in TMOS version 14.1, package bot protection features like Bot Signatures and Proactive Bot Defense previously found in L7 DoS profiles and Web Scraping protection found in ASM policies.

Configuring Unified Bot Defense profiles through BIG-IQ ensures configuration consistency over the centralized managed BIG-IP estate and enhanced reporting capabilities.

This article will guide you through the configuration of Unified Bot Defense profiles using BIG-IQ CM User Interface.

It is assumed that the BIG-IP device where the Bot Defense profile will be deployed is currently managed by the BIG-IQ cluster, at least one BIG-IQ Logging Node / Data Collection Device is available and the Virtual Server to be protected is already configured (in the example below, VS_12BOX) - the configuration of these elements will not be part of this article.

This article covers:

• configuring the Shared Security / Application Security Event Logging Profile
• configuring the Bot Defense profile
• monitoring the Bot Defense profiles

Configuration of the Security Log Profile

1. Go to Configuration->LOCAL TRAFFIC->Pools, click Create and fill in the settings:

-Name: Pool_DCD

-Device: select the BIG-IP device

-Health monitors: gateway_icmp

-New member: - Select \"New Node\"

- Address: Type the Log Node / DCD IP address

- Port: 8514 (this is the port that Web Application Security Service is listening on the Logging Node / DCD)

Note: Ensure that the Logging Node / Data Collection Device has the Web Application Security Service activated and the managed BIG-IP has LTM, SSM and ASM services Discovered/Imported.

2. Go to Configuration->LOCAL TRAFFIC->Logs ->Log Destination, click Create and fill in the settings:

- Name: Log_dst_HSL_DCD

- Type: Remote High-Speed

- Device: select the BIG-IP device

- Pool: select /Common/Pool_DCD

3. Go to Configuration->LOCAL TRAFFIC->Logs ->Log Destination, click Create and fill in the settings:

- Name: Log_dst_Splunk_DCD

- Type: SPLUNK

- Forward to: select /Common/Log_dst_HSL_DCD

4. Go to Configuration->LOCAL TRAFFIC->Logs ->Log Publishers, click Create and fill in the settings:

- Name: Log_pub_DCD

- Log destinations: select /Common/Log_dst_Splunk_DCD

5. Go to Configuration->LOCAL TRAFFIC->Pinning Policies and select the BIG-IP device - Filter the available Local Traffic Manager (LTM) objects by selecting Log Publishers from the dropdown menu - Check Log_pub_DCD and click Add Selected button

6. Go to Configuration->SECURITY->Shared Security ->Logging Profiles, click Create and fill in the settings:

-Name: Log_bot_protect_demo

-Bot Defense:

-Status: Enabled

-Local Publisher: Enabled

-Remote Publisher: /Common/Log_pub_DCD 

Attach the Log_bot_protect_demo log profile to the protected Virtual Server (in this example, VS_12BOX VS)

1. Go to Configuration->SECURITY->Shared Security->Virtual Servers and click on VS_12BOX VS

2. Select the Log_bot_protect_demo log profile for Logging profiles

Deploy the configuration to the BIG-IP

1.    Go to Deployment->EVALUATE & DEPLOY-> Local Traffic & Network, create a new Deployment. Once the evaluation has finished, click on Deploy.

2. Go to Deployment->EVALUATE & DEPLOY-> Shared Security, create a new Deployment. Once the evaluation has finished, click on Deploy.

Configuration of the Bot Defense Profile

Go to Configuration->SECURITY->Shared Security ->Bot Defense-> Bot Profiles, click Create and fill in the settings:

-Name: bot_defense_demo

-Enforcement Mode: Blocking

-Profile Template: Strict

-Browser Verification:

-Browser Access: Allowed

-Browser Verification: Verify After Access (Blocking)

Note: As per K42323285: Overview of the unified Bot Defense profile the available options for the configuration elements used in this examples are:

Enforcement Mode: Select one of the following modes, depending on the readiness of your application environment and requirements:

• Transparent—The system logs traffic mitigation and verification actions, according to your logging profile settings, but does not provide the following:
• JavaScript-based verification.
• Device ID collection.
• CAPTCHA challenge.
• Blocking—The system performs traffic mitigation and verification, and logs them according to your logging profile settings.

Profile Template: The template you select determines the default values for mitigation and verification settings. However, you can customize these settings to meet your application security requirements.

After the system saves the profile, you can't change this setting.

The following list contains descriptions of the available templates:

• Relaxed—Performs basic verification of browsers and blocks malicious bots based on bot signatures.
• Balanced—This is the default selection. Performs advanced verification of browsers, including:
• CAPTCHA challenges for suspicious browsers.
• Anomaly detection algorithms and bot signatures to detect and block malicious bots.
• Limiting the total request rate for unknown bots.
• Strict—This is the strictest policy; it has settings that:
• Only allow browsers access if they pass proactive verification.
• Block all bots except trusted ones.

Browser Verification: Specifies what and when the system sends challenges.

• None—The system does not perform JavaScript and header-based verification. However, some anomaly detection (such as Session Opening) still occurs.
• Challenge-Free Verification—The default value when Profile Template is set to Relaxed. The system performs header-based verification but does not perform JavaScript verification.
• Verify Before Access—The default value when Profile Template is set to Strict. The system sends a white page with JavaScript to challenge the client. If the client fails the challenge, the system performs the configured mitigation action and reports the anomaly. If the client passes the challenge, the system forwards the request to the server.
• Verify After Access (Blocking)—The default value when Profile Template is set to Balanced. The system injects a JavaScript challenge in the server response prior to sending the response to the client. If the client fails the challenge, the system performs the configured mitigation action and reports the anomaly. If the client passes the challenge, the system forwards the request to the server.
• Verify After Access (Detection Only)—The system injects JavaScript challenge in the server response prior to sending the response to the client. If the client fails the challenge, the system only reports the anomaly but does not perform any mitigation action. If the client passes the challenge, the system forwards the request to the server.

Device ID Mode: A unique identifier that BIG-IP ASM creates by sending JavaScript to get information about the client device. The default value for this setting is determined by your selection in Profile Template (under General Settings). F5 recommends you use the default values set by the Profile Template you selected unless you have specific application requirements.

• None—The default value when Profile Template is set to Relaxed. The system does not send JavaScript to collect the device ID.
• Generate After Access—The default value when Profile Template is set to Balanced. The system injects the JavaScript in the server response before forwarding to the client. This is less intrusive and has less of a latency impact.
• Generate Before Access—The default value when Profile Template is set to Strict. The system sends the JavaScript challenge to the client before forwarding the client request to the server. This guarantees that every request that reaches the server has a device ID. This has more of a latency impact compared to the previous option. The system blocks bots that attempt to present themselves as browsers but are unable to execute the JavaScript challenge.

Attach the bot_defense_demo bot protect profile to the VS_12BOX VS

1. Go to Configuration->SECURITY->Shared Security->Virtual Servers and click on VS_12BOX VS

2. Select the bot_defense_demo profile for Bot Defense profile

Deploy the Bot Defense profile to the BIG-IP

Go to Deployment->EVALUATE & DEPLOY-> Shared Security, create a new Deployment. Once the evaluation has finished, click on Deploy

Monitoring Bot Defense Profiles

To monitor Bot Protection operation, check the Monitoring->DASHBOARDS->Bot Traffic Dashboard and Monitoring->EVENTS->Bot->Bot requests logs

Hi,

• Here you use f5 big ip's self ip for sending log to DCD or management ip?
• I use only management ip for DCD reachablitiy not through self ip connectivity. But my issue is bot request log not showing on big iq. I think bot log will sent by pool, through mgmt interface can sent any data by pool? 
• Have any solution for sent log by management interface?

More info in my issue: https://community.f5.com/t5/technical-forum/bot-log-is-not-showing-in-big-iq/td-p/316000

Regards,

Emon Hossain