ASM blocks clients that don't accept cookies

Hi Mike, you're absolutely correct with the first paragraph there.

 

 

The documenation available online though is a little lacking and somewhat ambiguous - it lead my support company, the first-line F5 engineer and myself astray on the expected behaviour of web scraping.

 

 

Below is the final word from F5 on this which clarifies the position and functioning of web scraping:

 

 

_____

 

 

To protect against web scraping, the BIG-IP ASM system attempts to determine if a given HTTP request originates from a human, rather than from a web scraping bot. The mechanisms that BIG-IP ASM uses to determine this are called the Client-Side (CS) challenge, and Client-Side Human User Indicator (CSHUI). One or both may be used depending on the state of the BIG-IP ASM policy settings.

 

 

When the BIG-IP ASM policy enforcement mode is set to Blocking and the Web Scraping Detection violation is set to Block, the BIG-IP ASM will respond to requests first with the CS challenge injection. If the CS challenge is passed, the CSHUI injection occurs.

 

 

*NOTE* The second stage of web scraping has to do with a script that identifies human interaction in the browser, which is Client-Side Human User Indicator (CSHUI). This is the script that takes the grace period into consideration. So grace period has nothing to do with the first stage CS challenge. Grace period only applies to the second stage CSHUI when the BIG-IP ASM policy enforcement mode is set to Blocking and the Web Scraping Detection violation is also set to Block.

 

 

The BIG-IP ASM Client-Side (CS) challenge is an initial step used in the denial-of-service (DoS) prevention, brute force protection, and web scraping prevention components of the BIG-IP ASM anomaly detection feature. The primary function is to ensure that an HTTP request originates from a valid or JavaScript Proper client, and not a bot. A client is considered JavaScript Proper if it meets the following three criteria:

 

 

The client must support JavaScript

 

The client must support HTTP cookies

 

The client must be able to calculate the result of a computational challenge sent by BIG-IP ASM

 

 

Any client that does not satisfy these criteria is considered a bot, and will not be treated as legitimate traffic.

 

 

How CS challenge works

 

 

When an HTTP request is received, the BIG-IP ASM replies with an HTTP response that includes a block of JavaScript that the client must execute in order to complete the computational requirement of the CS challenge. When the client executes the JavaScript, it will automatically return an HTTP POST request to the BIG-IP ASM containing a new cookie with a name of the form TS_75. The value of this cookie will be the result of the computational challenge, which will be verified by the BIG-IP ASM. If the CS challenge is passed, processing continues on to the next appropriate stage for a given request.

 

 

When working with the anomaly detection features that use the CS challenge, it is important to note that the CS challenge injection does not occur immediately with the first response to a given HTTP request. By default, the BIG-IP ASM will not respond with the CS challenge injection until it has received 10 requests per URL within 5 minutes.

 

 

During the first CS challenge stage when web scraping is enabled and in blocking, this Javascript qualifies requests and then intercepts any request to these URL's regardless of the source IP. So once 10 requests have been made to the homepage URL, anyone requesting the homepage URL will need to pass the script before it can continue. If the client is unable to run the challenge, ASM drop the request and subsequent requests are served with the script and dropped until the challenge is answered.