Forum Discussion

THE_BLUE's avatar
THE_BLUE
Icon for Cirrostratus rankCirrostratus
Jul 16, 2020

web scraping

I am receiving we scrapping block , normally after clear cache and cookies this issue solved.

so is it possible to customize the response page when the issue related to web scrapping asking client to clear cache ?

ASM Advanced WAF
devops
iRules
LTM
security
  • Erik_Novak's avatar
    Erik_Novak
    Icon for Employee rankEmployee
    Jul 16, 2020

    Yes, it is possible to customize the blocking response page. In v14 and later, web scraping is a microservice within the bot defense profile. You can customize the response on the general settings page of the bot defense profile. Click the Custom tab as shown here and then edit and preview the response.

    • THE_BLUE's avatar
      THE_BLUE
      Icon for Cirrostratus rankCirrostratus
      Jul 17, 2020

      thank you, but it seems this feature not available in v13

  • Erik_Novak's avatar
    Erik_Novak
    Icon for Employee rankEmployee
    Jul 20, 2020

    Yes it is! In v13, web scraping is configured under Anomaly Detection. After you configure the desired thresholds, go to Security > Application Security > Policy > Response Pages. You can build a custom response page there in exactly the same way. Does this help?

    • THE_BLUE's avatar
      THE_BLUE
      Icon for Cirrostratus rankCirrostratus
      Jul 20, 2020

      yes i found below detalis in Security > Application Security > Policy > Response Page

      Default

      Login Page

      ResponseXMLSOAP

      Cookie Hijacking

       

      so which one related to web scraping?

      i don't want to customize bad request page that related to any violation , i need only that one which is related to web scrapping.

       

      thanks

  • Erik_Novak's avatar
    Erik_Novak
    Icon for Employee rankEmployee
    Jul 20, 2020

    I see. You will need to write an iRule for that. The iRule below should work--but you will need to find the exact name of the web scraping violation (I don't know it off the top of my head) and then add it to the iRule. Then make sure you enable iRule processing on the security policy.

     

    Something like this:

    when ASM_REQUEST_BLOCKING {

        if { [lindex [ASM::violation_data] 0] contains "WEB_SCRAPING_DETECTED" } {

        log local3. "WEB_SCRAPING_DETECTED"

        set response "<html><head><title>Your Page Title Here</title></head><body>Your message text here.</body></html>"

        ASM::payload replace 0 [ASM::payload length] ""

        ASM::payload replace 0 0 $response

        set asm_payload_length [ASM::payload length]

      }

    }

    when HTTP_RESPONSE_RELEASE {

      if { [info exists asm_payload_length] }{

        if {$asm_payload_length > 0} {

          HTTP::header remove Content-Length

          HTTP::header insert Content-Length $asm_payload_length

        }