Search Engines blocked with Web Scraping Option

Jorge, the ASM comes pre-loaded with whitelisted bots:

 

• Google (googlebot.com)
• Yahoo (crawl.yahoo.net)
• MSN (search.msn.com)
• Ask.com (ask.com)
• AOL (using googlebot.com)

Here's an AskF5 article that discusses the whitelisted bots: http://support.f5.com/kb/en-us/products/big-ip_asm/manuals/product/config_guide_asm_10_2_0/asm_dos_bruteforce.html1043772

 

By the way, here are some articles on web scraping...they might provide a little more insight as well:

 

Bot Detection

 

https://devcentral.f5.com/articles/more-web-scraping-bot-detection.UtW1dLHnbIU

 

Session Anomalies

 

https://devcentral.f5.com/articles/these-are-not-the-scrapes-youre-looking-for-session-anomalies.UtW10bHnbIU

 

The Application Security Manager does not perform web scraping detection on traffic from search engines that the system recognizes as being legitimate. If your web application has its own search engine, we recommend you add it to the system. Go to Options > Application Security > Advanced Configuration > Search Engines and add it to the list.

 

The default search engines included are Ask, Bing, Google and Yahoo