Forum Discussion
OCC_68606
Nimbostratus
NimbostratusSearch Engines blocked with Web Scraping Option
Hi, I have the ASM version 11.2.1 and Webscraping option active, I have problem with search engines crawling process, example: googlebot, bing, etc. I want to know how can I allow or whitelist the fr...
Jorge, the ASM comes pre-loaded with whitelisted bots:
- Google (googlebot.com)
- Yahoo (crawl.yahoo.net)
- MSN (search.msn.com)
- Ask.com (ask.com)
- AOL (using googlebot.com)
Here's an AskF5 article that discusses the whitelisted bots: http://support.f5.com/kb/en-us/products/big-ip_asm/manuals/product/config_guide_asm_10_2_0/asm_dos_bruteforce.html1043772
