For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

OCC_68606's avatar
OCC_68606
Icon for Nimbostratus rankNimbostratus
Jan 14, 2014

Search Engines blocked with Web Scraping Option

Hi, I have the ASM version 11.2.1 and Webscraping option active, I have problem with search engines crawling process, example: googlebot, bing, etc. I want to know how can I allow or whitelist the fr...
Marvin's avatar
Marvin
Icon for Cirrocumulus rankCirrocumulus
Apr 20, 2015

The Application Security Manager does not perform web scraping detection on traffic from search engines that the system recognizes as being legitimate. If your web application has its own search engine, we recommend you add it to the system. Go to Options > Application Security > Advanced Configuration > Search Engines and add it to the list.

 

The default search engines included are Ask, Bing, Google and Yahoo