Forum Discussion
JamesS_40157
Dec 16, 2011Nimbostratus
ASM search engine configuration query
Hi all,
We are currently in the process of setting up the ASM for production use, and unfortunately I'm finding the documentation on search engine configuration quite lacking.
Search engine crawlers / bots are very important to our site, but what is also important for us is to block malicious data scrapers. We have a list of various bots/crawlers we want to allow (by user agent). However, i'm struggling to understand how to translate this into a search engine list on the ASM.
I understand the "domain" part of the configuration is to do with reverse lookups on the IP addresses of suspected crawlers, however what happens if a particular crawler doesn't have reverse DNS configured?
Also, what should the "name" part of the search engine relate to - is this a string match against the user-agent they provide? And should both the user-agent and domain name match, or is just one or the other enough to suffice?
Thanks in advance
James
- Mike_MaherNimbostratusJames,
- JamesS_40157NimbostratusThanks for the response Mike.
- Mike_MaherNimbostratusWell actually depending on what version of ASM you are running you may want to disable Web Scraping detection. There is an XSS vulnerability within that feature in 10.1.0 - 10.2.2.
- JamesS_40157NimbostratusThanks Mike - i agree just looking for a UA string might be risky as it is easily spoofed, we may have to push back on our SEO team in this case if there is no easy way to define this. I'll try raising a formal information request with F5 too to see exactly what those search engine fields mean.
Recent Discussions
Related Content
DevCentral Quicklinks
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
Discover DevCentral Connects