JamesS_40157
Dec 16, 2011Nimbostratus
ASM search engine configuration query
Hi all,
We are currently in the process of setting up the ASM for production use, and unfortunately I'm finding the documentation on search engine configuration quite lacking.
Search engine crawlers / bots are very important to our site, but what is also important for us is to block malicious data scrapers. We have a list of various bots/crawlers we want to allow (by user agent). However, i'm struggling to understand how to translate this into a search engine list on the ASM.
I understand the "domain" part of the configuration is to do with reverse lookups on the IP addresses of suspected crawlers, however what happens if a particular crawler doesn't have reverse DNS configured?
Also, what should the "name" part of the search engine relate to - is this a string match against the user-agent they provide? And should both the user-agent and domain name match, or is just one or the other enough to suffice?
Thanks in advance
James