Forum Discussion
Mike_Maher
Dec 16, 2011Nimbostratus
James,
You are right there does not appear to much in the way of configuration for Search Engines, although I am not sure there has to be. If you have your site mapping done correctly and are using the right Attack Signatures, then the crawlers/bots from the legitimate search engines should get through and the Attack Signatures should take care of blocking malicious crawlers/bots. If you want to be double sure you are not catching one of the UA strings that needs to be allowed, you can port the logs off to a syslog server and then perhaps have a script that looks for the various UA strings.
Really once you setup proper site mapping in your policy any legitimate traffic should make it to your web site just fine.
Mike