Forum Discussion
JamesS_40157
Nimbostratus
NimbostratusASM search engine configuration query
Hi all,
We are currently in the process of setting up the ASM for production use, and unfortunately I'm finding the documentation on search engine configuration quite lacking.
Search engine crawlers / bots are very important to our site, but what is also important for us is to block malicious data scrapers. We have a list of various bots/crawlers we want to allow (by user agent). However, i'm struggling to understand how to translate this into a search engine list on the ASM.
I understand the "domain" part of the configuration is to do with reverse lookups on the IP addresses of suspected crawlers, however what happens if a particular crawler doesn't have reverse DNS configured?
Also, what should the "name" part of the search engine relate to - is this a string match against the user-agent they provide? And should both the user-agent and domain name match, or is just one or the other enough to suffice?
Thanks in advance
James
Mike_MaherJames,
You are right there does not appear to much in the way of configuration for Search Engines, although I am not sure there has to be. If you have your site mapping done correctly and are using the right Attack Signatures, then the crawlers/bots from the legitimate search engines should get through and the Attack Signatures should take care of blocking malicious crawlers/bots. If you want to be double sure you are not catching one of the UA strings that needs to be allowed, you can port the logs off to a syslog server and then perhaps have a script that looks for the various UA strings.
Really once you setup proper site mapping in your policy any legitimate traffic should make it to your web site just fine.
Mike
JamesS_40157Thanks for the response Mike.
My concerns are that they would be blocked by the "web scraping" portion of the ASM. I understand the web scraping logic sends javascript and cookie requests, looking to see if the client is moving the mouse, pressing the keyboard etc etc. I would have thought that even good bots would not be able to respond correctly to these queries? And even if they can, i'd imagine it would only be the big players (google, yahoo, bing etc) that would bother implementing such a thing. Smaller engines may not do this...?- Mike_MaherWell actually depending on what version of ASM you are running you may want to disable Web Scraping detection. There is an XSS vulnerability within that feature in 10.1.0 - 10.2.2.
https://support.f5.com/kb/en-us/solutions/public/12000/900/sol12953.html?sr=18293022
If you are running safe code to leave this enabled, another way to work around this might be to design an ASM iRule that matches the UA string against a safe list of UA strings you define as a data list, and then have it bypass the Web Scraping protection based upon that. I am definitely not an iRule guru so you will want to hunt through the Wiki's for more information. On how you might doe that, although you probably want to be careful doing something like that as it would be easy to spoof the UA string. - JamesS_40157Thanks Mike - i agree just looking for a UA string might be risky as it is easily spoofed, we may have to push back on our SEO team in this case if there is no easy way to define this. I'll try raising a formal information request with F5 too to see exactly what those search engine fields mean.
We're running v11.1 too so we shouldn't be vulnerable to the attack that you speak of.
Thanks
James