How to allow Search Engine Robots/Slurps through ASM?

Hi David,

 

 

I did a little digging and I haven't seen anyone else report this problem via the Support channel - that doesn't necessarily mean that nobody else is seeing this, of course, just that nobody else has come to Support with a request about it.

 

 

If you don't want to adjust your non-RFC blocking mask for your policy, which is entirely understandable, then I suspect your only recourse would be to filter these requests with either a Class or an iRule attached to the VIP and then pass them through a separate security policy with a more lenient blocking mask.

 

 

That approach would retain good security and open up the smallest possible attack vector (since, obviously, it is possible for someone to forge the user agent or source IP, depending on your filtering, and pass a malicious request through your more lenient policy).

 

 

That's the best way around this that I can think of, certainly - though I am intrigued as to why Yahoo are sending non-RFC compliant requests. That sounds like the kind of thing that ought to be mentioned to them in parallel with your efforts, since I would consider that "bad manners" on their part.

 

 

 

Let us know if you need assistance in filtering the requests with a Class or iRule and we can work from there.

 

 

 

Thanks,

 

Aaron