For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

Dave_Burnett_20's avatar
Dave_Burnett_20
Icon for Nimbostratus rankNimbostratus
Nov 10, 2008

How to allow Search Engine Robots/Slurps through ASM?

We have recently installed a pair of F56400s (v9.4.3) in front of our website with ASM in blocking mode.     We are seeing and blocking loads of Non-RFC compliant request violations. Exami...
app sec
BIG-IP Access Policy Manager (APM)
security
dburnett_103851's avatar
dburnett_103851
Icon for Nimbostratus rankNimbostratus
Jan 15, 2009

Posted By abrailsford on 11/11/2008 6:25 AM

 

 

That helps a great deal;

 

 

I passed one of those requests through my lab unit and my v9.4.5 ASM is triggering a violation on "Header name with no header value"

 

 

While that isn't actually an HTTP RFC violation, since both the 1.0 and 1.1 RFCs list the header value as optional, it is a configurable blocking setting which is enabled by default for newly created policies.

 

 

Unfortunately I can't remember if that was an easily configurable option in v9.4.3, but in v9.4.5 (and v9.4.4 IIRC) it certainly is, under Policy->Blocking->HTTP Protocol Compliance where you can simply uncheck that blocking option for your policy.

 

 

 

We've upgraded to v9.4.5 and I can see where we can allow the Yahoo slurps through by turning off the Header Name with No Header Value. However, what are the implications of turning this particular feature off? Are we potentially opening ourselves and making our site vulnerable? Is there an alternative way to allow the Yahoo robots through which is less risk?