Disable web scraping detection per geolocation?

Ok, i've been playing around with this and i've got the initial workings of an irule to potentially do this, but there are several things that dont work...what i've done is this:

 

 

1) Disabled the web scraping part of the policy on our default ASM policy/class, but still have it looking for attack signatures.

 

2) created a new policy / class with ONLY web scraping detection enabled, everything else it does not look for.

 

 

 

I've then applied both classes to a VIP, both classes are default apart from "security enabled". This is my first stumbling block - only the top class seems to fire. If you get through the policy for the class that is on top then it lets you through without going down to the second class.

 

 

 

 

 

Is this how it should behave, and am i going down the wrong track?

 

 

 

Thanks

 

...

Hi James,

 

 

Only one ASM policy will be used by default based on which HTTP class matches the request. If you want to disable web scraping protection for specific geo-locales, you could create two copies of the policy with web scraping disabled on one. Then using an iRule with whereis to check the client IP, you could call HTTP::class to select the correct class and ASM policy.

 

 

I see a lot more requests these days for making GeoIP based decisions. You could open a request for enhancement to add logic to the HTTP class filters to match a class based on GeoIP data. You could also ask for a new feature to disable ASM policy enforcement of specific validations by client IP or GeoIP data. If you do this, could you reply back with what you hear from Support?

 

 

Aaron

James,

 

What are you doing to sort traffic between the two classes? When you have more than one class attached to a VS you have to have something defined in the class to force traffic to one or the other. If you look as the class when you set it up, you will see that in the GUI you have options to be able to sort based on Host, URI, Header, or Cookie. Then it will process top down so you will need most specific at that top. You can also do sorting between classes with an iRule, in that case I think you just attach the iRule to the VS as a resource and you don't actually need to attach the class, as your sort should happen in the iRule. If you are not putting any criteria in the class or you have not attached an iRule to the VS then yes all your traffic will hit the 1st class listed and process down that direction.

So from an iRule perspective, I think something as simple this should work

 

 

when HTTP_REQUEST {

 

if {[whereis [IP::client_addr]] contains "US"} {

 

HTTP::class select webscraping_disable_class

 

} else {

 

HTTP::class select webscraping_enable_class

 

}

 

 

}

 

 

 

If you are just looking to white list one country and send everyone else to the other class. If you have list of countries you want to white list or black list though, you will want to create a data list first perhaps named Country_List and then reference that list like below.

 

 

when HTTP_REQUEST {

 

if {[whereis [IP::client_addr]] contains $::Country_List} {

 

HTTP::class select webscraping_disable_class

 

} else {

 

HTTP::class select webscraping_enable_class

 

}

 

 

}

 

 

Hope this helps.

 

 

Mike

Mike's suggestion was along the lines of what I was thinking. If you use a datagroup make sure to remove the $:: prefix from the datagroup name in the iRule.

 

 

If you want to use a single ASM policy, I think you could put Web Scraping in transparent mode and manually send a block message using HTTP::respond in the ASM_REQUEST_VIOLATION event:

 

http://devcentral.f5.com/wiki/iRules.ASM_REQUEST_VIOLATION.ashx

 

 

You'd want to check ASM::violation_data to see if web scraping was detected:

 

http://devcentral.f5.com/wiki/iRules.ASM__violation_data.ashx

 

 

Aaron

Hoolio and Mike, thank you very much for the responses. Firstly, can I ask what the best way is to request an enhancement, should i raise a websupport case?

 

 

Regarding the HTTP classes, is it the case that only one http class can fire? As i was attempting to have two classes, both with identical configuration (that is, both have the defaults apart from asm enabled. Neither is more specific than the other). In this scenario, it was always the http class at the top that fired, the one below it never seemed to fire, although i would have expected the asm to work down them in order, only the top one fired, and if the request passed through that class, it was then passed to ltm without going to the second class.

 

 

 

If I am to have two separate classes, how can i have two policies and keep them in sync, apart from having web scraping enabled or disabled. Is this possible? We still want to look for various attack signatures even from trusted countries, we just want to disable the web scraping part. If we are to have two separate policies, then this means twice the admin work to keep the policies up to date, which won't be possible for us as a team. We really need just one policy to administer, but to disable web scraping for trusted countries. It's a tough one!

James,

 

 

Yes the best way to request an enhancement is to open a support case and work through that chain.

 

 

Basically the way you are seeing traffic behave is correct, when you have two classes attached to a VS you have to have something defined in them to pull traffic towards one or the other. If they are both configured the same then traffic will always fire to the first class process through that policy and exit back to ltm. If I had to compare it to something it would be firewall rules once you match to class you are done it will not check lower classes in the list.

 

 

Aaron had made a creative suggestion in his last post about how to do just one policy by doing an http:respond in an ASM_REQUEST_VIOLATION event. Basically what he is saying is to set Web Scraping to Learn, Alarm but not Block, then build your iRule to fire on ASM::violation_data for VIOLATION_WEB_SCRAPING_DETECTED. I think something like this would work, you may want to re order the flow based upon whether you want to whitelist or blacklist countries, but I believe this will work. The only thing I am not sure about is exiting directly to the pool if you match the whitelist or do not match the blacklist, as I am not sure if all of the policy checking will have completed when this event fires, so please test to make sure you are not bypassing other checks in the policy that you want done.

 

 

 

when ASM_REQUEST_VIOLATION

 

{

 

set x [ASM::violation_data]

 

 

for {set i 0} { $i < 7 } {incr i} {

 

switch $i {

 

0 { log local0. "violation=[lindex $x $i]" }

 

1 { log local0. "support_id=[lindex $x $i]" }

 

2 { log local0. "web_application=[lindex $x $i]" }

 

3 { log local0. "severity=[lindex $x $i]" }

 

4 { log local0. "source_ip=[lindex $x $i]" }

 

5 { log local0. "attack_type=[lindex $x $i]" }

 

6 { log local0. "request_status=[lindex $x $i]" }

 

 

}}

 

 

if {([lindex $x 0] contains "VIOLATION_WEB_SCRAPING_DETECTED")

 

and ([whereis [IP::client_addr]] contains "US")

 

}

 

 

pool pool1

 

 

else {

 

HTTP::respond 200 content {

 

 

 

Blocking Page

 

 

 

Your Requested has been blocked please contact your systems adminstrator

 

Your support ID is: <%TS.request.ID()%>

 

 

 

}

 

}

 

 

}

 

 

}

 

So the forum strips the HTTP tags in the post, so you will need to do that formatting to make this work

So the forum strips the HTTP tags in the post, so you will need to do that formatting to make this work. You can check out the link below to see the formatting, if you don't know how to do it.

 

 

http://devcentral.f5.com/wiki/iRules.HTTP__respond.ashx