Forum Discussion
Jason_64846
Aug 14, 2012Nimbostratus
Web Scraping detection and referrers
We are fairly new to the BIG-IP world and
we host several sites which are all highly susceptible to web scraping so we
have enable ASM on these sites and it has been working very well for us. One ...
Jason_64846
Aug 15, 2012Nimbostratus
Here's the Irule i put together incase anyone else would need a sample or have any suggestions on improving it. I am checking the Method and the content length twice but I wasn't sure if that HTTP_Request_data event could/would get fired off by something else. It appears to be working for us so far.
when HTTP_REQUEST {if { [HTTP::method] eq "POST"} { Trigger collection for up to 1MB of dataif { [HTTP::header "Content-Length"] ne "" && [HTTP::header "Content-Length"] <= 1048576} { set content_length [HTTP::header "Content-Length"]} else {set content_length 1048576} Check if $content_length is not set to 0if { $content_length > 0} { HTTP::collect $content_length}}}when HTTP_REQUEST_DATA {if { [string tolower [HTTP::method]] eq "post" } { if { [HTTP::header exists "Content-Length"] } {log local0.info [HTTP::payload]foreach p [split [HTTP::payload] &] {set name [getfield $p = 1]if { $name ends_with "_rf" } {set value [getfield $p = 2]set value [URI::decode [getfield $p = 2]]log local0.info $namelog local0.info "Retaining Referrer: $value" HTTP::header replace Referer $value}}} }}
Recent Discussions
Related Content
DevCentral Quicklinks
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
Discover DevCentral Connects