Web Scraping detection and referrers
We are fairly new to the BIG-IP world and
we host several sites which are all highly susceptible to web scraping so we
have enable ASM on these sites and it has been working very well for us. One recent issue came up that our Google
organic clicks had dropped by about 40-50%.
I quickly realize that this was due to the fact that our users were
getting the JS challenge page by ASM then it would post that form and let the
user thru without issue but we lose that referrer data. ASM does retain the original referrer in the
form data so to resolve this I created an Irule to fish this out and push it
thru to our web server.
So is there a built in way already to
preserve this or is an Irule currently the only way to go? If not I would like to put a feature request
in for this as it would seem this would be important data to a lot of other
Big-IP users. Also the way to retrieve
this data from an Irule doesn’t appear to be the cleanest as you have to look
for a form value on a post in the payload that ends in “_rf” which could
possibly conflict with other form values if you aren’t careful.
We are running 3900’s with 11.2.0
Build 2451.0 Hotfix HF1
Thanks!