For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

Paul_Dawson_208's avatar
Paul_Dawson_208
Icon for Nimbostratus rankNimbostratus
Apr 20, 2006

prevention of web scraping - plausible?

Hello,     Given the flexibility of iRules, would it be possible to prevent web/screen scraping?     Perhaps defining certain criteria/behavior to look for and then applying a rate shapin...
application delivery
dev
devops
iRules
Tom_Spector_50's avatar
Tom_Spector_50
Historic F5 Account
Apr 20, 2006
Hi,

 

 

I would like to point the difference between two kinds of scraping –

 

- Scraping with a malicious intent – such as scanners or scripts intended to detect or attack various part of the site

 

- Scraping by innocent users – such as scripts intended for consolidating and enhancing capabilities of applications (such as scripts that consolidate information from different bank accounts or that monitor sites for specific information.)

 

 

Both kinds have common traffic characteristics with regards to the frequency of the requests per a given source (provided the source can be distinguished).

 

The iRules language is strong enough to be used for implementing such logic with options such as utilizing cookies for source distinguishing and counters of requests per a given time.

 

With regards to protecting applications against the malicious side of this issue, I would recommend using the F5 application security solution.