Web Scraping Configuration
We would like some clarification on the F5 Web Scraping Application Security that we can't seem to find. Does this block based on session or IP? If a bot is detected during the grace interval, and say we have the unsafe interval set to 100,000, shouldn't it block that IP for 100,000 requests following the detection? We are seeing that once the session is closed it allows that IP back through with another grace interval. The scraping we are receiving is intelligent enough to kill the session once it detects we blocked them, and then opens another session. So, in our event logs we see the same IPs listed multiple times back to back where they were blocked for say 11 requests, then just came right back through with another session. This isn't a desired configuration in our opinion. We were under the impression that if an IP was detected as a bot that it would be blocked for the subsequent unsafe interval we have set. We have tested this from an external connection by sending requests to get detected and blocked by the device, but once we opened a new session we were free and clear again. Is there a setting we need to change for our desired effect? We have looked through the documentation and don't see what would possibly need changed.
Hits from one IP address taking our site down
We are using Big IP ASM version 11.3, and have blocked all geolocations outside of the United States and Canada (we do business only within the US). This past weekend we had enough hard hits from an IP address in Germany, that our site was unavailable for a bit. This was immediately before and immediately after midnight Saturday. We stopped and restarted TomCat, and the issue was resolved. My question is this: Is there something more that we could or should be doing to prevent this type of attack? We are not using DoS, nor web scrapping because even at the lowest settings, we were actually blocking some of our punch-out customers from our eCommcerce site. I welcome any ideas and thoughts about this. Many thanks ~ Dianna