I'm looking for a more intelligent way to handle to prevent data aggregators from crushing my site. The way I do this today is with a policy in the LTM section of the BigIP (it's the only license we have) that matches a list of IPs in a datagroup, and redirects them to a dedicated VIP with a backend pool with just 1 server there that the aggregators crush. I want to get rid of this dedicated aggregator server...
Is there a way to specificly limit only the aggregator services from crushing my primary 2 servers without offloading them to and maintaining a dedicated server?
If these servers' IP are constant, you can add rules to reject any traffic from them instead of redirecting them.
You can use firewall rules in network tab to match on specific conditions and drop this traffic:
First thing which is not directly related to the question: while it is possible using a local traffic policy to redirect to another VS internally, you can instead use that same policy to directly select the intended pool, and getting rid of the second VS.
Regarding your question, I see a few ways you can use to rate limit these source IPs:
I think using a Bandwidth Controller or a Rate Class is less aggressive than specifying connection limits but all depends on the specific needs.