How to handle data aggregators hitting my site?

[academictame]

I'm looking for a more intelligent way to handle to prevent data aggregators from crushing my site. The way I do this today is with a policy in the LTM section of the BigIP (it's the only license we have) that matches a list of IPs in a datagroup, and redirects them to a dedicated VIP with a backend pool with just 1 server there that the aggregators crush. I want to get rid of this dedicated aggregator server...

Is there a way to specificly limit only the aggregator services from crushing my primary 2 servers without offloading them to and maintaining a dedicated server?

[Mohamed_Salah_]

Hello,

If these servers' IP are constant, you can add rules to reject any traffic from them instead of redirecting them.

You can use firewall rules in network tab to match on specific conditions and drop this traffic:

https://techdocs.f5.com/kb/en-us/products/big-ip-afm/manuals/product/network-firewall-policies-imple...

Thansk,

Mohamed Salah

[academictame]

unfortunatly rejecting traffic is not an option because aggregators serve a worthwhile purpose for users. 

[Amine_Kadimi]

First thing which is not directly related to the question: while it is possible using a local traffic policy to redirect to another VS internally, you can instead use that same policy to directly select the intended pool, and getting rid of the second VS.

Regarding your question, I see a few ways you can use to rate limit these source IPs:

1. Having a second VS identical to your primary VS except for the Source Address field where you specify an address list (shared object) containing the source IPs to limit. Then in the Advanced configuration, use Connection Limit, Connection Rate Limit and/or a Bandwidth Controller (or Acceleration Rate Class) to rate limit by number of opened/new connections, or by allocated bandwidth.
2. Use one VS for all the traffic, with a policy that selects a pool (e.g AGG_pool) if traffic is coming from the aggregators' source IPs and with a default rule that points to your default pool. In AGG_pool use the Connection Limit and Connection Rate Limit attributes of the members, which will allow you to control how many open/new connections the F5 can have with the pool member.

I think using a Bandwidth Controller or a Rate Class is less aggressive than specifying connection limits but all depends on the specific needs.

[LiefZimmerman]

@academictame - I recognize this is an older question but I wonder if you can identify (and select) Accept As Solution any replies that you feel resolved your issue? It's helpful to the community to surface successful solutions.

Thanks for being part of our community.