Technical Forum
Ask questions. Discover Answers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to handle data aggregators hitting my site?

academictame
Nimbostratus
Nimbostratus

I'm looking for a more intelligent way to handle to prevent data aggregators from crushing my site. The way I do this today is with a policy in the LTM section of the BigIP (it's the only license we have) that matches a list of IPs in a datagroup, and redirects them to a dedicated VIP with a backend pool with just 1 server there that the aggregators crush. I want to get rid of this dedicated aggregator server...

Is there a way to specificly limit only the aggregator services from crushing my primary 2 servers without offloading them to and maintaining a dedicated server?

3 REPLIES 3

Hello,

If these servers' IP are constant, you can add rules to reject any traffic from them instead of redirecting them.

You can use firewall rules in network tab to match on specific conditions and drop this traffic:

https://techdocs.f5.com/kb/en-us/products/big-ip-afm/manuals/product/network-firewall-policies-imple...

Thansk,

Mohamed Salah

unfortunatly rejecting traffic is not an option because aggregators serve a worthwhile purpose for users. 

First thing which is not directly related to the question: while it is possible using a local traffic policy to redirect to another VS internally, you can instead use that same policy to directly select the intended pool, and getting rid of the second VS.

Regarding your question, I see a few ways you can use to rate limit these source IPs:

  1. Having a second VS identical to your primary VS except for the Source Address field where you specify an address list (shared object) containing the source IPs to limit. Then in the Advanced configuration, use Connection Limit, Connection Rate Limit and/or a Bandwidth Controller (or Acceleration Rate Class) to rate limit by number of opened/new connections, or by allocated bandwidth.
  2. Use one VS for all the traffic, with a policy that selects a pool (e.g AGG_pool) if traffic is coming from the aggregators' source IPs and with a default rule that points to your default pool. In AGG_pool use the Connection Limit and Connection Rate Limit attributes of the members, which will allow you to control how many open/new connections the F5 can have with the pool member.

I think using a Bandwidth Controller or a Rate Class is less aggressive than specifying connection limits but all depends on the specific needs.