get_al_active_connections() crashes iControlPortal.cgi

Currently, *most* connection and persistence information can be mirrored between a redundant pair. In the past, I wrote an emulator for shining a little light on the mirroring process. Lately, it has occurred to me that this same sort of thing--connection add and delete events--might be useful for customers for data mining purposes.

 

 

If I were architecting something like this, I'd mirror the traffic to a machine setup for traffic monitoring* and put some logic on *it* to create classes** that I then passed to bigip via iControl. Furthermore, you could even use the existing aol class on BIGIP to help alleviate the monster proxy issue.

 

 

*I assume there are existing products with this capability. If not, this sort of filtering would be uber-trivial (it's easy since the IP header contains addressing information *as well as* the packets total length) to implement on top of Linux' SOCK_PACKET stuff. NB: "easy" depends, I guess, on how much traffic you have since you might have difficulty keeping up. That being said, the amount of processing you'd need to do for classification is tiny. Someone with higher performance requirements might do something less naive and put classification and preliminary aggregation in the kernel and have a corresponding user-space process grab data periodically via an ioctl. This is significantly more work.

 

 

**external or internal. Small benefit: the rule is constant. BIG benefit: classes are well-equipped for large datasets. Lookups are currently O(log n) for internal classes and O(1) [NB: at the expense of memory] for external ones.

Joe,

 

Thanks for helping out. Are the methods to get connection information for single IP's reliable enough to use in production?

 

 

Will

We're trying to sniff out client IP's that have (what we deem to be) excessively high connection and transfer rates (e.g. web spiders). .

 

 

Once these hosts are identified, we wish to be able to persist them all to a single node so one machine will handle their processing requirements w/o affecting "normal" traffic.

 

 

Given your reply, my current thought process is to ssh in, run bigpipe conn dump, parse that to identify potential "suspects" and then use iControl to track individual IP addresses. Problem is, I don't want to have to deal w/ all of the overhead and atomitcity (is that even a word?) of relying on an external shell, not to mention dealing w/ ssh keys and ssh-agent.

 

 

Any other way I can obtain a list of all client IP's and the total number of connections they have?

 

 

Thanks for your help,

 

Will

Absolutely, the get_all type methods that can return an enormous list are the ones that tend to cause problems. It's a classic chicken and egg problem. App developers don't want to make 10000 calls for 10000 objects, they want to make 1. But for large result sets, the 1 method call causes problems. We are working at segmenting these types of operations in new platforms but that brings with it it's own set of problems..

 

 

Again, I've pinged our dev group to see of any alternate solutions for you and if there is any thing on the 4.x platform in the near future that can help.

 

 

We really appreciate your feedback and knowing that you need this functionaility will help us priortize for future development efforts.

 

 

-Joe

This was an example of us providing too much functionality early on. Basically the connection table is too large in a production environment and the memory overhead for building the SOAP response. We've put limits on the memory overhead of the iControl processes to help limit the overall memory usage and it seems that this causes that limit to be reached. If you look at it, let's say there are 10000 active connections, and each connections requires ~1k of XML data for the SOAP response, we are looking at 1k*10000b=10M Bytes of payload. This would more than likely cause the client undo hardship in parsing as well.. The real issue is that on our 4.x platforms we have a CORBA interface that the SOAP interfaces act as a proxy to and this causes another level of memory allocation which isn't quite as large as the SOAP usage, but close.

 

 

We've optimized the way we handle large result sets in the next generation of products but are somewhat limited by the architecture in the 4.x platform. It is our recommendation that you do not use the Connections interface in it's present form on the 4.x series of products.

 

 

What is your need for viewing the connections table? What are the requirements for your application?

 

 

-Joe

Currently there isn't a way on 4.x to achieve this without using the Connections class. We are working on solutions for segmenting the large amount of data but it currently isn't available in 4.x.

 

 

I'll reply if I can find a better solution for you than by ssh'ing in.

 

 

-Joe

Thanks for the reply. I agree that mirroring traffic somewhere else and using another utility to create the connection counts would work best, but the timeline is the major consideration (mgmt. wants this up in the next day or two). If we do eventually go that route, we would want to utilize layer 7 information to do additional filtering (e.g. filter OUT any connections that have "normal" user-agent strings) so we would likely be doing more than just counting connections but I think utilities like snort can provide this information rather natively.

 

 

The difficult part polling this information is in obtaining connection rate information. Since I am only able to see a current snapshot of the connection table and will inevitably have a relatively large window between polls, knowing how many connections per second a client is establishing will be impossible. That being said we'll attempt to use some statistical analysis to derive that metric.

 

 

On a slightly different note, my plan to "persist" suspect clients to a single back end node will be possible if I am able to manually alter the persistence table via iControl. Is this the best way to go about this and is there an easy way to specify which node a client will "stick" to?

 

 

Thanks again. This has been a very enlightening discussion.

 

 

Will

No problem. I just sent of a message to one of the developers asking about the best way to alter individual persistence records. Odd are that you will have to use a Class within a Rule to accomplish this. I'll post back when I have an answer.

 

 

-Joe

[quote:85423441dc="Joe"]No problem. I just sent of a message to one of the developers asking about the best way to alter individual persistence records. Odd are that you will have to use a Class within a Rule to accomplish this. I'll post back when I have an answer.

-Joe[/quote:85423441dc]

Beyond clearing sticky records, manipulating persistence records via a user interface isn't available. As Joe alluded to above, you'd do something like the following instead (untested so I'll undoubtedly get the rule syntax wrong):

if (client_addr one of spider_class) { use (pool spider_pool) } 
 else { use (pool happy_customer_pool) } 
 

It's not persistence per se, but it's essentially the same thing. Furthermore, as I stated above, spider_class should behave well with large N.

NB: I suggest querying bigip for the aol class so you don't accidentally add the AOL mega-proxies into spider_class.

[quote:e72e26c272="gmtiwill"]What Interface / Method should be used for querying the BigIP for AOL mega proxies? I don't see this in the SDK.

Will[/quote:e72e26c272]

It's get_address_class in ITCMLocalLB/Class. I've included a snippet below:

 
 !/usr/bin/env python 
 from F5Proxy import ProxyFactory,loadEnv 
  
 loadEnv('./iControl.env',globals(),locals()) 
 factory = ProxyFactory(user,passwd,host,protocol) 
  
 print factory('ITCMLocalLB/Class').get_address_class(class_name = 'aol')