Log unique Client Addresses per (Hour|Day|Etc)

If you're on v10, you can use a sub table (The table command in iRules) to track IP addresses... (Use the IP address as the key. The value can be a count). Then after 60 minutes (Use the after {} command) you can send (HSL in v10, or a sideband connection in v11) the counts to a remote host.

 

 

If you look at my LDAP stats iRule in codeshare (http://devcentral.f5.com/wiki/iRules.LDAP_Measure_Reporting.ashx) , most of what you want is in there... (The after {} command, HSL, tables etc). Including how to keep multiple value data per key (e.g. Make a composite key).

 

 

H

We are on v10. Is tracking this data in a table a good idea? I've heard some negative comments about memory consumption.

If this is just for reporting, I'd consider using High Speed Logging to send a log message to a remote pool of syslog servers for every request and/or response. The CPU/memory/I/O overhead for HSL is quite low compared with using a subtable and logging locally. In informal testing on an 8900 I saw up to 200k TCP or UDP messages of 1000 bytes sent with ~10% CPU increase and nearly no additional memory usage. The additional CPU was much lower at lower message levels.

 

 

As Hamish pointed out in a recent thread here, it would be good to upgrade to the latest 10.2.x release when using HSL as there have been some bugs fixed in this version.

 

 

Aaron

 

Aaron

Thanks Aaron. How does logging through HSL alleviate my need though for a table? Or maybe it doesn't and you are only talking about the actual log function?

I'm thinking it's better in terms of LTM resource usage to log everything externally using HSL and parse the stats you want off of LTM.

 

 

Aaron

I originally started to do that with the LDAP work I did in the iRUle referenced above. Unfortunately it doesn't scale very well, and isn't very reliable.

 

 

Using UDP, is fire & forget. If the receiver can't keep up, you lose messages (This was bad for me. May be OK for your requirements though).

 

 

Using TCP is a stream of data (No loss). But the buffers are tiny. When they fill up (i.e. receiver can't keep up), the LTM opens a NEW connection to the logging pool... Then when the pool is busy and can't keep up, you get an endless spiral where the number of HSL connections quickly brings everything to it's knees (I had it up to thousands of connections :). WHich rapidly gets messy as it can take a LONG time to process all those streams. WHich uses up all that memory you're trying to avoid in not using the tables in the first place.

 

 

Oh... HSL also delivers messages out of order... With both UDP and TCP. Again, your requirements may not care, but mine did when trying to perform stats calculations from individual messages.

 

 

At least with tables you can remain in control... (e.g. Flush when necessary to free up memory etc, as well as keeping HSL within useful bounds). Whereas with HSL you have none.

 

 

H

Using TCP is a stream of data (No loss). But the buffers are tiny. When they fill up (i.e. receiver can't keep up), the LTM opens a NEW connection to the logging pool... Then when the pool is busy and can't keep up, you get an endless spiral where the number of HSL connections quickly brings everything to it's knees (I had it up to thousands of connections :). WHich rapidly gets messy as it can take a LONG time to process all those streams. WHich uses up all that memory you're trying to avoid in not using the tables in the first place.

 

 

Hey Hamish,

 

 

What kind of syslog servers were you using? If you increased the pool of servers, did you see better results?

 

 

I tested using an set of LTMs to act as a syslog pool. I saw less than 1% loss of messages up to 200k messages/second as long as I had a big enough trunk and pool of servers available to receive the syslog messages. Granted logging 1k messages x 200k messages / second to disk would require a beefy collection of syslog servers.

 

 

Aaron