Large classes in iRules

Hi Sam,

 

 

There was a post (Click here) from a user on 9.2.2 that noticed TMM restarted when merging in a class with ~200k lines.

 

 

Since then, there have been some improvements to external data group/class handling:

 

 

CR66389 - 9.4 external class handling improvements

 

 

Large class redefinition and the extremedb process (CR56743) (9.4.1)

 

In this version, if you create a data group with many members, and then later redefine the contents of the data group, you no longer receive an invalid cursor error. Instead, the redefinition process completes successfully.

 

 

b load with external IP class files (CR73983) (9.3.0)

 

Configurations with external IP class files no longer cause the system to reload the configuration file with each b load command if the file has not changed. Now, the system only reloads the configuration file when the file has changed.

 

 

I can't imagine it would be speedy to use findclass against a set of million(s) of records though. Have you tried testing it?

 

 

Aaron

The only two ways that I'd recommend trying to store/access sets of data that large would be either via the session table, or a TCL arrray, in that order.

 

 

The session table was, after all, specifically designed to hold huge numbers of strings with connection data. Not to mention it gives you the ability to time out entries so you don't have to go through and tear down the data later as you would in an array.

 

 

Classes (data groups) are great for data sets of hundreds or even thousands of entries, but with something this large I don't think they'll cut it.

 

 

Colin

Colin - would storing ~1MB or more in the session bog down the LTM at all? Would he need to "add" each record to the session every time the rule inits? Can you please provide some sample code to illustrate your suggestions?

oh, and what timeout value would you use if you wanted it to be permanent (as in Sam's case, for configuration settings) - would you just leave the timeout value blank?

My previous post was directed specifically at the issue of storing large quantities of data in an iRule and having it be accessible and responsive while processing.

 

 

As far as storing that information permanently and having the records there each time the rule inits, this is a different issue. Unfortunately there isn't currently a good way to achieve both, as the data structures that allow for the size and access speed necessary to handle the number of records we're talking about (arrays and the session table) don't allow for storing the data they contain permanently like classes do. So yes, you'd have to add each entry to either the array or the session table each time the iRule was loaded, which might not be viable for your situation, but isn't a problem for things like, say, connection limiting.

 

 

Colin

Thanks guys.

 

 

I am definitely after a solution which is persistent between iterations of the irule, and even potentially across reboots. Furthermore, the external classes will all need to be updated on the fly as new uri/pool mappings are generated.

 

 

I plan to look a bit further into the possibility of splitting the data into multiple external class files, getting them down to a more manageable size, and doing a hash of the uri submitted to pick which class to query.

 

 

Hopefully class updates are possible via iControl.

 

 

Will test and report back!

 

 

Sam

 

 

PS: Alternatively, is there any way to query a list stored off-box; such as querying an external database via HTTP or some such method?

You could mount a remote directory and provide the symlink as a means of loading the external class, but I think the safer solution is updating the external classes stored locally via iControl (yes, this can be done!)