Forum Discussion
Optimizing large lookup tables
I have several customers requesting various functionality such as selective rate limiting, access filtering, geo load balancing, etc, all of which would require very large reference tables (4000+ rows).
This post (http://devcentral.f5.com/Default.aspx?tabid=28&view=topic&forumid=5&postid=8241 Click here) seems to indicate that an external class is NOT the ideal way to approach this issue, instead perhaps the data could be imported into an array for more optimal access.
I'm hoping to get some clarification on unRuleY's comment in that post about initializing the array.
Is the recommended approach there to use another high-priority rule specifically written to access the external class and re-write it into an array? I'm assuming we'd need to leverage the RULE_INIT event, but I'm not sure how to parse through a class and re-write all the records to an array.
I'm a bit concerned that initializing the array in this manner might consume excessive system resources each time the config is reloaded.
Bottom line, I know I can create the rule logic to gain the filtering functionality required, but need to know for sure how to manage the large lookup tables and some idea as to real-world max lookup table size.
Any feedback much appreciated.
/deb
2 Replies
- Nope, I was not referring to using the RULE_INIT event to initialize the array. What I was referring to do was to create a seperate VIP/iRule that would be then used to manage the array by some external device/program.
This seperate VIP/iRule would serve the sole purpose of managing the array. So, you would want to come up with a protocol for doing that. Let's say we use HTTP, so the following "PUT " request would actually add the to the array. "GET " could be used to test if was in the array.
You could come up with all sorts of commands to access/manage the array from the external program. The biggest challenge with this approach is having the external program know when it needs to initialize the array. There are a number of ways I can think of to do that, but I'll leave it up to you to discover one. - Thanks unRuleY -
I think I understand, but do you think that approach could be leveraged to import a large static lookup table into an array? Or is that even a good idea? My concerns center around what happens on config re-load or reboot: Is that data then gone until it is added back again? (iRule is intended to deny access unless matching data is in the list, so I don't want to create a false "site down" condition by having the list go away unexpectedly.)
My customer has a list of numeric ranges that will need to be referenced in an array. The data is static, and it is provided monthly to the customer as a table with ~3000 rows, 2 columns each, like this:27049783822704978382
27049783942704978394
27049783962704978396
27049784652704978465
27049785062704978506
32624734863262473486
32624735443262473544
32624735493262473549
32624735683262473568
32624735833262473583
32624735873262473587
32624736643262473664
32624736893262473689
32624737363262473736
32624750803262475080The tools available for class manipulation are not sufficient for this exercise, and when I try to manupulate a class as if it were an array, I get a TCL error 'can't read "::class_name(xxxx)": variable isn't array while executing "array get $::class_name(xxxx)"'
thanks for any insight or alternative approaches
/deb
