#The101: Datagroups and Tables

[ltwagnon]

Well we’ve made it to the end. This is the 10^th and final installment of #The101: iRules. So far we’ve talked about everything ranging from the very basics of programming concepts to some more particular iRules type concepts such as events and string manipulation. Hopefully you’ve been following along and garnered some good bits of information along the way. For those that may just be catching the series, or those that could use a refresher, I’ll list out the articles in the series one last time. I highly recommend at least thumbing through the previous installments if you haven’t already. I’ll go through them one final time here, in order:

     1. Introduction to Programming & Tcl

     2. Introduction to F5 Technology & Terms

     3. Introduction to iRules

     4. Events & Priorities

     5. Control Structures & Operators

     6. Variables

     7. Logging & Comments

     8. Routing

     9. String Handling and Manipulation

For the finale of this introductory series we’ll dig into a couple of topics that are borderline at best when it comes to being fodder for beginners. They are relatively advanced concepts, certainly the most in-depth of the series, but they’re also near ubiquitous in the iRules world. Given the power and breadth, I figure it’s only fair to begin arming you with some knowledge on datagroups and tables. As such, this article will endeavor to answer the following:

     - What is a datagroup?

     - What is a class?

     - What are the benefits of a datagroup?

     - What command(s) would I use to access data in a datagroup?

     - What is a table?

    - What are the benefits of a table?

    - What command(s) would I use to access data in a table?

This will be a relatively high-level overview that walks through the basics of these constructs. There is much more to dig into out on DevCentral, but here’s a start:

What is a datagroup?

A datagroup is a particular type of memory structure within iRules. It is effectively a list in a key -> value pair format. You can have just a list of keys, or a list of matched keys and values in this unique structure. It’s unique and special in a couple of ways, but most obvious among them is that it is actually stored permanently as part of the configuration, rather than only existing as a part of an iRule, or in memory. This can be done either inline in the big-ip.conf, or as a separate file (known as an external datagroup). Both function effectively the same for the purposes of our discussion here.

Because it is stored on box and not solely in memory, datagroups are pre-populated with data before an iRule ever executes. This makes them ideal for stored lists of any sort, such as authorized IP ranges or URI redirection mappings and the like. This also means they are mirrored across all boxes in a cluster (or both in a pair) because they are effectively a config object. Modifying a datagroup is simple as well, thanks to direct access via the CLI or GUI. You can add, modify or remove entries in a datagroup without ever touching the iRule(s) referencing it, which makes it ideal for storing config bits or static info that may need to be updated from time to time. As long as the formatting of the data remains correct while you’re editing, there’s no real chance of breaking your iRule by modifying the datagroup, since the code itself isn’t being touched.

The only (possibly) limiting factor about datagroups being a config object is that iRules can’t affect configuration objects directly. This means that while you can read, sort, and reference datagroups from within your iRule, you can’t actually modify them directly. If you want to update a datagroup it has to be either done manually via the CLI/GUI, or scripted via TMSH or iControl. There is no direct iRules access for modifying the contents, making these a read-only data structure from iRules’ perspective.

The config object would look like this:

   1: /config/app_class.dat:

   2: "/trxdef/" := "trx_pool",

   3: "/aaa/" := "aaa_pool",

   4: "/abscon/" := "abs_pool"

What is a class?

A class is exactly the same thing as a datagroup. We have used the terms interchangeably for years, much to the chagrin and confusion of some users. We even called the command “class” while the structure is called “datagroup”. Don’t let that confuse you, they are the same thing, and regardless of which you hear someone mention, they’re talking about the memory structure that I just described above.

What are the benefits of a datagroup?

I mentioned before that one of the only drawbacks of datagroups is that they are read-only for all intents and purposes where iRules is concerned.That, however, is a tiny drawback in most cases when you consider the performance of datagroups. Datagroups are far and away the most efficient memory structure with which to perform lookups past only a few entries. if/else and switch are fine to a point, but past about 10 items even the more efficient switch can’t keep up with the linear scaling of datagroup lookups.

Whether you’re storing 100 or 100,000 entries, queries are roughly the same thanks to the indexed, hashed format of data groups. This makes them far and away the best option for storing large lists of data as well as frequently performed queries that can be represented in a read-only fashion. The bottom line is: If you’re not updating the data in the list, and you’re dealing with more than a handful of items, be it strings, IPs or otherwise, datagroups are likely your best bet.

They’re also resilient through failover, reboots, etc. unlike data structures without a disk based config object to back them up. Last, but in some cases not least at all, you can script the management of datagroups, especially external datagroups, via iControl. TMSH, etc. This makes it an ideal way to bulk load data or manage entries in the data structure externally. Just remember that you’ll have to re-instantiate the iRule for the changes to be recognized.

What command(s) would I use to access data in a datagroup?

The command you’ll be using to access datagroups is the “class” command. In versions prior to 10 there were other commands, but as of v10 datagroups were completely overhauled to be more effective and higher performance, and the class command was born. It’s extremely powerful and has many permutations. I won’t attempt to go into all of them here, but you can see some basic matching examples here.

When you combine the flexibility of the class command (full documentation here) with the performance and scalability of datagroups, you get some pretty powerful possibilities. Here are a couple examples from the wiki:

   1: when HTTP_REQUEST {

   2:   if { [class match [IP::client_addr] equals "localusers_dg" ] } {

   3:     COMPRESS::disable

   4:   }

   5: }

   6:  

   7:  

   8: # Not an iRule reference - showing an actual datagroup object

   9: datagroup app_class {

  10:   type string

  11:   filename app_class.dat

  12: }

   1: when HTTP_REQUEST {

   2:   set app_pool [class match -value -- [HTTP::uri] starts_with app_class]

   3:   if {$app_pool ne ""} {

   4:     pool $app_pool

   5:   } else {

   6:     pool default_pool

   7:   }

   8: }

What is a table?

A table is somewhat similar to a datagroup in that it is, at its core, a memory structure to contain lists. That’s about where the similarities end, though. Tables are stored in memory alone, which means they don’t have the config object backing that a datagroup does, but that also makes them inherently more flexible.

A table is a read/write memory structure that is maintained across connections, making it a solid contender for storing data that needs to be persistent, as opposed to connection based like normal iRules variables. Tables also allow a few extremely powerful concepts, namely sub tables and a built in timeout/lifetime structure. Both of these things add possibilities to what tables are capable of.

Tables are also extremely efficient, much like datagroups, though on a slightly smaller scale given their more flexible nature. Tables make use of the session table within LTM, which is designed to handle every connection into or out of the device, and as such is very high performance. Table queries share this performance and can be a quick way to construct simple, persistent memory structures or more complex near DB like sets of sub-tables, depending on your needs.

What are the benefits of a table?

Tables are a high performance, highly scalable, persistent (non-connection bound), read/write data structure. That fact alone makes them unique within iRules and extremely powerful. There simply aren’t other things that fill that role. There are only a couple of ways to have read/write variable data across connections, and tables are by far the best option in almost every case.

The ability to create what amounts to a mini database in memory from within your iRule is massively useful in many scenarios also. This is easy to do via subtables. You can create not just a flat list, but named lists to segregate data however you’d like. By doing so you can create relations and a relatively complex schema for data storage and accounting.

Now add in the fact that there are configurable timeout and lifetime options, which means you’ll never have to worry about memory management or programmatic cleanup of a table, as things will time out as designed, and you’ve got another layer of usability and ease of use that is unique to tables.

The bottom line is that tables are one of the more powerful, flexible features to hit iRules in quite a while.

What command(s) would I use to access data in a table?

To access tables, you make use of the table command, much like the class command to access datagroups. This command can also get quite complex quite fast, so I won’t attempt to cover all the possible permutations here. I’ll list a couple of simple examples, and give you a link to the full documentation on DevCentral. It’s also worth noting that there is an excellent article series that delves deep into the table command in particular written by the developer responsible for the command, out on DevCentral here.

   1: # Limit each client IP address to 20 concurrent connections

   2: when CLIENT_ACCEPTED {

   3:   # Check if the subtable has over 20 entries

   4:   if { [table keys -subtable connlimit:[IP::client_addr] -count] >= 20 } {

   5:     reject

   6:   } else {

   7:     # Add the client IP:port to the client IP-specific subtable 

   8:     # with a max lifetime of 180 seconds

   9:     table set -subtable connlimit:[IP::client_addr] [TCP::client_port] "" 180

  10:   }

  11: }

  12:  

  13: when CLIENT_CLOSED {

  14:   # When the client connection is closed, remove the table entry

  15:   table delete -subtable connlimit:[IP::client_addr] [TCP::client_port]

  16: }

As you can see both datagroups and tables are extremely powerful, and can add a couple of powerful tools to your quiver for your future iRule endeavors. There are hundreds if not thousands of examples of using these powerful commands out on DevCentral, so there is no shortage of information to be found and code to be lifted for re-use if you scour the forums and code-share. In an attempt to keep this consumable I've not gone through the full list of command permutations or anywhere near the full possibilities. I'll leave discovering those things to the more advanced and/or eager iRulers, but suffice to say the possibilities are vast.

And that will wrap up #The101: iRules! I truly hope the series has been helpful and informative. Thanks for reading along the way. Feedback to help shape future series such as this would be quite welcome. If you have any input share it in the comments and we'll take it into account when planning the next series for DevCentral.

[Paolo_Tosiani_7]

Hi, great series! thank you for your dedication. Would you consider adding the link to all the 10 articles to the rest of the series..? (that makes the articles much more easy to navigate) Thank you.

[Zuke_149069]

I think it'd be worthwhile to link this series of articles in a meaningful way. I had to search for the entire series under Colin's profile; the only order is by date. The "Read more articles in this series" link at the top of the page returns a 404.

[Ernesto_Mayol_2]

Great Series. I second the request to add the links to the entire series to each of the articles for ease of navigation. Thanks!