Intermediate iRules: Data-Groups

For this article in the Intermediate iRules series we’ll begin arming you with some knowledge on data-groups. As such, this article will endeavor to answer the following:

What is a data-group?
What is a class?
What are the benefits of a data-group?
What command(s) would I use to access data in a data-group?

This will be a relatively high-level overview that walks through the basics of these constructs. There is much more to dig into out on DevCentral, but here’s a start:

What is a data-group?

A data-group is a particular type of memory structure within iRules. It is effectively a list in a key -> value pair format consisting of IP addresses/subnets, strings, or integers. You can have just a list of keys, or a list of matched keys and values in this unique structure. It’s unique and special in a couple of ways, but most obvious among them is that it is actually stored permanently as part of the configuration, rather than only existing as a part of an iRule, or in memory. This can be done either inline in the big-ip.conf, or as a separate file (known as an external data-group). Both function effectively the same for the purposes of our discussion here. Because it is stored on box and not solely in memory, data-groups are pre-populated with data before an iRule ever executes. This makes them ideal for stored lists of any sort, such as authorized IP ranges or URI redirection mappings and the like. This also means they are mirrored across all boxes in a cluster (or both in a pair) because they are effectively a config object. Modifying a data-group is simple as well, thanks to direct access via the CLI or GUI. You can add, modify or remove entries in a data-group without ever touching the iRule(s) referencing it, which makes it ideal for storing config bits or static info that may need to be updated from time to time. As long as the formatting of the data remains correct while you’re editing, there’s no real chance of breaking your iRule by modifying the data-group, since the code itself isn’t being touched. To that end, there are two ways to store data-groups, internal or external. For internal data-groups, the data set is stored in the bigip.conf file. For external data-groups, they are maintained in their own file and referenced from the data-group object. Very large data sets should be kept in external data-groups. The only (possibly) limiting factor about data-groups being a config object is that iRules can’t affect configuration objects directly. This means that while you can read, sort, and reference data-groups from within your iRule, you can’t actually modify them directly. If you want to update a data-group it has to be either done manually via the CLI/GUI, or scripted via TMSH or iControl. There is no direct iRules access for modifying the contents, making these a read-only data structure from iRules’ perspective. The config object would look like this (v11+, see the class command in the wiki for earlier versions):

ltm data-group internal name_value_dg {
    records {
        name1 {
            data value1
        }
        name2 {
            data "value2 with spaces"
        }
    }
    type string
}

What is a class?

A class is exactly the same thing as a data-group. We have used the terms interchangeably for years, much to the chagrin and confusion of some users. We even called the command class while the structure is called “data-group”. Don’t let that confuse you, they are the same thing, and regardless of which you hear someone mention, they’re talking about the memory structure that I just described above.

What are the benefits of a data-group?

I mentioned before that one of the only drawbacks of data-groups is that they are read-only for all intents and purposes where iRules is concerned.That, however, is a tiny drawback in most cases when you consider the performance of data-groups. data-groups are far and away the most efficient memory structure with which to perform lookups past only a few entries. if/else and switch are fine to a point, but past about 10 items even the more efficient switch can’t keep up with the linear scaling of data-group lookups. Whether you’re storing 100 or 100,000 entries, queries are roughly the same thanks to the indexed, hashed format of data groups. This makes them far and away the best option for storing large lists of data as well as frequently performed queries that can be represented in a read-only fashion. The bottom line is: If you’re not updating the data in the list, and you’re dealing with more than a handful of items, be it strings, IPs or otherwise, data-groups are likely your best bet. They’re also resilient through failover, reboots, etc. unlike data structures without a disk based config object to back them up. Last, but in some cases not least at all, you can script the management of data-groups, especially external data-groups, via iControl. TMSH, etc. This makes it an ideal way to bulk load data or manage entries in the data structure externally. Just remember that you’ll have to re-instantiate the iRule for the changes to be recognized.

What command(s) would I use to access data in a data-group?

The command you’ll be using to access data-groups is the class command. In versions prior to 10 there were other commands (matchclass / findclass,) but as of v10 data-groups were completely overhauled to be more effective and higher performance, and the class command was born. It’s extremely powerful and has many permutations. I won’t attempt to go into all of them here, but you can see some basic matching examples here. When you combine the flexibility of the class command (full documentation here) with the performance and scalability of data-groups, you get some pretty powerful possibilities. Here are a couple examples from the wiki:

when HTTP_REQUEST {
    if { [class match [IP::client_addr] equals "localusers_dg" ] } {
        COMPRESS::disable
    }
}

when HTTP_REQUEST {
    set app_pool [class match -value -- [HTTP::uri] starts_with app_class]
    if {$app_pool ne ""} {
        pool $app_pool
    } else {
        pool default_pool
    }
}