Intermediate iRules: Data-Groups
For this article in the Intermediate iRules series we’ll begin arming you with some knowledge on data-groups. As such, this article will endeavor to answer the following: What is a data-group? What is a class? What are the benefits of a data-group? What command(s) would I use to access data in a data-group? This will be a relatively high-level overview that walks through the basics of these constructs. There is much more to dig into out on DevCentral, but here’s a start: What is a data-group? A data-group is a particular type of memory structure within iRules. It is effectively a list in a key -> value pair format consisting of IP addresses/subnets, strings, or integers. You can have just a list of keys, or a list of matched keys and values in this unique structure. It’s unique and special in a couple of ways, but most obvious among them is that it is actually stored permanently as part of the configuration, rather than only existing as a part of an iRule, or in memory. This can be done either inline in the big-ip.conf, or as a separate file (known as an external data-group). Both function effectively the same for the purposes of our discussion here. Because it is stored on box and not solely in memory, data-groups are pre-populated with data before an iRule ever executes. This makes them ideal for stored lists of any sort, such as authorized IP ranges or URI redirection mappings and the like. This also means they are mirrored across all boxes in a cluster (or both in a pair) because they are effectively a config object. Modifying a data-group is simple as well, thanks to direct access via the CLI or GUI. You can add, modify or remove entries in a data-group without ever touching the iRule(s) referencing it, which makes it ideal for storing config bits or static info that may need to be updated from time to time. As long as the formatting of the data remains correct while you’re editing, there’s no real chance of breaking your iRule by modifying the data-group, since the code itself isn’t being touched. To that end, there are two ways to store data-groups, internal or external. For internal data-groups, the data set is stored in the bigip.conf file. For external data-groups, they are maintained in their own file and referenced from the data-group object. Very large data sets should be kept in external data-groups. The only (possibly) limiting factor about data-groups being a config object is that iRules can’t affect configuration objects directly. This means that while you can read, sort, and reference data-groups from within your iRule, you can’t actually modify them directly. If you want to update a data-group it has to be either done manually via the CLI/GUI, or scripted via TMSH or iControl. There is no direct iRules access for modifying the contents, making these a read-only data structure from iRules’ perspective. The config object would look like this (v11+, see the class command in the wiki for earlier versions): ltm data-group internal name_value_dg { records { name1 { data value1 } name2 { data "value2 with spaces" } } type string } What is a class? A class is exactly the same thing as a data-group. We have used the terms interchangeably for years, much to the chagrin and confusion of some users. We even called the command classwhile the structure is called “data-group”. Don’t let that confuse you, they are the same thing, and regardless of which you hear someone mention, they’re talking about the memory structure that I just described above. What are the benefits of a data-group? I mentioned before that one of the only drawbacks of data-groups is that they are read-only for all intents and purposes where iRules is concerned.That, however, is a tiny drawback in most cases when you consider the performance of data-groups. data-groups are far and away the most efficient memory structure with which to perform lookups past only a few entries. if/else and switch are fine to a point, but past about 10 items even the more efficient switch can’t keep up with the linear scaling of data-group lookups. Whether you’re storing 100 or 100,000 entries, queries are roughly the same thanks to the indexed, hashed format of data groups. This makes them far and away the best option for storing large lists of data as well as frequently performed queries that can be represented in a read-only fashion. The bottom line is: If you’re not updating the data in the list, and you’re dealing with more than a handful of items, be it strings, IPs or otherwise, data-groups are likely your best bet. They’re also resilient through failover, reboots, etc. unlike data structures without a disk based config object to back them up. Last, but in some cases not least at all, you can script the management of data-groups, especially external data-groups, via iControl. TMSH, etc. This makes it an ideal way to bulk load data or manage entries in the data structure externally. Just remember that you’ll have to re-instantiate the iRule for the changes to be recognized. What command(s) would I use to access data in a data-group? The command you’ll be using to access data-groups is the classcommand. In versions prior to 10 there were other commands (matchclass / findclass,) but as of v10 data-groups were completely overhauled to be more effective and higher performance, and the classcommand was born. It’s extremely powerful and has many permutations. I won’t attempt to go into all of them here, but you can see some basic matching examples here. When you combine the flexibility of the classcommand (full documentation here) with the performance and scalability of data-groups, you get some pretty powerful possibilities. Here are a couple examples from the wiki: when HTTP_REQUEST { if { [class match [IP::client_addr] equals "localusers_dg" ] } { COMPRESS::disable } } when HTTP_REQUEST { set app_pool [class match -value -- [HTTP::uri] starts_with app_class] if {$app_pool ne ""} { pool $app_pool } else { pool default_pool } } Data-Group Formatting Differences The format at the command line has changed over the years, please reference these articles for your particular version of TMOS v10 Data-Group Formatting Rules v11+ Data-Group Formatting RulesIntermediate iRules: Iteration
In this article, we're going to cover loop control structures and how to gain efficiency therein. One of the things that makes iRules so powerful is the fact that it is effectively a full scripting language at your fingertips with which you can inspect and modify traffic in real time. As with any scripting language, TCL (which iRules is based on) makes use of different loop control structures to perform different tasks. These are usually used in more advanced iRules, but we'll take a look at a couple of simple examples for ease of reading. In the below examples we're going to be looking at a relatively simple situation. In both we're going to be taking a list of strings, domain names in this case, and cycling through that list one at a time, performing some given tasks on each item in the list. For & While Loops In the first examples, we use the for and whileoop. These loops are two of the most common loop structures around, and are used in many, many languages. While these examples certainly function, you'll notice there are many steps required to get to the point of actually performing the actions based on the data in question. These are the preferred loops in many languages as they are relatively robust and functional, and the difference in overhead is less important. But this is a bit expensive for iRules, unless there is an actual need. It ultimately doesn't matter if you use a for or while in this scenario, but for loops are far more common in the examples you'll find on DevCentral. when HTTP_REQUEST { set domains {bob.com ted.com domain.com} set countDomains [llength $domains] #for loop for {set i 0} { $i < $countDomains } {incr i} { set domain [lindex $domains $i] # This is slower } #while loop set i 0 while { $i < $countDomains } { set domain [lindex $domains $i] incr i } } If you do need the additional flexibility with these loops, know that you have continue and breakat your disposal as well. The former will stop processing of the current loop, but move on to the next test condition, whereas the latter will exit the loop immediately and move on to the next line of code. It's also important to remember that these loops are blocking, so given the single-threaded nature of TMM, you really don't want to iterate through too many test cases before moving on. Unless of course you are trying to slow BIG-IP down for testing purposes, then, loop away! Foreach Loops This next example is logically similar. It accomplishes the same task in this case, and does so with less overhead. See how we have less variables to set, and less functions to perform for each loop iteration? While foreach is sometimes thought of as slightly less powerful or flexible than a for loop, it is markedly more efficient. As such, in cases where it will accomplish the task at hand, it is by far the preferred structure. when HTTP_REQUEST { set domains {bob.com ted.com domain.com} foreach domain $domains { # This is faster, and more elegant } } Looping as a Sleep Timer? Many moons ago someone asked in the forums (yes, we used to have forums) if there was a way to make iRules sleep for a period of time. I was relatively new to iRules but eager to help, so I weighed in that using a for loop with a massive counter would be the way to go. Colin quickly responded that that might not be the best idea, given the blocking nature of doing such a thing and the single threaded nature of TMM I mentioned above. At the time, there wasn't a great way, but you can accomplish this now with the aftercommand, which is non-blocking.Intermediate iRules: Handling Lists
We've talked about variables in the Getting Started with iRules series, but we're going to cover specifically the list object type of variable. Most of the TCL commands for use in manipulating lists are available in iRules: list - Create a list split - Split a string into a proper Tcl list join - Create a string by joining together list elements concat - Join lists together llength - Count the number of elements in a list lindex - Retrieve an element from a list lrange - Return one or more adjacent elements from a list linsert - Insert elements into a list lreplace - Replace elements in a list with new elements lsort - Sort the elements of a list lsearch - See if a list contains a particular element The remaining documented list handling commands, lrepeat, lreverse, & dict, are not available within iRules. We’ll cover the commands from above that get the most traffic here on DevCentral. split split string ?splitChars? Returns a list created by splitting string at each character that is in the splitChars argument. Each element of the result list will consist of the characters from string that lie between instances of the characters in splitChars. Empty list elements will be generated if string contains adjacent characters in splitChars, or if the first or last character of string is in splitChars. If splitChars is an empty string then each character of string becomes a separate element of the result list. SplitChars defaults to the standard white-space characters. In this example, the split commandis used to separate the hours/minutes/seconds returned from the formatted clockcommand into separate list elements: % set caTime [clock format [clock seconds] -format {%H:%M:%S}] 10:38:30 % set l [split $caTime :] 10 38 30 join join string ?splitChars? The list argument must be a valid Tcl list. This command returns the string formed by joining all of the elements of list together with joinString separating each adjacent pair of elements. The joinString argument defaults to a space character. In this example, the user is building the IP address using the join commandand the “.” as the splitChar. foreach num $IPtmp { lappend IP [expr ($num + 0x100) % 0x100] } set ::attr_value1 [join $IP .] Using a TCL shell, the output follows: % set IPtmp { 0x8c 0xaf 0x55 0x44 } 0x8c 0xaf 0x55 0x44 % foreach num $IPtmp { lappend IP [expr ($num + 0x100) % 0x100] } % puts $IP 140 175 85 68 % set IP [join $IP .] 140.175.85.68 concat concat ?arg arg … ? This command joins each of its arguments together with spaces after trimming leading and trailing white-space from each of them. If all the arguments are lists, this has the same effect as concatenating them into a single list. It permits any number of arguments; if no args are supplied, the result is an empty string. Here’s an excellent example of the concat command, which joinsthe contents of a cookie with the values from LB::server and virtual name: HTTP::cookie insert path / name ltmcookie value [concat [virtual name] [LB::server]] using the TCL shell, need variables in place of the calls to [virtual name] and [LB::server]: % set virtual_name myVip myVip % set lb_server "myPool 10.10.10.10 443" myPool 10.10.10.10 443 % set ltmcookie_value [concat $virtual_name $lb_server] myVip myPool 10.10.10.10 443 lindex lindex list ?index … ? This command accepts a parameter, list, which it treats as a Tcl list. It also accepts zero or more indices into the list. The indices may be presented either consecutively on the command line, or grouped in a Tcl list and presented as a single argument. This example, from the same iRule as the concat example above, shows the extraction of the list elements into usable variables: set vipid [lindex [HTTP::cookie ltmcookie] 0] set poolid [lindex [HTTP::cookie ltmcookie] 1] set serverid [lindex [HTTP::cookie ltmcookie] 2] set portid [lindex [HTTP::cookie ltmcookie] 3] Using the cookie variable we set in the above concat example, we can now extract the list items: % set vipid [lindex $ltmcookie_value 0] myVip % set poolid [lindex $ltmcookie_value 1] myPool % set serverid [lindex $ltmcookie_value 2] 10.10.10.10 % set portid [lindex $ltmcookie_value 3] 443 Better to avoid the variables, but a good illustration none-the-less. Often you’ll see the split and lindex commands used together: set trimID [lindex [split [HTTP::cookie "JSESSIONID"] "!" ] 0] Stepping through that, you can see first the split occur, then the indexing: % set jsessionID_cookie "zytPJpxV0TnpssqZZRLBgsVMLhGS6M2ZNMZ622yCNvpv0gkpTwzn!956498630!-34852364" zytPJpxV0TnpssqZZRLBgsVMLhGS6M2ZNMZ622yCNvpv0gkpTwzn!956498630!-34852364 % set l [split $jsessionID_cookie !] zytPJpxV0TnpssqZZRLBgsVMLhGS6M2ZNMZ622yCNvpv0gkpTwzn 956498630 -34852364 % set jsessID [lindex $l 0] zytPJpxV0TnpssqZZRLBgsVMLhGS6M2ZNMZ622yCNvpv0gkpTwzn There is a shortcut command in iRules for the split and lindex command pairing called getfield. Note that whereas lindex begins at 0, getfield begins at 1. The example below shows a direct comparison: # lindex/split set trimID [lindex [split [HTTP::cookie "JSESSIONID"] "!"] 0] # getfield set trimID [getfield [HTTP::cookie "JSESSIONID"] "!" 1] Note that using iRules-specific commands might help readability, but doing so does require a jump from the Tcl virtual machine back over to TMM to process. In this case it's incredibility negligible, but if you are squeezing out every CPU cycle it's something to be aware of.Intermediate iRules: catch
One of the often overlooked features of iRules is the ability to use dynamic variables in assignments. This can make for some interesting iRules, but it can also get you into trouble at runtime when your variables are not defined in the current configuration. This tech tip will illustrate how to take input from an HTTP request and use that as the criteria for which pool of servers the connection is sent to, all while maintaining runtime error checking using the TCL catchcommand. The pool command This command is used to assign a connection to a specified pool name (and optionally a specific pool member). pool Specifies the pool to which you want to send the traffic. pool ?member ??? Specifies a pool member to which you want to directly send the traffic. Typically, we see iRules written something like this (using hard-coded pool names). when HTTP_REQUEST { switch -glob [HTTP::uri] { "*.gif" - "*.jpg" - "*.png" { pool images_pool } "*.asp" { pool ms_app_pool } "*.jsp" { pool java_app_pool } } } When this iRule is created, the poolcommands are syntax checked and validated and since the pool name arguments are literals, the save logic will validate whether the given pools exist in the configuration. If they do not exist, a save error will occur and you will have to configure the pools before you save your iRule. In this case, no runtime exception will occur (unless you remove your pools out from under your iRule, that is). But, let's say you want to build a more dynamic solution that can expand over time. For this example, let's say that you want to allow your client to determine which pool of servers it is sent to. This could be the server specifying a pool name in an HTTP cookie, or simply appending it as a GET parameter on the URI. We'll use the later scenario for this example. For this example, we'll define the following GET parameters that can control pool direction. pool=name So an example URI could be: http://somedomain.com/somefile.ext?param1=val1&pool=pool_name¶rm2=val2... We will interrogate the URI's GET parameters in search of the "pool" parameter, extract the pool_name value and use that variable as the argument in the poolcommand. For newer versions of BIG-IP, one could use the URI::querycommand to extract values, but this implementation should work all the way back to BIG-IP v9.0. when HTTP_REQUEST { set namevals [split [HTTP::query] "&"] set pool_name "" for {set i 0} {$i < [llength $namevals]} {incr i} { set params [split [lindex $namevals $i] "="] set name [lindex $params 0] set val [lindex $params 1] switch $name { "pool" { set pool_name $val } } } if { "" ne $pool_name } { pool $pool_name } } What's wrong with this implementation? Nothing if your configuration has the given value in the $pool_name variable. What happens if that value doesn't exist in the configuration? catch The answer is that you will get a runtime error and the given connection will be broken. This is not an ideal solution and the simple use of the catchcommand can avoid runtime connection termination and allow the request to continue on through to a default pool of servers. The syntax for the catchcommand is as follows: catch script ?varName? The catchcommand may be used to prevent errors from aborting command interpretation. It calls the Tcl interpreter recursively to execute script, and always returns without raising an error, regardless of any errors that might occur while executing script. If script raises an error, catchwill return a non-zero integer value corresponding to the exceptional return code returned by evaluation of script. Tcl defines the normal return code from script evaluation to be zero (0), or TCL_OK. Tcl also defines four exceptional return codes: 1 (TCL_ERROR), 2 (TCL_RETURN), 3 (TCL_BREAK), and 4 (TCL_CONTINUE). Errors during evaluation of a script are indicated by a return code of TCL_ERROR. The other exceptional return codes are returned by the return, break, and continue commands and in other special situations as documented. Tcl packages can define new commands that return other integer values as return codes as well, and scripts that make use of the return -code command can also have return codes other than the five defined by Tcl. If the varName argument is given, then the variable it names is set to the result of the script evaluation. When the return code from the script is 1 (TCL_ERROR), the value stored in varName is an error message. When the return code from the script is 0 (TCL_OK), the value stored in resultVarName is the value returned from script. Solution The following iRule is the same as above, except that it makes use of exception handling in the dynamic pool assignment. when HTTP_REQUEST { set namevals [split [HTTP::query] "&"] set pool_name "" for {set i 0} {$i < [llength $namevals]} {incr i} { set params [split [lindex $namevals $i] "="] set name [lindex $params 0] set val [lindex $params 1] switch $name { "pool" { set pool_name $val } } } if { "" ne $pool_name } { if { [catch { pool $pool_name } ] } { log local0. "ERROR: Attempting to assign traffic to non-existant pool $pool_name" pool default_pool } } } Now, the control of which pool the connection is directed to is completely in the hands of the request URI. And in those rare situations where your app logic changes before the network configuration does, the connection will fallback to a default pool of servers, while logging the issue to the system log for later examination. Conclusion The ability to use dynamic variables greatly enhances the flexibility of what you can do with iRules commands. The catchcommand should be used around any iRule command that makes use of dynamic variables to catch any runtime errors that occur. In fact, if you so desire, you and nest catchcommands for a multi-layered exception handling solution.Intermediate iRules: Evaluating Performance
Customers frequently ask, as poster CodeIT did: "I am wondering what the effect of writing more elaborate iRules will have on the F5’s memory and processor. Is there a way to predict the effect on the F5’s resources for a given rule and traffic load?" In this article, I'll show you how to collect and interpret iRule runtime statistics to determine whether one version of a rule is more efficient than another, or to estimate a theoretical maximum number of connections possible for an iRule running on your LTM. (Props to unRuleY, a1l0s2k9, citizenelah, Joe, acarandang, and stephengun for insightful posts on this topic.) Collecting Statistics To generate & collect runtime statistics, you can insert the command "timing on" into your iRule. When you run traffic through your iRule with timing enabled, LTM will keep track of how many CPU cycles are spent evaluating each iRule event. You can enable rule timing for the entire iRule, or only for specific events. To enable timing for the entire iRule, insert the "timing on" command at the top of the rule before the first "when EVENT_NAME" clause. Here's an example enabling timing for all events in a rule: rule my_fast_rule { timing on when HTTP_REQUEST { # Do some stuff } } To enable iRule timing for only a specific event, insert the "timing on" command between the "when EVENT_NAME" declaration and the open curly brace. Here's an example enabling timing for only a specific event: rule my_slow_rule { when HTTP_REQUEST timing on { # Do some other stuff } } (See the timing command documentation for more details on timing only selected events by toggling timing on & off.) With the timing command in place, each time the rule is evaluated, LTM will collect the timing information for the requested events. To get a decent average for each of the events, you'll want to run at least a couple thousand iterations of the iRule under the anticipated production load. For http traffic I generally do this with apache bench (ab) from the BIG-IP command line, though you can use netcat and other linux tools as well for non-http traffic. Viewing Statistics The statistics for your iRule (as measured in CPU cycles) may be viewed at the command line or console by running tmsh ltm rule rule_name show all The output includes totals for executions, failures & aborts along with minimum, average & maximum cycles consumed for each event since stats were last cleared. RULE rule_name +-> HTTP_REQUEST 729 total 0 fail 0 abort | Cycles (min, avg, max) = (3693, 3959, 53936) If you use the iRules Editor , you can instead view the same stats on the Statistics tab of the Properties dialog. (Double-click on the iRule in the left pane to open the Properties dialog.) Evaluating statistics Average cycles reported is the most useful metric of real-world performance, assuming a large representative load sample was evaluated. The maximum cycles reported is often very large since it includes some one-time and periodic system overhead. (More on that below.) The iRules Runtime Calculator Excel spreadsheet will calculate percentage of CPU load per iteration once you populate it with your clock speed and the statistics gathered with the "timing" command. (Clock speed can be found by running 'cat /proc/cpuinfo' at the command line.) Or if you are not a glutton for manual punishment, you can generate that spreadsheet automatically with iControl REST via the F5 Python SDK and the XlsxWriter module . Caveats Timing is intended to be used only as an optimization/debug tool, and does have a small impact on performance, so don't leave it turned on indefinitely. Timing functionality seems to exhibit a 70 - 100 cycle margin of error. Use average cycles for most analyses. Maximum cycles is not always an accurate indicator of actual iRule performance, as the very first call a newly edited iRule includes the cycles consumed for compile-time optimizations, which will be reflected in an inflated maximum cycles value. The simple solution to this is to wait until the first time the rule is hit, then reset the statistics. To reset statistics at the command line: tmsh ltm rule [rule_name | all] stats reset or "Reset Statistics" in the Statistics tab of the iRules Editor. However, maximum cycles is also somewhat inflated by OS scheduling overhead incurred at least once per tick, so the max value is often overstated even if stats are cleared after compilation. Resources timing command iRules Runtime Calculator Comparing iRule Control Statements IP Comparisons Testing Short walk through on what makes timing tick. Using python to test performance and insert resulting data in iRules headers Note: This article authored by Deb Allen, reprinted and updated here for inclusion in new article series.Intermediate iRules: Validating Your Logic
Sometimes an iRule will load and run without producing any errors, but does not achieve the desired results. In this article, I will outline the basic process of examining your iRule logic against live traffic by walking through a simple example: An iRule intended to perform bi-directional HTTP Host header modifications. I recently helped a customer who had a fairly common configuration: The webservers are configured to respond to a different domain name than the one in the URLs the customer advertises for the virtual server: The "correct" host header containing the internal server name is required for the webservers to process a request. In addition, any server responses which contain the internal hostname need to be modified to reflect the publicly advertised name instead. These are the host names we will use for this example: Advertised name: easyname.domain.com Real/internal name: long.internal.name.domain.com For all HTTP requests, the hostname "easyname.domain.com" must be translated to "long.internal.name.domain.com". For all HTTP responses, the hostname "long.internal.name.domain.com" must be translated to "easyname.domain.com". Here is the initial iRule the customer created: when HTTP_REQUEST { if { [HTTP::host] equals "easyname.domain.com"} { HTTP::header replace Location \ [string map -nocase {easyname.domain.com long.internal.name.domain.com} \ [HTTP::header value Location]] } } when HTTP_RESPONSE { if { [HTTP::host] equals "long.internal.name.domain.com"} { HTTP::header replace Location \ [string map -nocase {long.internal.name.domain.com easyname.domain.com} \ [HTTP::header value Location]] } } When traffic was run against the rule, no errors were seen in the logs and traffic flowed normally between client and server, but the intended replacements were not performed. Given such unexpected behaviour, the first step in validating your logic is to ensure that the information your iRule sees and acts upon is what you expected it to see and act upon. To do so, you can add some logging around conditional decisions. The best practice is to first log the value of the variables, objects, or commands used to make the decision just prior to each decision point, and then log another message after each decision point to indicate the expected code block is indeed executing. Here is the customer's rule modified to include some informational logging around the conditions: when HTTP_REQUEST { # First we'll log the 2 header values used in the conditional code block log local0. "Host = [HTTP::host]" log local0. "Location = [HTTP::header Location]" if { [HTTP::host] equals "easyname.domain.com"} { # inside the conditional block, add another log line saying that's where you are log local0. "Host matched, performing replacement operation" HTTP::header replace Location \ "[string map -nocase {easyname.domain.com long.internal.name.domain.com} [HTTP::header value Location]]" # you can even log the result of the replacement operation by running it again with the log command log local0. "Replacement text = \ [string map -nocase {easyname.domain.com long.internal.name.domain.com} [HTTP::header value Location]]" } } when HTTP_RESPONSE { # For the response, we'll again log the header values used in the conditional code block log local0. "Host = [HTTP::host]" log local0. "Location = [HTTP::header Location]" if { [HTTP::host] equals "long.internal.name.domain.com"} { # inside the conditional block, add another log line saying that's where you are log local0. "Host matched, performing replacement operation" HTTP::header replace Location \ "[string map -nocase {long.internal.name.domain.com easyname.domain.com} [HTTP::header value Location]]" # and again, you can log the result of the replacement operation log local0. "Replacement text = \ [string map -nocase {long.internal.name.domain.com easyname.domain.com} [HTTP::header value Location]]" } } When requests for "easyname.domain.com" were pushed through this new iRule, the result was the same (no replacements performed) but the following (very helpful!) log entries were generated: HTTP_REQUEST: Host = easyname.domain.com HTTP_REQUEST: Location = HTTP_REQUEST: Host matched, performing replacement operation HTTP_REQUEST: Replacement text = Responses sometimes logged like this: HTTP_RESPONSE: Host = HTTP_RESPONSE: Location = and sometimes like this: HTTP_RESPONSE: Host = HTTP_RESPONSE: Location = http://long.internal.name.domain.com/uri Sooo... what got replaced? Not surprisingly, nothing. (At least the wrong "something" didn't get replaced. Or did it...? More on that later.) You can tell by the absence of a value after some of the “=” that something wasn't seen where we expected it. You can tell by the absence of some log lines that some of the expected conditions were not met. In a nutshell, these are the 2 issues at hand: No replacement text was generated on request even though the conditional code block was obviously executed. No hostname was seen on response, so the conditional code block was NOT executed and no replacement text was generated. Taking a closer look at the actual traffic being sent both ways (using HTTPwatch, tcpdump, or your favorite alternative trace tool) revealed some interesting and relevant details: Request from the client does not include the Location header, only the Host header. Requests on the serverside of LTM have both Host and Location headers. Response does not include the Host header, and only contains a Location header for redirect responses. The internal hostname is clearly visible in hyperlinks in the HTTP payload (not just the Host and Location headers). These observations lead to the following revelations and adjustments: "Replacing" the wrong request header here -- actually LTM is inserting a new blank "Location" header into each request. (That would the the "wrong something" I mentioned earlier...) Use [HTTP::host] for the request condition and [HTTP::header replace] for the replacement operation. (No [string map] is necessary here since we already know the host header value.) Using the wrong information for the response condition. Use [HTTP::status] for the response condition to limit processing only to redirects, then use [HTTP::header replace] with [string map] to replace only the hostname. Payload replacement isn't being handled by the iRule at all, and rightly so: Use a stream profile applied to the same virtual server to translate the links in the HTTP payload. The final iRule, including all the correct references, manipulations and optimizations, and implemented along with the stream profile, is as follows: when HTTP_REQUEST { if { [HTTP::host] equals "easyname.domain.com"} { # replace header completely if it matches HTTP::header replace Host "long.internal.name.domain.com" } } when HTTP_RESPONSE { if { [HTTP::status] starts_with "3" }{ # replace the Location header only if the response is a redirect, # since no other HTTP server responses contain the hostname in a header. HTTP::header replace Location \ [string map -nocase {long.internal.name.domain.com easyname.domain.com} [HTTP::header value Location]] # depend on stream profile to perform the hostname replacements in the HTTP payload } } Note: Information in this article written by Deb Allen.Intermediate iRules: Handling Strings
To say we’re getting to the heart of the matter, dealing with string commands and parsing, re-arranging and modification, would almost be saying it too lightly…understating. String manipulation is a massive part of iRules, and is in fact a solid part of why we are using Tcl as our language of choice, along with many others that I’ve covered elsewhere in detail. String manipulation is useful in many ways, in many places. Whether it’s re-writing a URI for an inbound HTTP request or parsing part of the TCP payload and twiddling the bits to read a bit differently, or perhaps just determining the first n characters of a string to be used for some purpose or another…all of it revolves around strings. Fortunately for us, Tcl handles strings extremely well. In fact, everything is a string in Tcl’s eyes, and as such there are many powerful tools with which you can twist strings to your desires with relative ease, and great effect. To cover all of the options would be a huge process, but we’ll go over the basics here, the things seen most commonly within iRules, and you can research the more obscure wizardry at will. The publicly available documentation is good for most of the commands in question. So, in this article we will cover what a string is and why you should care, as well as a large subset of the string commands you're most likely to use. What is a string and why do I care? A string is a particular data type, and is generally understood to be a sequence of characters, either as a literal constant, or represented in variable form. This means that basically anything can be a string. A name, an IP address, a URL…all of them are strings, especially in Tcl. Generally speaking, unless things are specifically typed as an integer or some other data type, it is a safe bet to assume they are a string. That being said, since Tcl is not a statically typed language and thereby does not allow you to specify data types explicitly, it treats everything as a string save for a few specific conditions. This is a good thing for iRules, as it means that there isn’t a lot of messing about with data types, and that you can generally manipulate things in string format without much hassle. That means less programming fuss, and more getting the effect you want. What are the most commonly used string commands? First off, the most common and widely used command in Tcl for dealing with strings is, quite simply, “string”. Mind you, in and of itself this command has little use. There are many, many permutations of this command from changing the case of a string to referencing only a portion of it, to re-ordering it and more. With this single command, and the many sub commands, you can perform the lion’s share of your string work within iRules. So the question is really which “string” sub commands are most commonly used? This one is a bit of an intense, broad sweeping question. There are so many things that you can do with a string in Tcl that listing them all here would be an indigestible amount of information, and wouldn’t make sense to portray. As such, I’ll do my best to list a few string commands that seem to often crop up in iRules, and discuss what each does. For a full reference on the string command and other Tcl base commands, you can find the official Tcl documentation online here (http://www.tcl.tk/man/tcl8.4/TclCmd/contents.htm ) string tolower string tolower string ?first? ?last? Without question the most common and widely used string command within iRules is also one of the simplest. The tolowercommand does pretty much what it sounds like. It converts the entirety of a string’s contents to lowercase. Meaning, if you had a variable named $uriand the contents were, “/Admin/WebAccess”, you could run the string tolowercommand when performing comparisons to get a different result. For instance: set uri “/Admin/WebAccess” log local0. “Uri : $uri” log local0. “Lower Uri: [string tolower $uri]” Would result in “Uri: /Admin/WebAccess” for the first log message, and “Lower Uri: /admin/webaccess” for the second. This is thanks to the string tolowercommand. Why is this so useful in iRules? Because any time you’re performing a string based comparison, it is important to be sure you’re comparing things in the same case. Think about comparing a host name, a URI, etc. and suddenly you may see why there’s so much value in this simple command. This becomes increasingly important with things like data groups, where you are comparing a single value against a broad range of key values. Being able to assure they are all in the proper case, and then force the incoming comparison value to that case is extremely useful. Keep in mind that this, like most of the other string commands, does not actually modify the string itself. If you took our above example where we provided the lowercase URI and referenced $uri again, it would still maintain the original case, unaltered. For example, ensuring you directed users attempting to access the admin portion of an application while ensuring they aren’t worried about proper casing gets simpler with the tolowercommand: when HTTP_REQUEST { if {([HTTP::uri] starts_with “/admin”) || ([HTTP::uri] starts_with “/Admin”)} { pool auth_pool } } Becomes: when HTTP_REQUEST { if {[string tolower [HTTP::uri]] starts_with “/admin”} { pool auth_pool } } string length string length string Much as you’d expect given the name, the string length command returns the length of the string in question. This can be used for many different things, but probably the most common use-case observed so far in iRules has been to ascertain whether or not a given command returned a proper result. For instance: when HTTP_REQUEST { set cookie_val [HTTP::cookie “x-my-cookie”] if {[string length $cookie_val > 1} { log local0. “cookie was passed properly” pool http_pool } } Of course there are many ways to perform a similar check, and some are even more efficient if all you’re trying to do is identify whether or not a command returned null or not, but if you want to check to see if a specific answer was set of at least n characters, or for a few other very handy purposes I’ve seen, the string lengthcommand can be handy. string range string range string first last The string rangecommand allows you to reference a particular portion of a given string and retrieve only that specific range of characters. This could be characters 1-10, the first character to the 3rd, or perhaps the 15th to the end of the string. There are many different ways to reference string segments and divide things up using this command, but the result is the same. It returns the value of the portion of the string you define. This has proved useful time and time again in iRules for things like retrieving portions of a URI, ensuring that a hostname starts with a particular prefix, or dozens of other such seemingly simple requirements. Without the string rangecommand those benign tasks would be a major headache. Note that the first character in the string starts with an ID of 0, not 1. For instance, if you’re looking at a URI that is “/myApp?user=bob” where bob is a variable username, and you’re looking to return only the username you have a few options, but string rangemakes that quite simple: when HTTP_REQUEST { set user [string range [HTTP::uri] 12 end] log local0. “User: $user” } This next example shows the removal of a non-standard port from the value returned by HTTP::host. Notice the use of end-5, which will use the range from character in the zero index through the character five short of the end of the string. when HTTP_REQUEST { if { [HTTP::host] ends_with "8010" } { set http_host [string range [HTTP::host] 0 end-5] HTTP::redirect "https://$http_host[HTTP::uri]" } } string map string map mapping string Where string rangeallows you to select a given part of a string and return it, string mapallows you to actually modify sub strings in-line. Also, instead of acting on a count or range of characters, string mapworks with an actual string of characters. Whereas with string rangeyou may want to look up a particular part of a URI, such as the first 10 characters, and see if they match a string, or route based on them or…something; with string mapyou are able to make changes in real-time, changing one string of characters to another. For instance with string rangeyou may have a logic statement like “Do the first 10 characters of the URI match x”. You’d supply the string to fetch the range from and the number of characters you want, by giving a beginning and end character. With string mapyou’d be saying something like “look for any string that looks like x, and change it to y in the given string” by providing the string to work against as well as a source and destination string, meaning “Change all cases of http to https”. when HTTP_RESPONSE { set new_uri [string map {http https} [HTTP::header “Location”]] HTTP::header replace Location $new_uri } Of note, the string is only iterated over once, so earlier key replacements will have no affect for later key matches. For example: % string map {abc 1 ab 2 a 3 1 0} 1abcaababcabababc 01321221 What?? That's one of those not so intuitive examples in the TCL documentation. Actually, though, I like this one. Let's break it down. There are four key/value pairs here: abc, if found, will be replaced by a 1 ab, if found, will be replaced by a 2 a, if found, will be replaced by a 3 1, if found, will be replaced by a 0 String Map Multiple Key/Value Example Mapping Original String Resulting String 1 st (abc->1) 1abcaababcabababc 11aab1abab1 2 nd (ab->2) 11aab1abab1 11a21221 3 rd (a->3) 11a21221 11321221 4 th (1->0) 11321221 01321221 Note that with the fourth map, the returned string is 01321221, not 00320220. Why is that? Well, the string is only iterated over once, so earlier key replacements will have no affect for later key matches. string first string first string1 string2 ?startIndex? The string firstcommand allows you to identify the first occurrence of a given sub string of characters within a string. This can be extremely useful for combining with the string rangecommand. For instance, if I want to find the first occurrence of “/admin” in a URI and collect the URI from that point to the end, it would be quite difficult without the string firstcommand. What if I don’t know what the exact URI will be? What if there is a variable portion of the URI that comes before “/admin” that I don’t want to collect, but have to somehow account for even though it is variable in length? I can’t just set a static rangeand use the string rangecommand alone, so I have to get creative and combine commands. By making use of the string firstcommand, if I have a URI that looks something like “/users/apps/bob/bobsapp?user=admin” where the username is variable length and I can’t be certain of the length of the URI because of it, but I wanted to retrieve the user argument being passed in, I could do something like: set user [string range [HTTP::uri] [expr {[string first “user=” [HTTP::uri]] + 5}] end] What the above is doing is finding the first occurance of “user=” in the URI and returning the index of the first character. Then adding 5 to that, since that is the length of the string “user=”, and we want to reference what comes after “user=”, not include it, then take the range of the string from that point to the end, and return that as the value of the username being passed in. It looks a bit complex, but if you break it down command by command, you’re really just stringing together several smaller commands to get the functionality you want. And now you can start to see why string commands are so important and powerful in iRules. string last string last needleString haystackString ?lastIndex? Similar to string first, only returns the index of the first character in the last such match within haystackString. If there is no match, then return -1. If lastIndex is specified, then only the characters in haystackString at or before the specified lastIndex will be considered by the search. In this example, (combined with string range, the conditional returns true if the string beginning with the last "/" in the URI, and ending with the last character of the URI, contains a "." when HTTP_REQUEST { if {[class match [HTTP::uri] ends_with unmc_extends] or [class match [HTTP::method] equals unmc_methods] or [class match [HTTP::uri] contains unmc_sql] or [class match [IP::client_addr] equals unmc_restrict_ips]}{ discard } elseif { ([HTTP::uri] ends_with "/") or ([string range [string last / [HTTP::uri]] end] contains ".") or ([HTTP::uri] contains "unmcintranet")}{ pool unmc-intranet-proxy } elseif {[HTTP::uri] contains "google"}{ HTTP::redirect "http://[HTTP::host][HTTP::uri]&restrict=unmcintranet" } else { HTTP::redirect "http://[HTTP::host][HTTP::uri]/" } } string trim string trim string ?chars? Again, a command true to its name, the string trimcommand allows you to manipulate the beginning and end of a given string to trim off unwanted characters. Perhaps you have a string that you need to ensure doesn’t begin or end with white space, or you’re looking at URI comparisons and need to be sure that you don’t have a trailing slash in some cases and not others. Regardless, the string trimcommand makes that easy. All you’d do is specify which characters you want to ensure are removed, either whitespace or slashes in the examples just mentioned, and you’d be set to ensure standardized comparisons. For instance, if you want to ensure that someone is making a request to the admin section of your domain you can be sure that they are going to have a slash at the beginning of the URI, but they may or may not include a trailing slash. You could use the starts_withcomparison operator to ignore this, but that would also ignore anything else they may include after the specific string. If you want an exact match for either “/admin” or “/admin/” you could use an or, or could trim the URI, like so: when HTTP_REQUEST { if{[string trim [HTTP::uri] “/”] eq “admin”} { pool admin_pool } } Note that by using the trim command it removed both the preceding and trailing slashes. There are trimright and trimleftversions as well, for only trimming one end of a string, if that’s necessary. Note: The source material for this article was pulled from articles authored by Joe Pruitt, Colin Walker, and Jason Rahm. They are now archived but can still be referenced if necessary.Intermediate iRules: High Speed Logging - Spray Those Log Statements!
High Speed Logging has been around since version 10.1, and has been integral to many projects over the past few years. Prior to HSL's introduction, logging remotely was configured entirely in syslog or could be handled in iRules by specifying a destination in the log statement. One enhancement with HSL to that scenario was to allow a pool of servers to be configured for a destination, so given a pool of servers, the log messages were sure to arrive somewhere (ok, for TCP they were sure to arrive!) A drawback with either the log or HSL::send command, however, is that the message was only going to hit one destination. A workaround for that problem is to just use as many commands as necessary to hit all your destinations, but that's not very efficient. Enter the publisher. Beginning in version 11.3, a new option to the HSL::open command was added that allows you to send data to a log publisher instead of only to a pool. This allows you to spray that data to as many servers as you like. In my test setup, I used alias interfaces on a linux virtual machine as the destinations, and created a pool for each to be added to the publisher: ltm pool lp1 { members { 192.168.101.20:514 { address 192.168.101.20 } } } ltm pool lp2 { members { 192.168.101.21:514 { address 192.168.101.21 } } } ltm pool lp3 { members { 192.168.101.22:514 { address 192.168.101.22 } } } Once I have the pools defined, I create the log destinations: sys log-config destination remote-high-speed-log lp1 { pool-name lp1 protocol udp } sys log-config destination remote-high-speed-log lp2 { pool-name lp2 protocol udp } sys log-config destination remote-high-speed-log lp3 { pool-name lp3 protocol udp } Finally, I create the publisher for use in the iRules: sys log-config publisher lpAll { destinations { lp1 lp2 lp3 } } That's all the background magic required to get to the iRules showing off the -publisher option in HSL::open: ltm rule testrule { when CLIENT_ACCEPTED { set lpAll [HSL::open -publisher /Common/lpAll] } when HTTP_REQUEST { HSL::send $lpAll "<190> [IP::client_addr]:[TCP::client_port]-[IP::local_addr]:[TCP::local_port]; [HTTP::host][HTTP::uri]" } } Finally, some visual evidence for the skeptics out there: You can see that all three destinations got the message, and the message arrived as formatted. So now, armed with this new option (as of version 11.3), go forth and code!Intermediate iRules: String Handling Exercise
I had the privilege of watching an exercise in string handling unfold on an internal list recently that should have its day in the community. The Problem URLs for this particular application arrive in this format: /session/636971/abcd/hlsa/dcwiuh/index.html /session/128766/ssrg/hlsb/yuyt/yutyu/index.html /session/128766/dwlmhlsb/ewfwef/index.html /session/122623/wqs/wedew/mhlsb/ewfwef/uiy/index.html The useful data to extract exists between /session/[nnn] and index.html, so the final data we want, respectively, is: /abcd/hlsa/dcwiuh /ssrg/hlsb/yuyt/yutyu /dwlmhlsb/ewfwef /wqs/wedew/mhlsb/ewfwef/uiy The Solutions There are almost always multiple solutions to a problem set, and this scenario is no different. Each approach is shown below. The proc wrapper is only for testing purposes, the code within the proc is the actual solution for each approach. Approach 1: Split & Lists proc splitjoin {arg} { # convert: /session/1111/abcd/efg/hij/index.html # to: { {} session 1111 abcd efg hih index.html } set sl [split $arg {/}] # remove leading { {} session 1111 } and trailing index.html from $sl, # then produce / followed by remainder of $sl joined with / set result "/[join [lrange $sl 3 [expr { [llength $sl] -2}]] {/}]" } Approach 2a: Scan with all variables proc scana {arg} { scan $arg {/%[^/]/%[^/]%s} a b c set result [string range $c 0 [expr { [string last {/} $c] - 1 }]] } Approach 2b: Scan with the necessary variable proc scanb {arg} { # split the path, skipping ‘session’ and ‘[nnn]’ and setting c to ‘/[xxx]/[yyy]’ scan $arg {/%*[^/]/%*[^/]%s} c # remove the set of characters following the last slash (i.e., ‘/[yyy]’) in c set result [string range $c 0 [expr { [string last {/} $c] - 1 }]] } Approach 3: Regular Expressions proc regex1 {arg} { # after match, $whole is /session/[nnn]/[xxx]/ and $result is /[xxx] regexp {^/[^/]+/[^/]+(/.+)/} $arg whole result } The Results In order from least to most efficient: Approach Command Time Regular Expressions time {regex1 $x} 100000 17.22531 microseconds Scan will all variables time {scana $x} 100000 7.999566 microseconds Split & Lists time {splitjoin $x} 100000 6.90115 microseconds Scan with necessary variable time {scanb $x} 100000 6.12924 microseconds Where $x is the longest of the original strings at the top of this article. I ran these tests in the tclsh on BIG-IP LTM VE 11.2.1 running on an ESXi 4.1 installation. Actual numbers in iRules will likely be different, but the performance of these commands in relation to one another shouldn't vary much. Many thanks to F5er Vernon Wells, Cameron Jenkins, and Ken Wong for the source information!Intermediate iRules: Clarifying IP::tos command usage
The IP::tos command has been around since the v4 days, and there has been some confusion on IP tos versus DSCP. The command takes the supplied value (or returns a value) and applies to the tos field overall, which is eight bits. So if you take express forwarding, which is DSCP 46 and use IP::tos 46, you'll notice on the wire that something is amiss: What gives? Well, the DSCP markings are in the first 6 bits of the tos field, so you need to bit-shift to the left a couple places. That results in 46 * 2 2 = 184. So 184 is the value you would use with the IP::tos command. 00101110 = 46 10111000 = 184 By setting IP::tos 184, you can see on the wire that all is now well in the world: The below table maps the DSCP values in decimal to the value you'll need to use with the IP::tos command to get expected results. DSCP Mappings for IP::tos Command Precedence Type of Service DSCP Class DSCP Value IP::tos Value 0 0 none 0 0 1 0 cs1 8 32 1 1 af11 10 40 1 10 af12 12 48 1 11 af13 14 56 10 0 cs2 16 64 10 1 af21 18 72 10 10 af22 20 80 10 11 af23 22 88 11 0 cs3 24 96 11 1 af31 26 104 11 10 af32 28 112 11 11 af33 30 120 100 0 cs4 32 128 100 1 af41 34 136 100 10 af42 36 144 100 11 af43 38 152 101 0 cs5 40 160 101 11 ef 46 184 110 0 cs6 48 192 111 0 cs7 56 224 I updated the TCP Profile QoS article I wrote a few years back with this updated table, as well as the IP::tos command page in the iRules documentation. Thanks to F5er Alex for tracking this down!
