Forcing a reload of External Data Groups within an iRule

This article shows how to use iControl to force an iRule to reload external configuration data when that data has changed.

Background

One of the more common uses of iRules is to extract part of the request (client-ip, host, uri, payload, ...) and use that as input to a policy making decision.  Most often, the data you are comparing your input to (a list of blacklisted ip addresses, a list of valid hosts, uri to server mappings, etc) is too long to store directly within an iRule.  


Data Groups

That's where Data Groups come in.  Data Groups (or classes) are simply lists that can be references at the global scope from within an iRule to make the iRule itself more manageable.  You can use the matchclass and findclass helper commands to lookup values in Data Groups or you could use the TCL list commands to loop through the list manually.  There are four types of data groups: Address, String, Integer, and External.

Address Data Groups are used to store a list of IP Addresses (and optionally subnet masks).  These are often used when comparing an incoming client address against a list of valid or invalid addresses.

String Data Groups store a list newline delimited strings.  These are multi-purpose as you can use them for straight lookups (valid requested uri or hostname) or for building mapping (this uri gets routed to that pool).  Given a token delimiter, the findclass command can be used to look for the first token in the string and return the second token.

Integer Data Groups store a list of integers.  A common use for this type of data group is to compare an ID passed in as part of the URI to a valid value.  While this could be accomplished with a string data group, integer data groups require less resources and comparisons are faster.

External Data Groups are lists that are stored externally the data base (on the file system).  These can be of any of the other three types (Address, String, and Integer).  A common use for this type of Data Group is when you are storing thousands of values and those values change often.  It is much easier to manage an external file via a text editor, or a remote API like iControl.  

The Problem

The iRules engine utilizes a caching mechanism for external Data Groups.  The problem is that when a change is made to an External Data Group, The iRules engine has no way of knowning that the changes have actually been made.  Even a configuration reload will not do it as a reload is just an overwrite on the current configuration and since the iRule didnt' change the internal caching doesn't get invalidated and the next time the iRule is processed it still reads the cached values in the Data Group.

The Solution

Fear not, there is a fairly simple solution.  If you invalidate the iRule itself by changing it, the cache will be invalidated and the External Data Group will be reloaded.  How do you do this?  Well, we've found that the easiest way is to add a space to the end of the iRule.  This will not change the behaviour of the iRule but will cause it's checksum to change.  Just add a space to the end, save the iRule, then delete the space, and then save the iRule again!

This can be accomplished via the GUI, CLI, or iControl.  The GUI and CLI are fiarly intuitive, so I'll show you how to use iControl to do this.  More than likely for large External Data Groups, you are using iControl already to remotely add/remove items from the lists.  If that is the case, it is a very simple addition to your application to add this logic.  If you are not using iControl to manage your External Data Group, then you can plop this perl script right on your BIG-IP and use it from the command line.  Either way, we've got you covered.

Here's an iControl application written in Perl that will invalidate an iRule to force reloading of Data Groups.

#!/usr/bin/perl
#----------------------------------------------------------------------------
# The contents of this file are subject to the "END USER LICENSE AGREEMENT FOR F5 
# Software Development Kit for iControl"; you may not use this file except in
# compliance with the License. The License is included in the iControl
# Software Development Kit. 
#
# Software distributed under the License is distributed on an "AS IS"
# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See
# the License for the specific language governing rights and limitations
# under the License.
#
# The Original Code is iControl Code and related documentation
# distributed by F5.
#
# The Initial Developer of the Original Code is F5 Networks,
# Inc. Seattle, WA, USA. Portions created by F5 are Copyright (C) 1996-2006 F5 Networks,
# Inc. All Rights Reserved.  iControl (TM) is a registered trademark of F5 Networks, Inc.
#
# Alternatively, the contents of this file may be used under the terms
# of the GNU General Public License (the "GPL"), in which case the
# provisions of GPL are applicable instead of those above.  If you wish
# to allow use of your version of this file only under the terms of the
# GPL and not to allow others to use your version of this file under the
# License, indicate your decision by deleting the provisions above and
# replace them with the notice and other provisions required by the GPL.
# If you do not delete the provisions above, a recipient may use your
# version of this file under either the License or the GPL.
#----------------------------------------------------------------------------

#use SOAP::Lite + trace => qw(method debug);
use SOAP::Lite;

#----------------------------------------------------------------------------
# Variables
#----------------------------------------------------------------------------
my $DEBUG = 0;
my $sHost = "localhost";
my $sPort = 80;
my $sUID = "";
my $sPWD = "";
my $sProtocol = "http";
my $sRuleName = "MSM_reputation";

#----------------------------------------------------------------------------
# Transport Information
#----------------------------------------------------------------------------
sub SOAP::Transport::HTTP::Client::get_basic_credentials
{
  return "$sUID" => "$sPWD";
}

#----------------------------------------------------------------------------
# iControl interface declarations
#----------------------------------------------------------------------------
$LocalLBRule = SOAP::Lite
  -> uri('urn:iControl:LocalLB/Rule')
  -> readable(1)
  -> proxy("$sProtocol://$sHost:$sPort/iControl/iControlPortal.cgi");

#----------------------------------------------------------------------------
# Main logic
#----------------------------------------------------------------------------
&invalidateRule();

#----------------------------------------------------------------------------
# sub invalidateRule
#
# This iRule will query the contents of an iRule, save it with an added space
# and then re-save it with the original contents.
#----------------------------------------------------------------------------
sub invalidateRule()
{
  if ( $DEBUG ) { print "Querying rule definition for rule '$sRuleName'\n"; }

  # Query the rule definition
  $soapResponse = $LocalLBRule->query_rule(SOAP::Data->name(rule_names => [$sRuleName]));
  &checkResponse($soapResponse);
  
  @RuleDefinitionList = @{$soapResponse->result};

  # Extract the first element in the array
  $RuleDefinition = @RuleDefinitionList[0];
  
  # save the original and temporary changed values
  $rule_definition = $RuleDefinition->{"rule_definition"};
  $temp_rule_definition = "$rule_definition ";

  # update RuleDefinition with the temporary rule details
  if ( $DEBUG ) { print "modifying with temporary changes...\n"; }
  $RuleDefinition->{"rule_definition"} = $temp_rule_definition;
  $soapResponse = $LocalLBRule->modify_rule(SOAP::Data->name(rules => [$RuleDefinition]));
  &checkResponse($soapResponse);

  # update RuleDefinition with the original rule details.
  if ( $DEBUG ) { print "modifying with reverted changes...\n"; }
  $RuleDefinition->{"rule_definition"} = $rule_definition;
  $soapResponse = $LocalLBRule->modify_rule(SOAP::Data->name(rules => [$RuleDefinition]));
  &checkResponse($soapResponse);

  if ( $DEBUG ) { print "rule initialized...\n"; }
}

#----------------------------------------------------------------------------
# sub checkResponse
# Exit on SOAP Fault.
#----------------------------------------------------------------------------
sub checkResponse()
{
  my ($soapResponse) = (@_);
  if ( $soapResponse->fault )
  {
    print $soapResponse->faultcode, " ", $soapResponse->faultstring, "\n";
    exit();
  }
}