Working with subsets of data-group records via iControl REST

The BIG-IP iControl REST interface method for data-groups does not define the records as a subcollection (like pool members.) This is problematic for many because the records are just a list attribute in the data-group object. This means that if you want to add, modify, or delete a single record in a data-group, you have replace the entire list. A short while back I was on a call with iRule extraordinaire John Alam, and he was showing me a management tool he was working on where he could change individual records in a data-group via REST. I was intrigued so we dug into the details and I was floored at how simple the solution to this problem is!

UPDATE: Chris L mentioned in the comments below that working with subsets IS possible without the tmsh script, as he learned in this thread in the Q&A section. The normal endpoint (/mgmt/tm/ltm/data-group/internal/yourDGname) works just fine, but instead of trying to change a subset of the records attribute, which only results in the replace-all-with behavior, you can use the options query parameter and then pass the normal tmsh command records data as arguments. An example of this request would be to PATCH with an empty json payload ({}) to url https://{{host}}/mgmt/tm/ltm/data-group/internal/mydg?options=records%20modify%20%7B%20k3%20%7Bdata%20v3%20%7D%20%7D (without the encoding, that query value format is “records modify { k3 { data v3 } }”). As this article is still a good learning exercise on how to use tmsh scripts with iControl REST, I’ll keep the article as is, but an updated script for the specific problem we’re solving can be found in this gist on Github.

Enter the tmsh script!

Even though the iControl REST doesn’t treat data-group records as individual objects, the tmsh cli does. So if you can create a tmsh script to manage the local manipulation of the records, pass your record data into that script, and execute it from REST, well, that’s where the gold is, people. Let’s start with the tmsh script, written by John Alam but modified very slightly by me.

cli script dgmgmt {
proc script::init {} {
}

proc script::run {} {
set record_data [lindex $tmsh::argv 3]

switch [lindex $tmsh::argv 1] {
   "add-record" {
      tmsh::modify ltm data-group internal [lindex $tmsh::argv 2] type string records add $record_data
      puts "Record [lindex $tmsh::argv 3]  added."
   }
   "modify-record" {
      tmsh::modify ltm data-group internal [lindex $tmsh::argv 2] type string records modify $record_data
      puts "Record changed [lindex $tmsh::argv 3]."
   }
   "delete-record" {
      tmsh::modify ltm data-group internal [lindex $tmsh::argv 2] type string records delete $record_data
      puts "Record [lindex $tmsh::argv 3]  deleted."
   }
   "save" {
      tmsh::save sys config
      puts "Config saved."
   }
}
}
proc script::help {} {
}

proc script::tabc {} {
}
    total-signing-status not-all-signed
}

This script is installed on the BIG-IP and is a regular object in the BIG-IP configuration, stored in the bigip_script.conf file. There are four arguments passed. The first (arg 0) is always the script name. The other args we pass to the script are:

• arg 1 - action. Are we adding, modifying, or deleting records?
• arg 2 - data-group name
• arg 3 - data-group records to be changed

The commands are pretty straight forward. Notice, however, that the record data at the tail end of each of those commands is just the data passed to the script, so the required tmsh format is left to the remote side of this transaction. Since I’m writing that side of the solution, that’s ok, but if I were to put my best practices hat on, the record formatting work should really be done in the tmsh script, so that all I have to do on the remote side is pass the key/value data.

Executing the script!

Now that we have a shiny new tmsh script for the BIG-IP, we have two issues.

1. We need to install that script on the BIG-IP in order to use it
2. We need to be able to run that script remotely, and pass data to it

This is where you grab your programming language of choice and go at it! For me, that would be python. And I’ll be using the BIGREST SDK to interact with BIG-IP. Let’s start with the program flow:

if __name__ == "__main__":
    args = build_parser()
    b = instantiate_bigip(args.host, args.user)

    if not b.exist("/mgmt/tm/cli/script/dgmgmt"):
        print(
            "\n\tThe data-group management tmsh script is not yet on your system, installing..."
        )
        deploy_tmsh_script(b)
        sleep(2)

    if not b.exist(f"/mgmt/tm/ltm/data-group/internal/{args.datagroup}"):
        print(
            f"\n\tThe {args.datagroup} data-group doesn't exist. Please specify an existing data-group.\n"
        )
        sys.exit()

    cli_arguments = format_records(args.action, args.datagroup, args.dgvalues)
    dg_update(b, cli_arguments)
    dg_listing(b, args.datagroup)

This is a cli script, so we need to create a parser to handle the arguments. After collecting the data, we instantiate BIG-IP. Next, we check for the existence of the tmsh script on BIG-IP and install it if it is not present. We then format the record data and proceed to supply that output as arguments when we make the REST call to run the tmsh script. Finally, we print the results. This last step is probably not something you'd want to do for large data sets, but it's included here for validation. Now, let's look at each step of the flow.

The imports and tmsh script

# Imports used in this script
from bigrest.bigip import BIGIP
from time import sleep
import argparse
import getpass
import sys

# The tmsh script
DGMGMT_SCRIPT = 'proc script::init {} {\n}\n\nproc script::run {} {\nset record_data [lindex $tmsh::argv 3]\n\n' \
                'switch [lindex $tmsh::argv 1] {\n   "add-record" {\n      tmsh::modify ltm data-group internal ' \
                '[lindex $tmsh::argv 2] type string records add $record_data\n      ' \
                'puts "Record [lindex $tmsh::argv 3]  added."\n   }\n   "modify-record" {\n      ' \
                'tmsh::modify ltm data-group internal [lindex $tmsh::argv 2] type string records modify' \
                ' $record_data\n      puts "Record changed [lindex $tmsh::argv 3]."\n   }\n   "delete-record" {\n' \
                '      tmsh::modify ltm data-group internal [lindex $tmsh::argv 2] type string records delete' \
                ' $record_data\n      puts "Record [lindex $tmsh::argv 3]  deleted."\n   }\n   "save" {\n ' \
                '     tmsh::save sys config\n      puts "Config saved."\n   }\n}\n}\n' \
                'proc script::help {} {\n}\n\nproc script::tabc {} {\n}\n'

These are defined at the top of the script and are necessary to the appropriate functions defined in the below sections. You could move the script into a file and load it, but it's small enough that it doesn't clutter the script and makes it easier not to have to manage multiple files.

The parser

def build_parser():
    parser = argparse.ArgumentParser()
    parser.add_argument("host", help="BIG-IP IP/FQDN")
    parser.add_argument("user", help="BIG-IP Username")
    parser.add_argument(
        "action", help="add | modify | delete", choices=["add", "modify", "delete"]
    )
    parser.add_argument("datagroup", help="Data-Group name you wish to change")
    parser.add_argument(
        "dgvalues", help='Key or KV Pairs, in this format: "k1,k2,k3=v3,k4=v4,k5"'
    )
    return parser.parse_args()

This is probably the least interesting part, but I'm including it here to be thorough. The one thing to note is the cli format to supply the key/value pairs for the data-group records. I could have also added an alternate option to load a file instead, but I'll leave that as an exercise for future development. If you supply no arguments or the optional -h/--help, you'll get the help message.

% python dgmgmt.py -h
usage: dgmgmt.py [-h] host user {add,modify,delete} datagroup dgvalues

positional arguments:
  host                 BIG-IP IP/FQDN
  user                 BIG-IP Username
  {add,modify,delete}  add | modify | delete
  datagroup            Data-Group name you wish to change
  dgvalues             Key or KV Pairs, in this format: "k1,k2,k3=v3,k4=v4,k5"

optional arguments:
  -h, --help           show this help message and exit

Instantiation

def instantiate_bigip(host, user):
    pw = getpass.getpass(prompt=f"\n\tWell hello {user}, please enter your password: ")
    try:
        obj = BIGIP(host, user, pw)
    except Exception as e:
        print(f"Failed to connect to {args.host} due to {type(e).__name__}:\n")
        print(f"{e}")
        sys.exit()
    return obj

I don't like typing out my passwords on the cli so I use getpass here to ask for it after I kick off the script. You'll likely want to add an argument for the password if you automate this script with any of your tooling. This function makes a request to BIG-IP and builds a local python object to be used for future requests.

Uploading the tmsh script

def deploy_tmsh_script(bigip):
    try:
        cli_script = {"name": "dgmgmt", "apiAnonymous": DGMGMT_SCRIPT}
        bigip.create("/mgmt/tm/cli/script", cli_script)
    except Exception as e:
        print(f"Failed to create the tmsh script due to {type(e).__name__}:\n")
        print(f"{e}")
        sys.exit()

Because tmsh scripts are BIG-IP objects, we don't have to interact with the file system. It's just a simple object creation like creating a pool. I have taken the liberty to hardcode the script name to limit the number of arguments required to pass on the cli, but that can be updated if you so desire by either changing the name in the script, or adding arguments.

Formatting the records

def format_records(action, name, records):
    recs = ""
    for record in records.split(","):
        x = record.split("=")
        record_key = x[0]
        if len(x) == 1 and action != 'modify':
            recs += f"{record_key} "
        elif len(x) == 1 and action == 'modify':
            recs += f'{record_key} {{ data \\\"\\\" }} '
        elif len(x) == 2:
            record_value = x[1]
            recs += f'{record_key} {{ data \\\"{record_value}\\\" }} '
        else:
            raise ValueError("Max record items is 2: key or key/value pair.")
    return f"{action}-record {name} '{{ {recs} }}'"

This is the function I spent the most time ironing out. As I pointed out earlier, it would be better handled in the tmsh script, but since that work was already completed by John, I focused on the python side of things. The few things that I fleshed out in testing that I didn't consider while making it work the first time:

1. Escaping all the special characters that make iControl REST unhappy.
2. Handling whitespace in the data value. This requires quotes around the data value.
3. Modifying a key by removing an existing value. This requires you to provide an empty data reference.

Executing the script

def dg_update(bigip, cli_args):
    try:
        dg_mods = {"command": "run", "name": "/Common/dgmgmt", "utilCmdArgs": cli_args}
        bigip.command("/mgmt/tm/cli/script", dg_mods)
    except Exception as e:
        print(f"Failed to modify the data-group due to {type(e).__name__}:\n")
        print(f"{e}")
        sys.exit()

With all the formatting out of the way, the update is actually anticlimactic. iControl REST requires a json payload for the command, which is running the cli script. The cli arguments for that script are passed in the utilCmdArgs attribute.

Validating the results

def dg_listing(bigip, dgname):
    dg = b.load(f'/mgmt/tm/ltm/data-group/internal/{dgname}')
    print(f'\n\t{args.datagroup}\'s updated record set: ')
    for i in dg.properties['records']:
        print(f'\t\tkey: {i["name"]}, value: {i["data"]}')
    print('\n\n')

And finally, we bask in the validity of our updates! Like the update function, this one doesn't have much to do. It grabs the data-group contents from BIG-IP and prints out each of the key/value pairs. As I indicated earlier, this may not be desirable on large data sets. You could modify the function by passing the keys you changed and compare that to the full results returned from BIG-IP and only print the updates, but I'll leave that as another exercise for future development.

This is a cool workaround to the non-subcollection problem with data-groups that I wish I'd thought of years ago! The full script is in the codeshare. I hope you got something out of this article, drop a comment below and let me know!