Demystifying iControl REST Part 2 - Understanding sub collections and how to use them

iControl REST. It’s iControl SOAP’s baby brother, introduced back in TMOS version 11.4 as an early access feature but was released fully in version 11.5.

Several articles on basic usage have been written on iControl REST (see the resources at the bottom of this article) so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. The first article of this series covered the URI’s role in the API. This second article will cover how the URI path plays a role in how the API functions.

Working with Subcollections

When manipulating F5 configuration items with iControl Rest, subcollections are a powerful tool.  They allow one to manipulate specific items in the subcollection instead of having to manipulate the entire sub collection. Take the pool object for example. If we just query the pool (in this case testpool,) you’ll notice the returned data does not list the pool members

#query (via the Chrome advanced REST client using Authorization & Content-Type headers:)
https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool
#query (via curl:)
curl -k -u admin:admin https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool

#response:
{
kind: "tm:ltm:pool:poolstate"
name: "testpool"
fullPath: "testpool"
generation: 1
selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool?ver=11.6.0"
allowNat: "yes"
allowSnat: "yes"
ignorePersistedWeight: "disabled"
ipTosToClient: "pass-through"
ipTosToServer: "pass-through"
linkQosToClient: "pass-through"
linkQosToServer: "pass-through"
loadBalancingMode: "round-robin"
minActiveMembers: 0
minUpMembers: 0
minUpMembersAction: "failover"
minUpMembersChecking: "disabled"
queueDepthLimit: 0
queueOnConnectionLimit: "disabled"
queueTimeLimit: 0
reselectTries: 0
serviceDownAction: "none"
slowRampTime: 10
membersReference: {
link: "https://localhost/mgmt/tm/ltm/pool/~Common~testpool/members?ver=11.6.0"
isSubcollection: true
}-
}

Notice the isSubcollection: true for the membersReference? This is an indicator that there is a subcollection for the members keyword. If you then query the members for that pool, you will get the subcollection.

#query (via the Chrome advanced REST client using Authorization & Content-Type headers:)
https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members
#query (via curl:)
curl -k -u admin:admin https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members

#response:
{
kind: "tm:ltm:pool:members:memberscollectionstate"
selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool/members?ver=11.6.0"
items: [4]
0:  {
kind: "tm:ltm:pool:members:membersstate"
name: "192.168.103.10:80"
partition: "Common"
fullPath: "/Common/192.168.103.10:80"
generation: 1
selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool/members/~Common~192.168.103.10:80?ver=11.6.0"
address: "192.168.103.10"
connectionLimit: 0
dynamicRatio: 1
ephemeral: "false"
fqdn: {
autopopulate: "disabled"
}-
inheritProfile: "enabled"
logging: "disabled"
monitor: "default"
priorityGroup: 0
rateLimit: "disabled"
ratio: 1
session: "user-enabled"
state: "unchecked"
}
}

You can see that there are four pool members as the item count is four, but I’m only showing one of them here for brevity. The pool members can be added, modified, or deleted at this level.

• Add a pool member
  • URI: https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members
  • Method: POST 
  • JSON: {“name”:”192.168.103.12:80”}
• Modify a pool member
  • URI: https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members/~Common~192.168.103.12:80 
  • Method: PUT 
  • JSON: {“name”:”192.168.103.12:80”,”connectionLimit”:”50”}
• Delete a pool member
  • URI: https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members/~Common~192.168.103.12:80
  • Method: DELETE
  • JSON: none (an error will trigger if you send any data)

So for subcollections like pool members, adding and deleting is pretty straight forward. Unfortunately, not all lists of configuration items are treated as subcollections. For example, take the data group. You can see for the data group testdb below, the records are not a subcollection.

{
kind: "tm:ltm:data-group:internal:internalstate"
name: "testdb"
fullPath: "testdb"
generation: 1
selfLink: "https://localhost/mgmt/tm/ltm/data-group/internal/testdb?ver=11.6.0"
type: "string"
records: [3]
0:  {
name: "a"
data: "one"
}-
1:  {
name: "b"
data: "two"
}-
2:  {
name: "c"
}-
-
}

Because this is not a subcollection, any modifications to the records of this object are treated as complete replacements. Thus, this request to add a record (d) to the list:

• URL: https://172.16.44.128/mgmt/tm/ltm/data-group/internal/testdb
• Method: PUT
• JSON: {“records” : [ { “name”: “d” } ] }

will result in a data group with only 1 entry (d)! If one wanted to add (d) to the data group, one would have to issue the same request above, but with complete JSON data representing the original records PLUS the new record. The same goes if you want to delete a record. You need to submit all the records in JSON format sans the one you wish to delete via the PUT request above. 

Note: Whereas it's true an update to the records attribute requires a full replacement, it IS possible to update individual records by using the options query parameter instead of updating the records attribute. For details, see the update section of this article.

Thus, to modify items that are not subcollections, one would have to issue a get and parse the existing items into a list. Then one would need to modify the list as desired (adding and/or deleting items), and then issue a PUT to the object URI with the modified list as the json data. A python example of doing just that is shown below. This script grabs the records for the MyNetworks data group, adds three new networks to it, then removes those three networks to return it to its original state.

__author__ = 'rahm'

def get_dg(rq, url, dg_details):
    dg = rq.get('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1])).json()
    return dg

def extend_dg(rq, url, dg_details, additional_records):
    dg = rq.get('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1])).json()

    current_records = dg['records']
    new_records = []
    for record in current_records:
        nr = [ {'name': record['name']}]
        new_records.extend(nr)
    for record in additional_records:
        nr = [ {'name': record}]
        new_records.extend(nr)

    payload = {}
    payload['records'] = new_records
    rq.put('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1]), json.dumps(payload))

def contract_dg(rq, url, dg_details, removal_records):
    dg = rq.get('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1])).json()

    new_records = []
    for record in removal_records:
        nr = [ {'name': record}]
        new_records.extend(nr)

    current_records = dg['records']
    new_records = [x for x in current_records if x not in new_records]

    payload = {}
    payload['records'] = new_records
    rq.put('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1]), json.dumps(payload))

if __name__ == "__main__":
    import requests, json

    b = requests.session()
    b.auth = ('admin', 'admin')
    b.verify = False
    b.headers.update({'Content-Type' : 'application/json'})

    b_url_base = 'https://172.16.44.128/mgmt/tm'

    dg_details = ['internal', 'myNetworks']
    net_changes = ['3.0.0.0/8', '4.0.0.0/8']

    print "\nExisting Records for %s Data-Group:\n\t%s" % (dg_details[1], get_dg(b, b_url_base, dg_details)['records'])
    extend_dg(b, b_url_base, dg_details, net_changes)
    print "\nUpdated Records for %s Data-Group:\n\t%s" % (dg_details[1], get_dg(b, b_url_base, dg_details)['records'])
    contract_dg(b, b_url_base, dg_details, net_changes)
    print "\nUpdated Records for %s Data-Group:\n\t%s" % (dg_details[1], get_dg(b, b_url_base, dg_details)['records'])

When running this against my lab BIG-IP, I get this output on my console

Existing Records for myNetworks Data-Group:
[{u'name': u'1.0.0.0/8'}, {u'name': u'2.0.0.0/8'}]

Updated Records for myNetworks Data-Group:
[{u'name': u'1.0.0.0/8'}, {u'name': u'2.0.0.0/8'}, {u'name': u'3.0.0.0/8'}, {u'name': u'4.0.0.0/8'}]

Updated Records for myNetworks Data-Group:
[{u'name': u'1.0.0.0/8'}, {u'name': u'2.0.0.0/8'}]

Process finished with exit code 0

Hopefully this has been helpful in showing the power of subcollections and the necessary steps to update objects like data-groups that are not. Much thanks again to Pat Chang for the bulk of the content in this article Next up: Query Parameters and Options.