Demystifying iControl REST Part 2 - Understanding sub collections and how to use them

iControl REST. It’s iControl SOAP’s baby brother, introduced back in TMOS version 11.4 as an early access feature but was released fully in version 11.5.

Several articles on basic usage have been written on iControl REST (see the resources at the bottom of this article) so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. The first article of this series covered the URI’s role in the API. This second article will cover how the URI path plays a role in how the API functions.

Working with Subcollections

When manipulating F5 configuration items with iControl Rest, subcollections are a powerful tool.  They allow one to manipulate specific items in the subcollection instead of having to manipulate the entire sub collection. Take the pool object for example. If we just query the pool (in this case testpool,) you’ll notice the returned data does not list the pool members

#query (via the Chrome advanced REST client using Authorization & Content-Type headers:)
https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool
#query (via curl:)
curl -k -u admin:admin https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool

#response:
{
kind: "tm:ltm:pool:poolstate"
name: "testpool"
fullPath: "testpool"
generation: 1
selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool?ver=11.6.0"
allowNat: "yes"
allowSnat: "yes"
ignorePersistedWeight: "disabled"
ipTosToClient: "pass-through"
ipTosToServer: "pass-through"
linkQosToClient: "pass-through"
linkQosToServer: "pass-through"
loadBalancingMode: "round-robin"
minActiveMembers: 0
minUpMembers: 0
minUpMembersAction: "failover"
minUpMembersChecking: "disabled"
queueDepthLimit: 0
queueOnConnectionLimit: "disabled"
queueTimeLimit: 0
reselectTries: 0
serviceDownAction: "none"
slowRampTime: 10
membersReference: {
link: "https://localhost/mgmt/tm/ltm/pool/~Common~testpool/members?ver=11.6.0"
isSubcollection: true
}-
}

Notice the isSubcollection: true for the membersReference? This is an indicator that there is a subcollection for the members keyword. If you then query the members for that pool, you will get the subcollection.

#query (via the Chrome advanced REST client using Authorization & Content-Type headers:)
https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members
#query (via curl:)
curl -k -u admin:admin https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members

#response:
{
kind: "tm:ltm:pool:members:memberscollectionstate"
selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool/members?ver=11.6.0"
items: [4]
0:  {
kind: "tm:ltm:pool:members:membersstate"
name: "192.168.103.10:80"
partition: "Common"
fullPath: "/Common/192.168.103.10:80"
generation: 1
selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool/members/~Common~192.168.103.10:80?ver=11.6.0"
address: "192.168.103.10"
connectionLimit: 0
dynamicRatio: 1
ephemeral: "false"
fqdn: {
autopopulate: "disabled"
}-
inheritProfile: "enabled"
logging: "disabled"
monitor: "default"
priorityGroup: 0
rateLimit: "disabled"
ratio: 1
session: "user-enabled"
state: "unchecked"
}
}

You can see that there are four pool members as the item count is four, but I’m only showing one of them here for brevity. The pool members can be added, modified, or deleted at this level.

  • Add a pool member
    • URI: https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members
    • Method: POST 
    • JSON: {“name”:”192.168.103.12:80”}
  • Modify a pool member
    • URI: https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members/~Common~192.168.103.12:80 
    • Method: PUT 
    • JSON: {“name”:”192.168.103.12:80”,”connectionLimit”:”50”}
  • Delete a pool member
    • URI: https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members/~Common~192.168.103.12:80
    • Method: DELETE
    • JSON: none (an error will trigger if you send any data)

So for subcollections like pool members, adding and deleting is pretty straight forward. Unfortunately, not all lists of configuration items are treated as subcollections. For example, take the data group. You can see for the data group testdb below, the records are not a subcollection.

{
kind: "tm:ltm:data-group:internal:internalstate"
name: "testdb"
fullPath: "testdb"
generation: 1
selfLink: "https://localhost/mgmt/tm/ltm/data-group/internal/testdb?ver=11.6.0"
type: "string"
records: [3]
0:  {
name: "a"
data: "one"
}-
1:  {
name: "b"
data: "two"
}-
2:  {
name: "c"
}-
-
}

Because this is not a subcollection, any modifications to the records of this object are treated as complete replacements. Thus, this request to add a record (d) to the list:

  • URL: https://172.16.44.128/mgmt/tm/ltm/data-group/internal/testdb
  • Method: PUT
  • JSON: {“records” : [ { “name”: “d” } ] }

will result in a data group with only 1 entry (d)! If one wanted to add (d) to the data group, one would have to issue the same request above, but with complete JSON data representing the original records PLUS the new record. The same goes if you want to delete a record. You need to submit all the records in JSON format sans the one you wish to delete via the PUT request above. 

Note: Whereas it's true an update to the records attribute requires a full replacement, it IS possible to update individual records by using the options query parameter instead of updating the records attribute. For details, see the update section of this article.

Thus, to modify items that are not subcollections, one would have to issue a get and parse the existing items into a list. Then one would need to modify the list as desired (adding and/or deleting items), and then issue a PUT to the object URI with the modified list as the json data. A python example of doing just that is shown below. This script grabs the records for the MyNetworks data group, adds three new networks to it, then removes those three networks to return it to its original state.

__author__ = 'rahm'

def get_dg(rq, url, dg_details):
    dg = rq.get('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1])).json()
    return dg

def extend_dg(rq, url, dg_details, additional_records):
    dg = rq.get('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1])).json()

    current_records = dg['records']
    new_records = []
    for record in current_records:
        nr = [ {'name': record['name']}]
        new_records.extend(nr)
    for record in additional_records:
        nr = [ {'name': record}]
        new_records.extend(nr)

    payload = {}
    payload['records'] = new_records
    rq.put('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1]), json.dumps(payload))

def contract_dg(rq, url, dg_details, removal_records):
    dg = rq.get('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1])).json()

    new_records = []
    for record in removal_records:
        nr = [ {'name': record}]
        new_records.extend(nr)

    current_records = dg['records']
    new_records = [x for x in current_records if x not in new_records]

    payload = {}
    payload['records'] = new_records
    rq.put('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1]), json.dumps(payload))

if __name__ == "__main__":
    import requests, json

    b = requests.session()
    b.auth = ('admin', 'admin')
    b.verify = False
    b.headers.update({'Content-Type' : 'application/json'})

    b_url_base = 'https://172.16.44.128/mgmt/tm'

    dg_details = ['internal', 'myNetworks']
    net_changes = ['3.0.0.0/8', '4.0.0.0/8']

    print "\nExisting Records for %s Data-Group:\n\t%s" % (dg_details[1], get_dg(b, b_url_base, dg_details)['records'])
    extend_dg(b, b_url_base, dg_details, net_changes)
    print "\nUpdated Records for %s Data-Group:\n\t%s" % (dg_details[1], get_dg(b, b_url_base, dg_details)['records'])
    contract_dg(b, b_url_base, dg_details, net_changes)
    print "\nUpdated Records for %s Data-Group:\n\t%s" % (dg_details[1], get_dg(b, b_url_base, dg_details)['records'])

When running this against my lab BIG-IP, I get this output on my console

Existing Records for myNetworks Data-Group:
[{u'name': u'1.0.0.0/8'}, {u'name': u'2.0.0.0/8'}]

Updated Records for myNetworks Data-Group:
[{u'name': u'1.0.0.0/8'}, {u'name': u'2.0.0.0/8'}, {u'name': u'3.0.0.0/8'}, {u'name': u'4.0.0.0/8'}]

Updated Records for myNetworks Data-Group:
[{u'name': u'1.0.0.0/8'}, {u'name': u'2.0.0.0/8'}]

Process finished with exit code 0

Hopefully this has been helpful in showing the power of subcollections and the necessary steps to update objects like data-groups that are not. Much thanks again to Pat Chang for the bulk of the content in this article Next up: Query Parameters and Options.

Published Jun 23, 2015
Version 1.0
devops
iControlREST
python
series-demystifying-icontrol-rest

Was this article helpful?

5 Comments

Sort By
  • someguy_126006's avatar
    someguy_126006
    Icon for Nimbostratus rankNimbostratus
    Jan 14, 2016
    Are datagroup updates atomic? ie, if my virtual is dependent on datagroup values to route traffic based on URI and I update via REST, will that work seamlessly under high traffic or will some requests route incorrectly as it updates?
  • Mike_Mckinney's avatar
    Mike_Mckinney
    Icon for Nimbostratus rankNimbostratus
    Nov 08, 2018

    I am trying to find all pool members with a certain IP address. Am I going to have to basically execute "mgmt/tm/ltm/pool/poolname/members" for each individual pool or is there a way to wild card poolname? I was hoping to make one call to get all and then search the entire response. Basically the REST equivalent of "tmsh list ltm pool one-line | grep 1.1.1.1"

     

  • JRahm's avatar
    JRahm
    Icon for Admin rankAdmin
    Nov 08, 2018

    you can use mgmt/tm/ltm/pool?expandSubcollections=true and you'll get all pool configurations with pool members that you can then parse locally.

     

  • rahvee's avatar
    rahvee
    Icon for Altocumulus rankAltocumulus
    Mar 07, 2023

    Thanks for this, but I have a problem:

    I query a virtual server

    virtual_server = f5_session.load("/mgmt/tm/ltm/virtual")[0]

    It has `profilesReference`, which is a subcollection, and I'm able to load it from the link (remove 'https://localhost' from the beginning). Now I've got a list of the profiles on that virtual server.

    I need to remove a profile which has "default for SNI" enabled, and add a different profile that has "default for SNI." But due to the nature of SNI, there must be exactly one. If I try to do the DELETE first, it fails because one profile is needed to have "default for SNI." If I try to do the ADD first, it fails because only one can have "default for SNI." I need to have a way of simultaneously adding & deleting, in a single operation.