Demystifying iControl REST Part 7 - Understanding Transactions
iControl REST. It’s iControl SOAP’s baby, brother, introduced back in TMOS version 11.4 as an early access feature but released fully in version 11.5. Several articles on basic usage have been written about the rest interface so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. A few months ago, a question in Q&A from community member spirrello asking how to update a tcp profile on a virtual. He was using bigsuds, the python wrapper for the soap interface. For the rest interface on this particular object, this is easy; just use the put method and supply the payload mapping the updated profile. But for soap, this requires a transaction. There are some changes to BIG-IP via the rest interface, however, like updating an ssl cert or key, that likewise will require a transaction to accomplish. In this article, I’ll show you how to use transactions with the rest interface. The Fine Print From the iControl REST user guide, the life cycle of a transaction progresses through three phases: Creation - This phase occurs when the transaction is created using a POST command. Modification - This phase occurs when commands are added to the transaction, or changes are made to the sequence of commands in the transaction. Commit - This phase occurs when iControl REST runs the transaction. To create a transaction, post to /tm/transaction POST https://192.168.25.42/mgmt/tm/transaction {} Response: { "transId":1389812351, "state":"STARTED", "timeoutSeconds":30, "kind":"tm:transactionstate", "selfLink":"https://localhost/mgmt/tm/transaction/1389812351?ver=11.5.0" } Note the transId, the state, and the timeoutSeconds. You'll need the transId to add or re-sequence commands within the transaction, and the transaction will expire after 30 seconds if no commands are added. You can list all transactions, or the details of a specific transaction with a get request. GET https://192.168.25.42/mgmt/tm/transaction GET https://192.168.25.42/mgmt/tm/transaction/transId To add a command to the transaction, you use the normal method uris, but include the X-F5-REST-Coordination-Id header. This example creates a pool with a single member. POST https://192.168.25.42/mgmt/tm/ltm/pool X-F5-REST-Coordination-Id:1389812351 { "name":"tcb-xact-pool", "members": [ {"name":"192.168.25.32:80","description":"First pool for transactions"} ] } Not a great example because there is no need for a transaction here, but we'll roll with it! There are several other option methods for interrogating the transaction itself, see the user guide for details. Now we can commit the transaction. To do that, you reference the transaction id in the URI, remove the X-F5-REST-Coordination-Id header and use the patch method with payload key/value state: VALIDATING . PATCH https://localhost/mgmt/tm/transaction/1389812351 { "state":"VALIDATING" } That's all there is to it! Now that you've seen the nitty gritty details, let's take a look at some code samples. Roll Your Own In this example, I am needing to update and ssl key and certificate. If you try to update the cert or the key, it will complain that they do not match, so you need to update both at the same time. Assuming you are writing all your code from scratch, this is all it takes in python. Note on line 21 I post with an empty payload, and then on line 23, I add the header with the transaction id. I make my modifications and then in line 31, I remove the header, and finally on line 32, I patch to the transaction id with the appropriate payload. import json import requests btx = requests.session() btx.auth = (f5_user, f5_password) btx.verify = False btx.headers.update({'Content-Type':'application/json'}) urlb = 'https://{0}/mgmt/tm'.format(f5_host) domain = 'mydomain.local_sslobj' chain = 'mychain_sslobj try: key = btx.get('{0}/sys/file/ssl-key/~Common~{1}'.format(urlb, domain)) cert = btx.get('{0}/sys/file/ssl-cert/~Common~{1}'.format(urlb, domain)) chain = btx.get('{0}/sys/file/ssl-cert/~Common~{1}'.format(urlb, 'chain')) if (key.status_code == 200) and (cert.status_code == 200) and (chain.status_code == 200): # use a transaction txid = btx.post('{0}/transaction'.format(urlb), json.dumps({})).json()['transId'] # set the X-F5-REST-Coordination-Id header with the transaction id btx.headers.update({'X-F5-REST-Coordination-Id': txid}) # make modifications modkey = btx.put('{0}/sys/file/ssl-key/~Common~{1}'.format(urlb, domain), json.dumps(keyparams)) modcert = btx.put('{0}/sys/file/ssl-cert/~Common~{1}'.format(urlb, domain), json.dumps(certparams)) modchain = btx.put('{0}/sys/file/ssl-cert/~Common~{1}'.format(urlb, 'le-chain'), json.dumps(chainparams)) # remove header and patch to commit the transaction del btx.headers['X-F5-REST-Coordination-Id'] cresult = btx.patch('{0}/transaction/{1}'.format(urlb, txid), json.dumps({'state':'VALIDATING'})).json() A Little Help from a Friend The f5-common-python library was released a few months ago to relieve you of a lot of the busy work with building requests. This is great, especially for transactions. To simplify the above code just to the transaction steps, consider: # use a transaction txid = btx.post('{0}/transaction'.format(urlb), json.dumps({})).json()['transId'] # set the X-F5-REST-Coordination-Id header with the transaction id btx.headers.update({'X-F5-REST-Coordination-Id': txid}) # do stuff here # remove header and patch to commit the transaction del btx.headers['X-F5-REST-Coordination-Id'] cresult = btx.patch('{0}/transaction/{1}'.format(urlb, txid), json.dumps({'state':'VALIDATING'})).json() With the library, it's simplified to: tx = b.tm.transactions.transaction with TransactionContextManager(tx) as api: # do stuff here api.do_stuff Yep, it's that simple. So if you haven't checked out the f5-common-python library, I highly suggest you do! I'll be writing about how to get started using it next week, and perhaps a follow up on how to contribute to it as well, so stay tuned!Demystifying iControl REST Part 6: Token-Based Authentication
iControl REST. It’s iControl SOAP’s baby, brother, introduced back in TMOS version 11.4 as an early access feature but released fully in version 11.5. Several articles on basic usage have been written on iControl REST so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. This article will cover the details on how to retrieve and use an authentication token from the BIG-IP using iControl REST and the python programming language. This token is used in place of basic authentication on API calls, which is a requirement for external authentication. Note that for configuration changes, version 12.0 or higher is required as earlier versions will trigger an un-authorized error. The tacacs config in this article is dependent on a version that I am no longer able to get installed on a modern linux flavor. Instead, try this Dockerized tacacs+ server for your testing. The Fine Print The details of the token provider are here in the wiki. We’ll focus on a provider not listed there: tmos. This provider instructs the API interface to use the provider that is configured in tmos. For this article, I’ve configured a tacacs server and the BIG-IP with custom remote roles as shown below to show BIG-IP version 12’s iControl REST support for remote authentication and authorization. Details for how this configuration works can be found in the tacacs+ article I wrote a while back. BIG-IP tacacs+ configuration auth remote-role { role-info { adm { attribute F5-LTM-User-Info-1=adm console %F5-LTM-User-Console line-order 1 role %F5-LTM-User-Role user-partition %F5-LTM-User-Partition } mgr { attribute F5-LTM-User-Info-1=mgr console %F5-LTM-User-Console line-order 2 role %F5-LTM-User-Role user-partition %F5-LTM-User-Partition } } } auth remote-user { } auth source { type tacacs } auth tacacs system-auth { debug enabled protocol ip secret $M$Zq$T2SNeIqxi29CAfShLLqw8Q== servers { 172.16.44.20 } service ppp } Tacacs+ Server configuration id = tac_plus { debug = PACKET AUTHEN AUTHOR access log = /var/log/access.log accounting log = /var/log/acct.log host = world { address = ::/0 prompt = "\nAuthorized Access Only!\nTACACS+ Login\n" key = devcentral } group = adm { service = ppp { protocol = ip { set F5-LTM-User-Info-1 = adm set F5-LTM-User-Console = 1 set F5-LTM-User-Role = 0 set F5-LTM-User-Partition = all } } } group = mgr { service = ppp { protocol = ip { set F5-LTM-User-Info-1 = mgr set F5-LTM-User-Console = 1 set F5-LTM-User-Role = 100 set F5-LTM-User-Partition = all } } } user = user_admin { password = clear letmein00 member = adm } user = user_mgr { password = clear letmein00 member = mgr } } Basic Requirements Before we look at code, however, let’s take a look at the json payload requirements, followed by response data from a query using Chrome’s Advanced REST Client plugin. First, since we are sending json payload, we need to add the Content-Type: application/json header to the query. The payload we are sending with the post looks like this: { "username": "remote_auth_user", "password": "remote_auth_password", "loginProviderName": "tmos" } You submit the same remote authentication credentials in the initial basic authentication as well, no need to have access to the default admin account credentials. A successful query for a token returns data like this: { username: "user_admin" loginReference: { link: "https://localhost/mgmt/cm/system/authn/providers/tmos/1f44a60e-11a7-3c51-a49f-82983026b41b/login" }- token: { uuid: "4d1bd79f-dca7-406b-8627-3ad262628f31" name: "5C0F982A0BF37CBE5DE2CB8313102A494A4759E5704371B77D7E35ADBE4AAC33184EB3C5117D94FAFA054B7DB7F02539F6550F8D4FA25C4BFF1145287E93F70D" token: "5C0F982A0BF37CBE5DE2CB8313102A494A4759E5704371B77D7E35ADBE4AAC33184EB3C5117D94FAFA054B7DB7F02539F6550F8D4FA25C4BFF1145287E93F70D" userName: "user_admin" user: { link: "https://localhost/mgmt/cm/system/authn/providers/tmos/1f44a60e-11a7-3c51-a49f-82983026b41b/users/34ba3932-bfa3-4738-9d55-c81a1c783619" }- groupReferences: [1] 0: { link: "https://localhost/mgmt/cm/system/authn/providers/tmos/1f44a60e-11a7-3c51-a49f-82983026b41b/user-groups/21232f29-7a57-35a7-8389-4a0e4a801fc3" }- - timeout: 1200 startTime: "2015-11-17T19:38:50.415-0800" address: "172.16.44.1" partition: "[All]" generation: 1 lastUpdateMicros: 1447817930414518 expirationMicros: 1447819130415000 kind: "shared:authz:tokens:authtokenitemstate" selfLink: "https://localhost/mgmt/shared/authz/tokens/4d1bd79f-dca7-406b-8627-3ad262628f31" }- generation: 0 lastUpdateMicros: 0 } Among many other fields, you can see the token field with a very long hexadecimal token. That’s what we need to authenticate future API calls. Requesting the token programmatically In order to request the token, you first have to supply basic auth credentials like normal. This is currently required to access the /mgmt/shared/authn/login API location. The basic workflow is as follows (with line numbers from the code below in parentheses): Make a POST request to BIG-IP with basic authentication header and json payload with username, password, and the login provider (9-16, 41-47) Remove the basic authentication (49) Add the token from the post response to the X-F5-Auth-Token header (50) Continue further requests like normal. In this example, we’ll create a pool to verify read/write privileges. (1-6, 52-53) And here’s the code (in python) to make that happen: def create_pool(bigip, url, pool): payload = {} payload['name'] = pool pool_config = bigip.post(url, json.dumps(payload)).json() return pool_config def get_token(bigip, url, creds): payload = {} payload['username'] = creds[0] payload['password'] = creds[1] payload['loginProviderName'] = 'tmos' token = bigip.post(url, json.dumps(payload)).json()['token']['token'] return token if __name__ == "__main__": import os, requests, json, argparse, getpass requests.packages.urllib3.disable_warnings() parser = argparse.ArgumentParser(description='Remote Authentication Test - Create Pool') parser.add_argument("host", help='BIG-IP IP or Hostname', ) parser.add_argument("username", help='BIG-IP Username') parser.add_argument("poolname", help='Key/Cert file names (include the path.)') args = vars(parser.parse_args()) hostname = args['host'] username = args['username'] poolname = args['poolname'] print "%s, enter your password: " % args['username'], password = getpass.getpass() url_base = 'https://%s/mgmt' % hostname url_auth = '%s/shared/authn/login' % url_base url_pool = '%s/tm/ltm/pool' % url_base b = requests.session() b.headers.update({'Content-Type':'application/json'}) b.auth = (username, password) b.verify = False token = get_token(b, url_auth, (username, password)) print '\nToken: %s\n' % token b.auth = None b.headers.update({'X-F5-Auth-Token': token}) response = create_pool(b, url_pool, poolname) print '\nNew Pool: %s\n' % response Running this script from the command line, we get the following response: FLD-ML-RAHM:scripts rahm$ python remoteauth.py 172.16.44.15 user_admin myNewestPool1 Password: user_admin, enter your password: Token: 2C61FE257C7A8B6E49C74864240E8C3D3592FDE9DA3007618CE11915F1183BF9FBAF00D09F61DE15FCE9CAB2DC2ACC165CBA3721362014807A9BF4DEA90BB09F New Pool: {u'generation': 453, u'minActiveMembers': 0, u'ipTosToServer': u'pass-through', u'loadBalancingMode': u'round-robin', u'allowNat': u'yes', u'queueDepthLimit': 0, u'membersReference': {u'isSubcollection': True, u'link': u'https://localhost/mgmt/tm/ltm/pool/~Common~myNewestPool1/members?ver=12.0.0'}, u'minUpMembers': 0, u'slowRampTime': 10, u'minUpMembersAction': u'failover', u'minUpMembersChecking': u'disabled', u'queueTimeLimit': 0, u'linkQosToServer': u'pass-through', u'queueOnConnectionLimit': u'disabled', u'fullPath': u'myNewestPool1', u'kind': u'tm:ltm:pool:poolstate', u'name': u'myNewestPool1', u'allowSnat': u'yes', u'ipTosToClient': u'pass-through', u'reselectTries': 0, u'selfLink': u'https://localhost/mgmt/tm/ltm/pool/myNewestPool1?ver=12.0.0', u'serviceDownAction': u'none', u'ignorePersistedWeight': u'disabled', u'linkQosToClient': u'pass-through'} You can test this out in the Chrome Advanced Rest Client plugin, or from the command line with curl or any other language supporting REST clients as well, I just use python for the examples well, because I like it. I hope you all are digging into iControl REST! What questions do you have? What else would you like clarity on? Drop a comment below.Demystifying iControl REST Part 1 - Understanding the request URI
iControl REST. It’s iControl SOAP’s baby brother, introduced back in TMOS version 11.4 as an early access feature but was released fully in version 11.5. Several articles on basic usage have been written on iControl REST (see the resources at the bottom of this article) so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. This article is the first of a four part series and will cover how the URI path plays a role in how the API functions. Examining the REST interface URI With iControl SOAP, all the interfaces and methods are defined in the WSDLs. With REST, the bulk of the interface and method is part of the URI. Take for example a request to get the list of iRules on a BIG-IP. https://x.x.x.x/mgmt/tm/ltm/rule With a simple get request with the appropriate headers and credentials (see below for curl and python examples,) this breaks down to the Base URI, the Module URI, and the Sub-Module URI. #python example >>> import requests >>> import json >>> b = requests.session() >>> b.auth = ('admin', 'admin') >>> b.verify = False >>> b.headers.update({'Content-Type':'application/json'}) >>> b.get('https://172.16.44.128/mgmt/tm/ltm/rule') #curl example curl -k -u admin:admin -H “Content-Type: application/json” -X GET https://172.16.44.128/mgmt/tm/ltm/rule Beyond the Sub-Module URI is the component URI. https://x.x.x.x/mgmt/tm/ltm/rule/testlog The component URI is the spot that snags most beginners. When creating a new configuration object on an F5, it is fairly obvious which URI to use for the REST call. If you were creating a new virtual server, it would be /mgmt/tm/ltm/virtual, while a new pool would be /mgmt/tm/ltm/pool. Thus a REST call to create a new pool on an LTM with the IP address 172.16.44.128 would look like the following: #python example >>> import requests >>> import json >>> b = requests.session() >>> b.auth = ('admin', 'admin') >>> b.verify = False >>> b.headers.update({'Content-Type':'application/json'}) >>> pm = ['192.168.25.32:80', '192.168.25.33:80'] >>> payload = { } >>> payload['kind'] = 'tm:ltm:pool:poolstate' >>> payload['name'] = 'tcb-pool' >>> payload['members'] = [ {'kind': 'ltm:pool:members', 'name': member } for member in pm] >>> b.post('https://172.16.44.128/mgmt/tm/ltm/pool', data=json.dumps(payload)) #curl example curl -k -u admin:admin -H "Content-Type: \ application/json" -X POST -d \ '{"name":"tcb-pool","members":[ \ {"name":"192.168.25.32:80","description":"first member”}, \ {"name":"192.168.25.33:80","description":"second member”} ] }' \ https://172.16.44.128/mgmt/tm/ltm/pool The call would create the pool “tcb-pool” with members at 192.168.25.32:80 and 192.168.25.33:80. All other aspects of the pool would be the defaults. Thus this pool would be created in the Common partition, have the Round Robin load balancing method, and no monitor.At first glance, some programmers would then modify the pool “tcb-pool” with a command like the following (same payload in python, added the .text attribute to see the error response on the requests object): #python example >>> b.put('https://172.16.44.128/mgmt/tm/ltm/pool', data=json.dumps(payload)).text #return data u'{"code":403,"message":"Operation is not supported on component /ltm/pool.","errorStack":[]}' #curl example curl -k -u admin:admin -H "Content-Type: \ application/json" -X PUT -d \ '{"name":"tcb-pool","members":[ \ {"name":"192.168.25.32:80","description":"first member"} {"name":"192.168.25.33:80","description":"second member"} ] }' \ https://172.16.44.128/mgmt/tm/ltm/pool #return data {"code":403,"message":"Operation is not supported on component /ltm/pool.","errorStack":[]} You can see because they use the sub module URI used to create the pool this returns a 403 error. The command fails because one is trying to modify a specific pool at the generic pool level. Now that the pool exists, one must use the URI that specifies the pool. Thus, the correct command would be: #python example >>> b.put('https://172.16.44.128/mgmt/tm/ltm/pool/~Common~tcb-pool', data=json.dumps(payload)) #curl example curl -k -u admin:admin -H "Content-Type: \ application/json" -X PUT -d \ '{"members":[ \ {"name":"192.168.25.32:80","description":"first member"} {"name":"192.168.25.33:80","description":"second member"} ] }' \ https://172.16.44.128/mgmt/tm/ltm/pool/~Common~tcb-pool We add the ~Common~ in front of the pool name because it is in the Common partition. However, this would also work with https://172.16.44.128/mgmt/tm/ltm/pool/tcb-pool. It is just good practice to explicitly insert the partition name since not all configuration objects will be in the default Common partition. Because we specify the pool in the URI, it is no longer necessary to have the “name” key value pair. In practice, programmers usually correctly modify items such as virtual servers and pools. However, we encounter this confusion much more often in configuration items that are ifiles. This may be because the creation of configuration items that are ifiles is a 3-step process. For instance, in order to create an external data group, one would first scp the file to 172.16.44.128/config/filestore/data_mda_1, then issue 2 Rest commands: curl -sk -u admin:admin -H "Content-Type: application/json" -X POST -d '{"name":"data_mda_1","type":"string","source-path":"file:///config/filestore/data_mda_1"}' https://172.16.44.128/mgmt/tm/sys/file/data-group curl -sk -u admin:admin -H "Content-Type: application/json" -X POST -d '{"name":"dg_mda","external-file-name":"/Common/data_mda_1"}' https://172.16.44.128/mgmt/tm/ltm/data-group/external/ To update the external data group, many programmers first try something like the following: curl -sk -u admin:admin -H "Content-Type: application/json" -X POST -d '{"name":"data_mda_2","type":"string","source-path":"file:///config/filestore/data_mda_2”}’ https://172.16.44.128/mgmt/tm/sys/file/data-group curl -sk -u admin:admin -H "Content-Type: application/json" -X PUT -d '{"name":"dg_mda","external-file-name":"/Common/data_mda_2"}' https://172.16.44.128/mgmt/tm/ltm/data-group/external/ The first command works because we are creating a new ifile object. However, the second command fails because we are trying to modify a specific external data group at the generic external data group level. The proper command is: curl -sk -u admin:admin -H "Content-Type: application/json" -X PUT -d '{"external-file-name":"/Common/data_mda_2"}' https://172.16.44.128/mgmt/tm/ltm/data-group/external/dg_mda The python code gets a little more complex with the data-group examples, so I've uploaded it to the codeshare here. Much thanks to Pat Chang for the bulk of the content in this article. Stay tuned for part 2, where we'll cover sub collections and how to use them.Demystifying iControl REST Part 5: Transferring Files
iControl REST. It’s iControl SOAP’s baby, brother, introduced back in TMOS version 11.4 as an early access feature but released fully in version 11.5. Several articles on basic usage have been written on iControl REST so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. This article will cover the details on how to transfer files to/from the BIG-IP using iControl REST and the python programming language. (Note: this functionality requires 12.0+.) The REST File Transfer Worker The file transfer worker allows a client to transfer files through a series of GET operations for downloads and POST operations for uploads. The Content-Range header is used for both as a means to chunk the content. For downloads, the worker listens onthe following interfaces. Description Method URI File Location Download a File GET /mgmt/cm/autodeploy/software-image-downloads/ /shared/images/ Upload an Image File POST /mgmt/cm/autodeploy/software-image-uploads/ /shared/images/ Upload a File POST /mgmt/shared/file-transfer/uploads/ /var/config/rest/downloads/ Download a QKView GET /mgmt/shared/file-transfer/qkview-downloads/ /var/tmp/ Download a UCS GET /mgmt/shared/file-transfer/ucs-downloads/ /var/local/ucs/ Upload ASM Policy POST /mgmt/tm/asm/file-transfer/uploads/ /var/ts/var/rest/ Download ASM Policy GET /mgmt/tm/asm/file-transfer/downloads/ /var/ts/var/rest/ Binary and text files are supported. The magic in the transfer is the Content-Range header, which has the following format: Content-Range: start-end/filesize Where start/end are the chunk's delimiters in the file and filesize is well, the file size. Any file larger than 1M needs to be chunked with this header as that limit is enforced by the worker. This is done to avoid potential denial of service attacks and out of memory errors. There are benefits of chunking as well: Accurate progress bars Resuming interrupted downloads Random access to file content possible Uploading a File The function is shown below. Note that whereas normally with the REST API the Content-Type is application/json, with file transfers that changes to application/octet-stream. The workflow for the function works like this (line number in parentheses) : Set the Chunk Size (3) Set the Content-Type header (4-6) Open the file (7) Get the filename (apart from the path) from the absolute path (8) If the extension is an .iso file (image) put it in /shared/images, otherwise it’ll go in /var/config/rest/downloads (9-12) Disable ssl warnings requests (required with my version: 2.8.1. YMMV) (14) Set the total file size for use with the Content-Range header (15) Set the start variable to 0 (17) Begin loop to iterate through the file and upload in chunks (19) Read data from the file and if there is no more data, break the loop (20-22) set the current bytes read, if less than the chunk size, then this is the last chunk, so set the end to the size from step 7. Otherwise, add current bytes length to the start value and set that as the end. (24-28) Set the Content-Range header value and then add that to the header (30-31) Make the POST request, uploading the content chunk (32-36) Increment the start value by the current bytes content length (38) def _upload(host, creds, fp): chunk_size = 512 * 1024 headers = { 'Content-Type': 'application/octet-stream' } fileobj = open(fp, 'rb') filename = os.path.basename(fp) if os.path.splitext(filename)[-1] == '.iso': uri = 'https://%s/mgmt/cm/autodeploy/software-image-uploads/%s' % (host, filename) else: uri = 'https://%s/mgmt/shared/file-transfer/uploads/%s' % (host, filename) requests.packages.urllib3.disable_warnings() size = os.path.getsize(fp) start = 0 while True: file_slice = fileobj.read(chunk_size) if not file_slice: break current_bytes = len(file_slice) if current_bytes < chunk_size: end = size else: end = start + current_bytes content_range = "%s-%s/%s" % (start, end - 1, size) headers['Content-Range'] = content_range requests.post(uri, auth=creds, data=file_slice, headers=headers, verify=False) start += current_bytes Downloading a File Downloading is very similar but there are some differences. Here is the workflow that is different, followed by the code. Note that the local path where the file will be downloaded to is given as part of the filename. URI is set to downloads worker. The only supported download directory at this time is /shared/images. (8) Open the local file so received data can be written to it (11) Make the request (22-26) If response code is 200 and if size is greater than 0, increment the current bytes and write the data to file, otherwise exit the loop (28-40) Set the value of the returned Content-Range header to crange and if initial size (0), set the file size to the size variable (42-46) If the file is smaller than the chunk size, adjust the chunk size down to the total file size and continue (51-55) Do the math to get ready to download the next chunk (57-62) def _download(host, creds, fp): chunk_size = 512 * 1024 headers = { 'Content-Type': 'application/octet-stream' } filename = os.path.basename(fp) uri = 'https://%s/mgmt/cm/autodeploy/software-image-downloads/%s' % (host, filename) requests.packages.urllib3.disable_warnings() with open(fp, 'wb') as f: start = 0 end = chunk_size - 1 size = 0 current_bytes = 0 while True: content_range = "%s-%s/%s" % (start, end, size) headers['Content-Range'] = content_range #print headers resp = requests.get(uri, auth=creds, headers=headers, verify=False, stream=True) if resp.status_code == 200: # If the size is zero, then this is the first time through the # loop and we don't want to write data because we haven't yet # figured out the total size of the file. if size > 0: current_bytes += chunk_size for chunk in resp.iter_content(chunk_size): f.write(chunk) # Once we've downloaded the entire file, we can break out of # the loop if end == size: break crange = resp.headers['Content-Range'] # Determine the total number of bytes to read if size == 0: size = int(crange.split('/')[-1]) - 1 # If the file is smaller than the chunk size, BIG-IP will # return an HTTP 400. So adjust the chunk_size down to the # total file size... if chunk_size > size: end = size # ...and pass on the rest of the code continue start += chunk_size if (current_bytes + chunk_size) > size: end = size else: end = start + chunk_size - 1 Now you know how to upload and download files. Let’s do something with it! A Use Case - Upload Cert & Key to BIG-IP and Create a Clientssl Profile! This whole effort was sparked by a use case in Q&A, so I had to deliver the goods with more than just moving files around. The complete script is linked at the bottom, but there are a few steps required to get to a clientssl certificate: Upload the key & certificate Create the file object for key/cert Create the clientssl profile You know how to do step 1 now. Step 2 is to create the file object for the key and certificate. After a quick test to see which file is the certificate, you set both files, build the payload, then make the POST requests to bind the uploaded files to the file object. def create_cert_obj(bigip, b_url, files): f1 = os.path.basename(files[0]) f2 = os.path.basename(files[1]) if f1.endswith('.crt'): certfilename = f1 keyfilename = f2 else: keyfilename = f1 certfilename = f2 certname = f1.split('.')[0] payload = {} payload['command'] = 'install' payload['name'] = certname # Map Cert to File Object payload['from-local-file'] = '/var/config/rest/downloads/%s' % certfilename bigip.post('%s/sys/crypto/cert' % b_url, json.dumps(payload)) # Map Key to File Object payload['from-local-file'] = '/var/config/rest/downloads/%s' % keyfilename bigip.post('%s/sys/crypto/key' % b_url, json.dumps(payload)) return certfilename, keyfilename Notice we return the key/cert filenames so they can be used for step 3 to establish the clientssl profile. In this example, I name the file object and the clientssl profile to the name of the certfilename (minus the extension) but you can alter this to allow the objects names to be provided. To build the profile, just create the payload with the custom key/cert and make the POST request and you are done! def create_ssl_profile(bigip, b_url, certname, keyname): payload = {} payload['name'] = certname.split('.')[0] payload['cert'] = certname payload['key'] = keyname bigip.post('%s/ltm/profile/client-ssl' % b_url, json.dumps(payload)) Much thanks to Tim Rupp who helped me get across the finish line with some counting and rest worker errors we were troubleshooting on the download function. Get the Code Upload a File Download a File Upload Cert/Key & Build a Clientssl ProfileDemystifying iControl REST: Merging BIG-IP Config Files
A coworker/friend of mine here at F5 was working last week on some super secret project I hope I’ll get to see soon and reached out on the best approach for merging config files via iControl REST. The command in tmsh tmsh load sys config merge file “file” is simple enough, but there are a few things we need to solve for this to work remotely. First, we need to get the file to the BIG-IP. Second, we need to run the merge command. Finally, we need to clean up that file so we don’t leave unnecessary remnants. So let’s address these one by one. Uploading the File This is the easy part! I’ve already written stolen this code from another coworker and shared that with the community in part 5 of thisDemystifying iControl REST series. Bottom line, there is a file worker that you can use to upload files. That’s the good news. The bad news is that you cannot direct that file to a destination of your choosing, it’ll go to /var/config/rest/downloads or /shared/images, depending on the URL you specify in your transfer. More on that when we address the third problem below. Merging the File As shown in the introduction, the tmsh command is simple. But what does that look like in the REST call? Here’s the breakdown: URL: https://bigiphostname/mgmt/tm/sys/config Method: POST Payload: {“command”:”load”, “options”:[{“file”: “/var/config/rest/downloads/uploaded_file_name”, “merge”:true}]} The URL sets the tmsh path (tmsh sys config,) and the tmsh command is established in the payload, with those command line options following. In python, that makes for a simple function. def _merge_config(host, creds, file): requests.packages.urllib3.disable_warnings() b_url = 'https://%s/mgmt/tm/sys/config' % host b = requests.session() b.auth = creds b.verify = False b.headers.update({'Content-Type': 'application/json'}) options = {} options['file'] = '/var/config/rest/downloads/%s' % file options['merge'] = True payload = {} payload['command'] = 'load' payload['options'] = [options] try: merge = b.post(b_url, json.dumps(payload)) if merge.status_code is not 200: print "Merge failed, check rest log file" exit() except Exception, e: print e Cleaning up the File Now that the merge has occurred, we no longer need the file taking up space on the filesystem. Because the upload directory is /var/config/rest/downloads, the space on the /var partition is not nearly as large (a tenth on my VE image) as /shared, depending on the frequency/size of your file uploads it could become a precarious situation if you fill up the /var partition. So the point is…clean up your merge files! And this takes us to another thing we can demystify about iControl REST: how do we move/remove files on the file system, or run cli utilities? Remember that iControl REST is essentially a wrapper for tmsh, so the method needs to be defined there. If you execute run util ? you’ll see a list of commands made available from the system to tmsh. Here is the interesting one for this application: unix-rm Remove files or directories So we can remove system files from tmsh and thus iControl REST…huzzah! But what does that look like in the REST call? Another breakdown: URL: https://bigiphostname/mgmt/tm/util/unix-rm Method: POST Payload: {“command”:”run”, “utilCmdArgs”:”/var/config/rest/downloads/uploaded_file_name” And in python code, that function looks like this: def _cleanup_mergefile(host, creds, file): requests.packages.urllib3.disable_warnings() b_url = 'https://%s/mgmt/tm/util/unix-rm' % host b = requests.session() b.auth = creds b.verify = False b.headers.update({'Content-Type': 'application/json'}) payload = {} payload['command'] = 'run' payload['utilCmdArgs'] = '/var/config/rest/downloads/%s' % file try: cleanup = b.post(b_url, json.dumps(payload)) if cleanup.status_code is not 200: print "Cleanup failed, please check system." except Exception, e: print e Wrapping Up So there you have it: a simple python script utilizing iControl REST to upload and merge BIG-IP config files! The full script is available here in the codeshare. What else about iControl REST do you need demystified? Post down in the comments!Demystifying iControl REST Part 2 - Understanding sub collections and how to use them
iControl REST. It’s iControl SOAP’s baby brother, introduced back in TMOS version 11.4 as an early access feature but was released fully in version 11.5. Several articles on basic usage have been written on iControl REST (see the resources at the bottom of this article) so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. The first article of this series covered the URI’s role in the API. This second article will cover how the URI path plays a role in how the API functions. Working with Subcollections When manipulating F5 configuration items with iControl Rest, subcollections are a powerful tool. They allow one to manipulate specific items in the subcollection instead of having to manipulate the entire sub collection. Take the pool object for example. If we just query the pool (in this case testpool,) you’ll notice the returned data does not list the pool members #query (via the Chrome advanced REST client using Authorization & Content-Type headers:) https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool #query (via curl:) curl -k -u admin:admin https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool #response: { kind: "tm:ltm:pool:poolstate" name: "testpool" fullPath: "testpool" generation: 1 selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool?ver=11.6.0" allowNat: "yes" allowSnat: "yes" ignorePersistedWeight: "disabled" ipTosToClient: "pass-through" ipTosToServer: "pass-through" linkQosToClient: "pass-through" linkQosToServer: "pass-through" loadBalancingMode: "round-robin" minActiveMembers: 0 minUpMembers: 0 minUpMembersAction: "failover" minUpMembersChecking: "disabled" queueDepthLimit: 0 queueOnConnectionLimit: "disabled" queueTimeLimit: 0 reselectTries: 0 serviceDownAction: "none" slowRampTime: 10 membersReference: { link: "https://localhost/mgmt/tm/ltm/pool/~Common~testpool/members?ver=11.6.0" isSubcollection: true }- } Notice the isSubcollection: true for the membersReference? This is an indicator that there is a subcollection for the members keyword. If you then query the members for that pool, you will get the subcollection. #query (via the Chrome advanced REST client using Authorization & Content-Type headers:) https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members #query (via curl:) curl -k -u admin:admin https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members #response: { kind: "tm:ltm:pool:members:memberscollectionstate" selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool/members?ver=11.6.0" items: [4] 0: { kind: "tm:ltm:pool:members:membersstate" name: "192.168.103.10:80" partition: "Common" fullPath: "/Common/192.168.103.10:80" generation: 1 selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool/members/~Common~192.168.103.10:80?ver=11.6.0" address: "192.168.103.10" connectionLimit: 0 dynamicRatio: 1 ephemeral: "false" fqdn: { autopopulate: "disabled" }- inheritProfile: "enabled" logging: "disabled" monitor: "default" priorityGroup: 0 rateLimit: "disabled" ratio: 1 session: "user-enabled" state: "unchecked" } } You can see that there are four pool members as the item count is four, but I’m only showing one of them here for brevity. The pool members can be added, modified, or deleted at this level. Add a pool member URI: https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members Method: POST JSON: {“name”:”192.168.103.12:80”} Modify a pool member URI: https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members/~Common~192.168.103.12:80 Method: PUT JSON: {“name”:”192.168.103.12:80”,”connectionLimit”:”50”} Delete a pool member URI:https://172.16.44.128/mgmt/tm/ltm/pool/~Common~testpool/members/~Common~192.168.103.12:80 Method: DELETE JSON: none (an error will trigger if you send any data) So for subcollections like pool members, adding and deleting is pretty straight forward. Unfortunately, not all lists of configuration items are treated as subcollections. For example, take the data group. You can see for the data group testdb below, the records are not a subcollection. { kind: "tm:ltm:data-group:internal:internalstate" name: "testdb" fullPath: "testdb" generation: 1 selfLink: "https://localhost/mgmt/tm/ltm/data-group/internal/testdb?ver=11.6.0" type: "string" records: [3] 0: { name: "a" data: "one" }- 1: { name: "b" data: "two" }- 2: { name: "c" }- - } Because this is not a subcollection, any modifications to the records of this object are treated as complete replacements. Thus, this request to add a record (d) to the list: URL: https://172.16.44.128/mgmt/tm/ltm/data-group/internal/testdb Method: PUT JSON: {“records” : [ { “name”: “d” } ] } will result in a data group with only 1 entry (d)! If one wanted to add (d) to the data group, one would have to issue the same request above, but with complete JSON data representing the original records PLUS the new record. The same goes if you want to delete a record. You need to submit all the records in JSON format sans the one you wish to delete via the PUT request above. Note: Whereas it's true an update to the records attribute requires a full replacement, it IS possible to update individual records by using the options query parameter instead of updating the records attribute. For details, see the update section of this article. Thus, to modify items that are not subcollections, one would have to issue a get and parse the existing items into a list. Then one would need to modify the list as desired (adding and/or deleting items), and then issue a PUT to the object URI with the modified list as the json data. A python example of doing just that is shown below. This script grabs the records for the MyNetworks data group, adds three new networks to it, then removes those three networks to return it to its original state. __author__ = 'rahm' def get_dg(rq, url, dg_details): dg = rq.get('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1])).json() return dg def extend_dg(rq, url, dg_details, additional_records): dg = rq.get('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1])).json() current_records = dg['records'] new_records = [] for record in current_records: nr = [ {'name': record['name']}] new_records.extend(nr) for record in additional_records: nr = [ {'name': record}] new_records.extend(nr) payload = {} payload['records'] = new_records rq.put('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1]), json.dumps(payload)) def contract_dg(rq, url, dg_details, removal_records): dg = rq.get('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1])).json() new_records = [] for record in removal_records: nr = [ {'name': record}] new_records.extend(nr) current_records = dg['records'] new_records = [x for x in current_records if x not in new_records] payload = {} payload['records'] = new_records rq.put('%s/ltm/data-group/%s/%s' % (url, dg_details[0], dg_details[1]), json.dumps(payload)) if __name__ == "__main__": import requests, json b = requests.session() b.auth = ('admin', 'admin') b.verify = False b.headers.update({'Content-Type' : 'application/json'}) b_url_base = 'https://172.16.44.128/mgmt/tm' dg_details = ['internal', 'myNetworks'] net_changes = ['3.0.0.0/8', '4.0.0.0/8'] print "\nExisting Records for %s Data-Group:\n\t%s" % (dg_details[1], get_dg(b, b_url_base, dg_details)['records']) extend_dg(b, b_url_base, dg_details, net_changes) print "\nUpdated Records for %s Data-Group:\n\t%s" % (dg_details[1], get_dg(b, b_url_base, dg_details)['records']) contract_dg(b, b_url_base, dg_details, net_changes) print "\nUpdated Records for %s Data-Group:\n\t%s" % (dg_details[1], get_dg(b, b_url_base, dg_details)['records']) When running this against my lab BIG-IP, I get this output on my console Existing Records for myNetworks Data-Group: [{u'name': u'1.0.0.0/8'}, {u'name': u'2.0.0.0/8'}] Updated Records for myNetworks Data-Group: [{u'name': u'1.0.0.0/8'}, {u'name': u'2.0.0.0/8'}, {u'name': u'3.0.0.0/8'}, {u'name': u'4.0.0.0/8'}] Updated Records for myNetworks Data-Group: [{u'name': u'1.0.0.0/8'}, {u'name': u'2.0.0.0/8'}] Process finished with exit code 0 Hopefully this has been helpful in showing the power of subcollections and the necessary steps to update objects like data-groups that are not. Much thanks again to Pat Chang for the bulk of the content in this articleNext up: Query Parameters and Options.Demystifying iControl REST Part 3 - How to pass query parameters and tmsh options
iControl REST. It’s iControl SOAP’s baby, brother, introduced back in TMOS version 11.4 as an early access feature but released fully in version 11.5. Several articles on basic usage have been written on iControl REST (see the resources at the bottom of this article) so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. The first article covered URI specifics, the second article discussed subcollections, and this third article will cover query parameters. Query Parameter Definitions F5 has documented a number of query parameters that can be passed into iControl ReST calls in order to modify their behavior. The first set follows the OData (open data protocol) standard. The filter parameter also supports several operators. $filter $select $skip $top Yes, the dollar sign is important and necessary on these parameters. The operators you can use on these parameters are below. Note that the eq operator can only be used with the filter. eq - equal ne - not equal lt - less than le - less than or equal gt - greater than ge - greater than or equal Logical Operators: and or not Beyond the OData parameters, there are a few custom parameters as well. expandSubcollections - allows you to get the subcollection data in the initial request for objects that have subcollections. options - allows you to add arguments to the tmsh equivalent command. An example will be shown below. ver - This is for the specific TMOS version. Setting this parameter guarantees consistent behavior through code upgrades. Please note that the JSON return data for a number of calls has changed between the initial release in 11.5.0 and the current release. No items have been removed, but key/value pairs in the output have been added. Note the lack of a dollar sign on the custom parameters. Example #1 - Filter Now that we have the parameters and operators defined, let’s take a look at some examples. First, we’ll take a look at the $filter parameter. If you want to limit your results to a particular partition, your URL will look something like this: https://172.16.44.128/mgmt/tm/ltm/pool?$filter=partition eq staging https://172.16.44.128/mgmt/tm/ltm/pool?$filter=partition%20eq%20staging https://172.16.44.128/mgmt/tm/ltm/pool?$filter=partition+eq+staging As long as your client tool supports it, any of these formats will work, but the resulting selfLink reflects the latter format: selfLink:"https://localhost/mgmt/tm/ltm/pool?$filter=partition+eq+staging Example #2 - Select I didn’t post the return data from example 1 because it’s a lot of data, even for a small set of returned results. Most of it is all the fields in a pool that are there and important, but default and not of as immediate importance as others. This is where the $select parameter comes in. If you just want to take a look at the name of the pool and say the load balancing mode, your URL will look like this (still filtering for the staging partition:) https://172.16.44.128/mgmt/tm/ltm/pool?$filter=partition+eq+staging&$select=name,loadBalancingMode This results in a smaller subset of data limited to the fields we “selected" items: [5] 0: { name: "sp1" loadBalancingMode: "round-robin" }- 1: { name: "sp2" loadBalancingMode: "round-robin" }- 2: { name: "sp3" loadBalancingMode: "round-robin" } Example #3 - Top & Skip For larger sets of data, you can page through the objects in chunks with $top and $skip. If $skip is not specified when $top is used, it behaves as though set to 0.Please note, however, that paging is restricted to collections and sub collections, so whereas this would work to page through the defined data groups, it would not work to page through the records of a data group. Let’s add the top parameter to our previous URL: https://172.16.44.128/mgmt/tm/ltm/pool?$filter=partition eq staging&$select=name,loadBalancingMode&$top=2 #Results { kind: "tm:ltm:pool:poolcollectionstate" selfLink: "https://localhost/mgmt/tm/ltm/pool?$filter=partition+eq+staging&$select=name%2CloadBalancingMode&$top=2&ver=12.0.0" currentItemCount: 2 itemsPerPage: 2 pageIndex: 1 startIndex: 1 totalItems: 5 totalPages: 3 items: [2] 0: { name: "sp1" loadBalancingMode: "round-robin" } - 1: { name: "sp2" loadBalancingMode: "round-robin" } - - nextLink: "https://localhost/mgmt/tm/ltm/pool?$filter=partition+eq+staging&$select=name%2CloadBalancingMode&$top=2&$skip=2&ver=12.0.0" So we got the same data back as before, only 2 items instead of the original 5, as well as some additional fields that weren’t there previously.Note that once $top is used, the key/value pairs “nextLink”, “currentItems”, and “totalItems” are added to the response. “nextLink” is the URI that will grab the next $top number of results from the query. If you parse this value and use it (once you have replaced the localhost with your actual host information), you will not have to perform any paging calculations. “currentItems” tells you how many items have been returned in the current call. If this value is less than $top, then you know you have reached the end of the items. “totalItems” tells you how many items would be returned if one did not page using $top. Example #4 - Options For this next example, we’ll start with tmsh. If you want to get the connections on the BIG-IP, you type “tmsh show sys conn” at the command line. This can be a very large set of data, however, so there are options on the command line to narrow this down, like cs-client-addr, cs-client-port, and so on. So a narrowed down request at the command line would look like “tmsh show sys conn cs-client-addr 10.0.0.1 cs-client-port 62223.” To translate this command to an API request, you need to use the options parameter https://172.16.44.128/mgmt/tm/sys/connection?options=cs-server-addr+192.168.102.50+cs-server-port+80 #Results { kind: "tm:sys:connection:connectionstats" selfLink: "https://localhost/mgmt/tm/sys/connection?options=cs-server-addr+192.168.102.50+cs-server-port+80&ver=11.6.0" apiRawValues: { apiAnonymous: "Sys::Connections 192.168.102.5:57359 192.168.102.50:80 192.168.102.5:57359 192.168.103.11:8080 tcp 2 (tmm: 1) none Total records returned: 1 " }- } Example #5 - Expanding Subcollections For our final example, we’ll use the expandSubcollections parameter. This is useful for querying objects like pools that have subcollections. Without the parameter specified, the pool data is returned with the specification that pool members is a subcollection, but doesn’t return the set of pool members. By providing the parameter, the pool members are returned along with the pool definition (only first one shown for brevity.) https://172.16.44.128/mgmt/tm/ltm/pool/testpool?expandSubcollections=true #Results { kind: "tm:ltm:pool:poolstate" name: "testpool" fullPath: "testpool" generation: 1 selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool?expandSubcollections=true&ver=11.6.0" allowNat: "yes" allowSnat: "yes" ignorePersistedWeight: "disabled" ipTosToClient: "pass-through" ipTosToServer: "pass-through" linkQosToClient: "pass-through" linkQosToServer: "pass-through" loadBalancingMode: "round-robin" minActiveMembers: 0 minUpMembers: 0 minUpMembersAction: "failover" minUpMembersChecking: "disabled" queueDepthLimit: 0 queueOnConnectionLimit: "disabled" queueTimeLimit: 0 reselectTries: 0 serviceDownAction: "none" slowRampTime: 10 membersReference: { link: "https://localhost/mgmt/tm/ltm/pool/~Common~testpool/members?ver=11.6.0" isSubcollection: true items: [7] 0: { kind: "tm:ltm:pool:members:membersstate" name: "192.168.103.10:80" partition: "Common" fullPath: "/Common/192.168.103.10:80" generation: 1 selfLink: "https://localhost/mgmt/tm/ltm/pool/~Common~testpool/members/~Common~192.168.103.10:80?ver=11.6.0" address: "192.168.103.10" connectionLimit: 0 dynamicRatio: 1 ephemeral: "false" fqdn: { autopopulate: "disabled" } ... Example #6 - Careful! Select Revisited If you want to select just the name and address from the pool members in the previous example, it's not as simple as adding the $select=name,address to that query. Remember from the earlier article about your object/component/sub-component tiers. Instead, you need to specify the members subcollection in the URL, then attach your select parameter. https://172.16.44.128/mgmt/tm/ltm/pool/testpool/members?$select=name,address #Results { kind: "tm:ltm:pool:members:memberscollectionstate" selfLink: "https://localhost/mgmt/tm/ltm/pool/testpool/members?$select=name%2Caddress&ver=11.6.0" items: [7] 0: { name: "192.168.103.10:80" address: "192.168.103.10" }- 1: { name: "192.168.103.10:8080" address: "192.168.103.10" }- 2: { name: "192.168.103.11:80" address: "192.168.103.11" } ... A Whiteboard Wednesday Shout Out to iControl REST I’ve summarized a lot of what was covered in this article in the following Whiteboard Wednesday video. Enjoy!Demystifying iControl REST Part 4: Working with JSON Data
iControl REST. It’s iControl SOAP’s baby, brother, introduced back in TMOS version 11.4 as an early access feature but released fully in version 11.5. Several articles on basic usage have been written on iControl REST so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. This article will cover the details on how to work with the json data format used by iControl REST. Formatting returned data for inspection When first working with iControl REST, you might simply need to display the data queried with a browser REST client, or with curl from the command line. However, that doesn’t look too pretty by default: [root@localhost:Active:Standalone] config # curl -sk -u admin:admin https://172.16.44.15/mgmt/tm/ltm/rule/linkinfo {"kind":"tm:ltm:rule:rulestate","name":"linkinfo","fullPath":"linkinfo","generation":1,"selfLink":"https://localhost/mgmt/tm/ltm/rule/linkinfo?ver=12.0.0","apiAnonymous":"when FLOW_INIT {\n \tlog local0. \"Lasthop ID: [LINK::lasthop id], Type: [LINK::lasthop type], Name: [LINK::lasthop name].\"\n \tlog local0. \"Nexthop ID: [LINK::nexthop id], Type: [LINK::nexthop type], Name: [LINK::nexthop name].\"\n}"}[root@localhost:Active:Standalone] That’s where the following formatting options come into play. There are examples for perl and php floating around as well, but I’ve yet to make them work locally on the BIG-IP, so I haven’t included them here to avoid confusion. python -m json.tool (all versions) json-format (12.0+) jq . (confirmed in 12.0, but if not in 11.x versions, the executable can be installed and referenced accordingly) Using each of these, results in a nice formatting of the data: [root@localhost:Active:Standalone] config # curl -sk -u admin:admin \ https://172.16.44.15/mgmt/tm/ltm/rule/linkinfo | python -m json.tool { "apiAnonymous": "when FLOW_INIT {\n \tlog local0. \"Lasthop ID: [LINK::lasthop id], Type: [LINK::lasthop type], Name: [LINK::lasthop name].\"\n \tlog local0. \"Nexthop ID: [LINK::nexthop id], Type: [LINK::nexthop type], Name: [LINK::nexthop name].\"\n}", "fullPath": "linkinfo", "generation": 1, "kind": "tm:ltm:rule:rulestate", "name": "linkinfo", "selfLink": "https://localhost/mgmt/tm/ltm/rule/linkinfo?ver=12.0.0" } [root@localhost:Active:Standalone] config # curl -sk -u admin:admin \ https://172.16.44.15/mgmt/tm/ltm/rule/linkinfo | json-format { "kind": "tm:ltm:rule:rulestate", "name": "linkinfo", "fullPath": "linkinfo", "generation": 1, "selfLink": "https://localhost/mgmt/tm/ltm/rule/linkinfo?ver\u003d12.0.0", "apiAnonymous": "when FLOW_INIT {\n \tlog local0. \"Lasthop ID: [LINK::lasthop id], Type: [LINK::lasthop type], Name: [LINK::lasthop name].\"\n \tlog local0. \"Nexthop ID: [LINK::nexthop id], Type: [LINK::nexthop type], Name: [LINK::nexthop name].\"\n}" } [root@localhost:Active:Standalone] config # curl -sk -u admin:admin \ https://172.16.44.15/mgmt/tm/ltm/rule/linkinfo | jq . { "kind": "tm:ltm:rule:rulestate", "name": "linkinfo", "fullPath": "linkinfo", "generation": 1, "selfLink": "https://localhost/mgmt/tm/ltm/rule/linkinfo?ver=12.0.0", "apiAnonymous": "when FLOW_INIT {\n \tlog local0. \"Lasthop ID: [LINK::lasthop id], Type: [LINK::lasthop type], Name: [LINK::lasthop name].\"\n \tlog local0. \"Nexthop ID: [LINK::nexthop id], Type: [LINK::nexthop type], Name: [LINK::nexthop name].\"\n}" } If you are on the BIG-IP command line, you will notice that with jq, you also get pretty printing with color, clarifying the key/value pairs. Also with jq, you can return specific fields in the data, see this article for more details. Of course, you might want to do more than just query data. You might also want to use the REST API to create or modify objects as well. You will need to create the POST/PUT/PATCH payload for this, and it should be in JSON format as well. You don’t need to fully populate an object’s fields to successfully submit and create/modify it, you just need to make sure the required fields are present. That changes for each object and for the nature of the changes you are making to an object. Follow the tmsh documentation’s lead for this, either from the reference guide or from interrogating objects in the tmsh shell. For example, to create a pool, you just need to supply the pool name. curl -sk -u admin:admin https://172.16.44.15/mgmt/tm/ltm/pool \ -H 'Content-Type: application/json' \ -X POST \ -d '{"name":"myNewPool"}' Interrogating returned data programmatically So browser tools and curl are great, but that’s a lot of typing if you are going to be using the REST API frequently to perform tasks. This is where a programming language of your choice comes into play. By moving all of your work into code, you are automating the tedium of typing (and remembering) all the nuances of working with the interface. My language of choice is python. For one, it has a nice interpreter shell where you can code line by line and see results. Python’s json module makes it really easy to interrogate data returned from the BIG-IP, and to package data for presenting to the BIG-IP in an object type of choice. Let’s start the shell by typing “python” at the cli and getting the environment ready to make queries. >>> import requests, json >>> b = requests.session() >>> b.auth = ('admin', 'admin') >>> b.verify = False >>> b.headers.update({'Content-TYpe':'application/json'}) >>> b_url_base = 'https://172.16.44.15/mgmt/tm' >>> dg_details = ['internal', 'images'] Now we can make a query to the API, substituting the base url (https://172.16.44.15/mgmt/tm,) the data-group type (internal,) and the name (images.) The requests module object includes the payload and headers, methods, etc, from http, so we need to append another method to interrogate the payload. Let’s start with .text. >>> dg1 = b.get('%s/ltm/data-group/%s/%s' % (b_url_base, dg_details[0], dg_details[1])).text >>> type(dg1) <type 'unicode'> You can see that the .text method results in a unicode string: u'{"kind":"tm:ltm:data-group:internal:internalstate","name":"images","fullPath":"images","generation":1,"selfLink":"https://localhost/mgmt/tm/ltm/data-group/internal/images?ver=12.0.0","type":"string","records":[{"name":".bmp"},{"name":".gif"},{"name":".jpg"}]}' This is helpful to see what your return data is, but not very helpful programmatically. If you try to tickle the kind attribute on a string, you’ll get an error. But by switching from the .text method to .json(), it’ll convert the json data to the native python dictionary object. >>> dg2 = b.get('%s/ltm/data-group/%s/%s' % (b_url_base, dg_details[0], dg_details[1])).json() >>> type(dg) <type 'dict'> >>> dg2 {u'kind': u'tm:ltm:data-group:internal:internalstate', u'name': u'images', u'generation': 1, u'records': [{u'name': u'.bmp'}, {u'name': u'.gif'}, {u'name': u'.jpg'}], u'fullPath': u'images', u'type': u'string', u'selfLink': u'https://localhost/mgmt/tm/ltm/data-group/internal/images?ver=12.0.0'} Now that the data is in a dictionary, you can tickle the attributes of the dg2 object. Let’s print out the records from that dictionary: >>> dg2 = b.get('%s/ltm/data-group/%s/%s' % (b_url_base, dg_details[0], dg_details[1])).json() >>> for record in dg2['records']: ... print record['name'] ... .bmp .gif .jpg So that’s great for return data. But what about data you need to push to the BIG-IP? Well, that’s also easy! Let’s step through this in the python shell: >>> dg3 = {} >>> dg3['name'] = 'images2' >>> dg3['type'] = 'string' >>> records = ['.png', '.bmp', '.jpg'] >>> record_obj = [] >>> for record in records: ... nr = [ {'name': record}] ... record_obj.extend(nr) ... >>> dg3['records'] = record_obj >>> dg3 {'records': [{'name': '.png'}, {'name': '.bmp'}, {'name': '.jpg'}], 'type': 'string', 'name': 'images2'} >>> b.post('%s/ltm/data-group/internal' % b_url_base, json.dumps(dg3)).json() {u'kind': u'tm:ltm:data-group:internal:internalstate', u'name': u'images2', u'generation': 330, u'records': [{u'name': u'.bmp'}, {u'name': u'.jpg'}, {u'name': u'.png'}], u'fullPath': u'images2', u'type': u'string', u'selfLink': u'https://localhost/mgmt/tm/ltm/data-group/internal/images2?ver=12.0.0'} Start by creating an empty dictionary and assigning to dg3. Then populate the name and type keys. For the records, iterate through the image type list, extending the records obj to add to the dictionary. After verifying the dictionary is populated correctly, the post can be issued. Notice in that command the json.dumps(dg3)? That is where python converts the native dictionary type to json data before submitting the payload. I added the .json() on the end to verify the images2 data-group was actually created. Conclusion There are a lot of ways to work with json data. This article covered a couple approaches you can use in your journey to world domination. What’s your approach? Any tools on your tool belt you’d like to share with the community? Drop a comment below!
