Tinkering with the BIGREST Python SDK - Part 1
A couple months back, Leonardo Souza (one of our MVPs) released a new SDK for interacting with the iControl REST interface. A write up right here on DevCentral covers the high level basics of why he rolled his own when we already had one (f5-common-python). In this article, I'm going to compare and contrast the basic functionality of the old SDK with Leo's new SDK, and then add some color commentary after each section. Instantiating BIG-IP This first task is required to build a session with the BIG-IP (or BIG-IQ, but this article will focus on BIG-IP.) This can be done with basic authentication or with tokens, the latter of which you’ll need for remote authentication servers. For this article, the device objects will be labeled fcp for the f5-common-python SDK and br for the bigrest SDK. f5-common-python from f5.bigip import ManagementRoot # Basic Auth fcp = ManagementRoot('10.0.2.15', 'admin', 'admin') # Token Auth fcp = ManagementRoot('10.0.2.15', 'admin', 'admin', token=True) bigrest from bigrest.bigip import BIGIP # Basic Auth br = BIGIP('10.0.2.15', 'admin', 'admin') # Token Auth br = BIGIP('10.0.2.15', 'admin', 'admin', request_token=True) Conclusion For basic auth, instantiation is a wash with the exception of the method itself. In the f5-common-python SDK, ManagementRoot is used for both BIG-IP and BIG-IQ systems, whereas in BIGREST, it is system specific. Being explicit is a good thing, as with BIGREST you’ll always know if you are on BIG-IP or BIG-IQ, but if you are automating tests, using a single nomenclature would likely reduce lines of code. (Updated 9/25) For token authentication, the token and request_token parameters exist to handle exactly that. No differentiator in functionality between the two at all, both work as advertised. What you don’t see in either of these is how they handle SSL certificates. Ideally, from a security standpoint you should have to opt out of a secure transaction, but both libraries are set to ignore certificate errors. f5-common-python has an attribute (verify) that is defaulted to false and it doesn’t look configurable in bigrest. In my view both should be refactored to default to a secure posture so insecurity is an explicit decision by the programmer. The final observation I have on instantiation is proxy support, which is built-in for f5-common-python but not bigrest. Retrieving a List of Pools In this quick comparison, I just want to see how easy it is to grab a list of objects and display their names. It could be anything, but I’ve chosen the pool here. We’ll assume instantiation has already taken place. f5-common-python p1 = fcp.tm.ltm.pools.get_collection() for p in p1: ... print(p.name) ... NewPool p_tcpopt testpool bigrest p2 = br.load('/mgmt/tm/ltm/pool') for p in p2: ... print(p.properties.get('name')) ... NewPool p_tcpopt testpool Conclusion Here is where you can start to see some differences in design choices. In the f5-common-python approach, the SDK methods are associated to defined iControl REST interfaces. With bigrest, that approach is avoided so that no development effort is needed to support missing or new interfaces in the SDK. Whereas that’s a smart choice in my opinion, it does put a little more effort on the programmer to know the interfaces, but on the flip side, the programmer doesn’t need to learn another somewhat different nomenclature in the SDK if they are already familiar with the tmsh endpoints. As far as the data itself is concerned, I do like the easy access to the key/value pairs in f5-common-python (p.name) versus bigrest (p.properties.get(‘name’), or p.properties[’name’] if you prefer), but that’s a minor gripe given the flexibility of loading any endpoint with a single simple load command. CRUD Operations for a Pool and its Members f5-common-python Creating a pool and a pool member pool = fcp.tm.ltm.pools.pool.create(name='fcp_pool', partition='Common') fcp.tm.ltm.pools.pool.exists(name='fcp_pool', partition='Common') True pool_member = pool.members_s.members.create(name='192.168.102.44:80', partition='Common') pool.members_s.members.exists(name='192.168.102.44:80', partition='Common') True Refreshing and updating a pool # Load and refresh without updating p1 = fcp.tm.ltm.pools.pool.load(name='fcp_pool', partition='Common') p1.description Traceback (most recent call last): File "", line 1, in File "/Users/rahm/Documents/PycharmProjects/scripts/py38/lib/python3.8/site-packages/f5/bigip/mixins.py", line 102, in __getattr__ raise AttributeError(error_message) AttributeError: '<class 'f5.bigip.tm.ltm.pool.Pool'>' object has no attribute 'description' p1.description = 'updating description' p1.description 'updating description' p1.refresh() p1.description Traceback (most recent call last): File "", line 1, in File "/Users/rahm/Documents/PycharmProjects/scripts/py38/lib/python3.8/site-packages/f5/bigip/mixins.py", line 102, in __getattr__ raise AttributeError(error_message) AttributeError: '<class 'f5.bigip.tm.ltm.pool.Pool'>' object has no attribute 'description' # Load, update, and refresh p1 = fcp.tm.ltm.pools.pool.load(name='fcp_pool', partition='Common') p2 = fcp.tm.ltm.pools.pool.load(name='fcp_pool', partition='Common') assert p1.name == p2.name p1.description = 'updating description' p1.update() p1.description 'updating description' p2.description Traceback (most recent call last): File "", line 1, in File "/Users/rahm/Documents/PycharmProjects/scripts/py38/lib/python3.8/site-packages/f5/bigip/mixins.py", line 102, in __getattr__ raise AttributeError(error_message) AttributeError: '<class 'f5.bigip.tm.ltm.pool.Pool'>' object has no attribute 'description' p2.refresh() p2.description 'updating description' Deleting a pool p1 = fcp.tm.ltm.pools.pool.load(name='fcp_pool', partition='Common') p1.delete() fcp.tm.ltm.pools.pool.exists(name='fcp_pool', partition='Common') False bigrest Creating a pool and a pool member pool_data = {} pool_data['name'] = 'br_pool' pool = br.create('/mgmt/tm/ltm/pool', pool_data) br.exist('/mgmt/tm/ltm/pool/br_pool') True pm_data = {} pm_data['name'] = '192.168.102.44:80' pool_member = br.create('/mgmt/tm/ltm/pool/br_pool/members', pm_data) br.exist('/mgmt/tm/ltm/pool/br_pool/members/~Common~192.168.102.44:80') True Refreshing and updating a pool # Load and refresh without updating p1 = br.load('/mgmt/tm/ltm/pool/br_pool')[0] p1.properties['description'] Traceback (most recent call last): File "", line 1, in KeyError: 'description' p1.properties['description'] = 'updating description' p1.properties['description'] 'updating description' p1 = br.load('/mgmt/tm/ltm/pool/br_pool')[0] p1.properties['description'] Traceback (most recent call last): File "", line 1, in KeyError: 'description' # Load, update and refresh p1 = br.load('/mgmt/tm/ltm/pool/br_pool')[0] p2 = br.load('/mgmt/tm/ltm/pool/br_pool')[0] assert p1.properties['name'] == p2.properties['name'] p1.properties['description'] = 'updating description' p1 = br.save(p1) p1.properties['description'] 'updating description' p2.properties['description'] Traceback (most recent call last): File "", line 1, in KeyError: 'description' p2 = br.load('/mgmt/tm/ltm/pool/br_pool')[0] p2.properties['description'] 'updating description' Deleting a pool br.delete('/mgmt/tm/ltm/pool/br_pool') br.exist('/mgmt/tm/ltm/pool/br_pool') False Conclusion With both approaches, you need to understand the tmsh hierarchy of BIG-IP objects. That said, with bigrest, you only need to know that. With f5-common-python, you need to understand as well the nuances of the SDK authors and how it was built (see pools.pool.members_s.members above in the code.) I also like with bigrest that you can delete an object with one command; in f5-common-python you have to load the object before deleting it. I don’t like that bigrest doesn’t distinguish between collections and single objects, so that means you have to do a little extra repetitive work with isolating the single list object instead of that being handled for you. Being able to work with any collection or single object, though, is super handy and a big differentiator, as bigrest isn’t bogged down with defining the endpoints. I’m sure there are some (BIG-IP) version specific gotchas with the bigrest save command throwing the whole object back to the REST interface, as I recall coding around many of the nuances with f5-common-python where we’d have to strip some attributes before updating, but that’s a design choice where keeping things super “dumb” makes everything more flexible. There is no refresh method in bigrest, but reloading the object works just as well, and it should be clear by now that bigrest’s update method is called save and works at the device level instead of the local python object level but that’s also a design choice that doesn’t bother me at all. My final observation here is the requirement in bigrest to build a data object. I don't hate it, but I do like with f5-common-python how I can just pass an attribute to the method and it creates the data object for me. Updating Unnamed Objects Most objects are created and thus have names, but there are many that exist in the system that are not named and cannot be created or deleted. f5-common-python Adding DNS servers to the BIG-IP configuration settings = fcp.tm.sys.dns.load() settings.nameServers = ['8.8.8.8'] settings.update() bigrest Adding DNS servers to the BIG-IP configuration settings = br.load('/mgmt/tm/sys/dns')[0] settings.properties['nameServers'] = ['8.8.8.8'] settings = br.save(settings) Conclusion There are no real challenging or intriguing differences here that I haven’t already covered, so let’s just move on! More to come... In part two I’ll move beyond the basic building blocks of configuration to look at how to run commands, upload and download files, and work with stats, transactions, and task. Afterward, I’ll weigh in on my overall thoughts for both projects and where to go from here.Tinkering with the BIGREST Python SDK - Part 2
A couple months back, Leonardo Souza (one of our MVPs) released a new SDK for interacting with the iControl REST interface. A write up right here on DevCentral covers the high level basics of why he rolled his own when we already had one (f5-common-python). In part one I covered typical CRUD operations. In this article, I'm going to compare and contrast operational functionality of the old SDK with Leo's new SDK, adding commentary after each section and concluding with overall impressions. Uploading and Downloading Files We’ll start with file management since we’ll need some files in place for a couple of the following sections. The upload/download options are shown below. For things like (small) config files, certs, keys, iFiles, etc, I upload everything to /var/config/rest/downloads. Images and big files always to /shared/images. For non-UCS downloads, I move files to /shared/images, download the appropriate files, then delete them as necessary. Description Method URI File Location Upload Image POST /mgmt/cm/autodeploy/sotfware-image-uploads/* /shared/images Upload File POST /mgmt/shared/file-transfer/uploads/* /var/config/rest/downloads Upload UCS POST /mgmt/shared/file-transfer/ucs-uploads/* /var/local/ucs Download UCS GET /mgmt/shared/file-transfer/ucs-downloads/* /var/local/ucs Download Image/File GET /mgmt/cm/autodeploy/sotfware-image-downloads/* /shared/images For this example, I'll upload the myfile.txt file that contains a pool configuration that I'll merge later in this article, and I'll download an image file. f5-common-python # Upload myfile.txt to /var/config/rest/downloads/ fcp.shared.file_transfer.uploads.upload_file('myfile.txt') # Download BIGIP-13.1.3-0.0.6.iso from /shared/images/ fcp.cm.autodeploy.software_image_downloads.download_file('BIGIP-13.1.3-0.0.6.iso', 'BIGIP-13.1.3-0.0.6.iso') bigrest # Upload myfile.txt to /var/config/rest/downloads/ br.upload('/mgmt/shared/file-transfer/uploads/', 'myfile.txt') # Download BIGIP-13.1.3-0.0.6.iso from /shared/images/ br.download('/mgmt/cm/autodeploy/sotfware-image-downloads/', 'BIGIP-13.1.3-0.0.6.iso') Conclusion Given that the iControlREST interface itself is not very helpful in keeping track of where to upload/download files, it’s hardly a fault of either SDK in approach. That said, I like bigrest’s angle in that the endpoints are direct tmsh mappings, so as long as you have the table memorized or available, you won’t have to hunt in the SDK source to find out how to manage files, so that’s a big win. If I were to make a tweak to bigrest here, however, I’d leave the upload/download methods as is but add a type (image/ucs/file) argument so the SDK would handle the correct endpoint for me. The tradeoff there though is violating the consistency of bigrest in forcing the user to supply the endpoints, and isn’t future-proofed in the event those endpoints change. So maybe I’m walking myself out of that argument… :) Running Commands Sometimes instead of managing the BIG-IP configuration, you need to interact with the system, whether it’s loading or saving a configuration or checking dns resolution from the system’s perspective. For this illustration, I’ll show how to merge the following myfile.txt configuration file that I uploaded to my BIG-IP with the above steps. # File to merge: myfile.txt ltm pool mergepool { members { 192.168.103.20:http { address 192.168.103.20 session monitor-enabled state down } } monitor http } f5-common-python fcp.tm.ltm.pools.pool.exists(name='mergepool') False options = {} options['file'] = '/var/config/rest/downloads/myfile.txt' options['merge'] = True fcp.tm.sys.config.exec_cmd('load', options=[options]) fcp.tm.ltm.pools.pool.exists(name='mergepool') True bigrest br.exist('/mgmt/tm/ltm/pool/mergepool') False options = {} options['file'] = '/var/config/rest/downloads/myfile.txt' options['merge'] = True data = {} data['command'] = 'load' data['options'] = [options] br.command('/mgmt/tm/sys/config', data) br.exist('/mgmt/tm/ltm/pool/mergepool') True Conclusion In both cases I have to pack some data to supply the tmsh sys load command. F5-common-python does some of the work for me, but bigrest requires me to pack both blobs. But again, the simple command method is more clear with bigrest than f5-common-python’s exec_cmd. Working with Stats The stats interface in iControl REST is not for the faint of heart. Both libraries provide assistance to make the return data more immediately usable. f5-common-python from f5.utils.responses.handlers import Stats pool = fcp.tm.ltm.pools.pool.load(name='testpool') poolstats = Stats(pool.stats.load()) for k, v in poolstats.stat.items(): if v.get('description') != None: print(f'{k}: {v.get("description")}') elif v.get('value') != None: print(f'{k}: {v.get("value")}') # Result activeMemberCnt: 0 availableMemberCnt: 0 ... status_availabilityState: offline status_enabledState: enabled status_statusReason: The children pool member(s) are down totRequests: 0 bigrest poolstats = br.show('/mgmt/tm/ltm/pool') for p in poolstats: for k, v in p.properties.items(): if v.get('description') != None: print(f'{k}: {v.get("description")}') elif v.get('value') != None: print(f'{k}: {v.get("value")}') # Result ...same as above from f5-common-python Conclusion Stats management is built-in with bigrest with no requirement to load additional utilities, and it is bound to the show method which makes a lot of sense since that’s how they are consumed on the tmsh cli. They are stored slightly differently in the sdk objects, with f5-common-python storing them as a list of dictionaries and bigrest storing them as a dictionary of dictionaries, but once that is managed on the python side, the data is the data. Working with Request Parameters You can read more about using request parameters with the iControl REST interface here and here. For this example, I’ll pull the BIG-IP’s list of pools, but only their name and load balancing mode attributes by utilizing the select parameter. f5-common-python data = {} data['params'] = '$select=name,loadBalancingMode' pools = fcp.tm.ltm.pools.get_collection(requests_params=data) for p in pools: ... print(p) ... {'name': 'mergepool', 'loadBalancingMode': 'round-robin'} {'name': 'nerdlife_pool', 'loadBalancingMode': 'round-robin'} {'name': 'testpool', 'loadBalancingMode': 'round-robin'} {'name': 'p1', 'loadBalancingMode': 'round-robin'} bigrest pools = br.load('/mgmt/tm/ltm/pool/?$select=name,loadBalancingMode') for p in pools: ... print(p) ... { "name": "mergepool", "loadBalancingMode": "round-robin" } { "name": "nerdlife_pool", "loadBalancingMode": "round-robin" } { "name": "testpool", "loadBalancingMode": "round-robin" } Conclusion Because the endpoint URIs are specified directly in bigrest, that makes using the request parameters much more simple. In f5-common-python, you have to package the parameters and pass it as an object to the method, which is a little clunky. Working with Transactions Sometimes changes are dependent on other changes, for example, updating key and certificate files while they are already applied to a clientssl profile in use. Or, you create some helper objects while preparing to create a virtual server, but the virtual server creation fails and leaves you with some artifacts. In cases like that, you will want to use a transaction so that any of the pre-work that is accomplished before an error doesn’t actually get committed to the system. For this example, I’ll create a pool and then try to create an already existing virtual server to see the failure, then create a virtual server to show success. f5-common-python # Import the transaction manager from f5.bigip.contexts import TransactionContextManager # Setup the transaction tx = fcp.tm.transactions.transaction # Execute the transaction with TransactionContextManager(tx) as api: api.tm.ltm.pools.pool.create(name='testpool_fcp') api.tm.ltm.virtuals.virtual.create(name='testvip', pool='testpool_fcp', destination='192.168.102.199:80') # Error because the virtual already exists f5.sdk_exception.TransactionSubmitException: 409 Unexpected Error: Conflict for uri: https://ltm3.test.local:443/mgmt/tm/transaction/1599859075279313/ Text: '{"code":409,"message":"transaction failed:01020066:3: The requested Virtual Server (/Common/testvip) already exists in partition Common.","errorStack":[],"apiError":2}' # Note that the pool was not created fcp.tm.ltm.pools.pool.exists(name='testpool_fcp') False # Execute the transaction with TransactionContextManager(tx) as api: api.tm.ltm.pools.pool.create(name='testpool_fcp') api.tm.ltm.virtuals.virtual.create(name='testvip199', pool='testpool_fcp', destination='192.168.102.199:80') # No error, check to see if pool and virtual exist fcp.tm.ltm.pools.pool.exists(name='testpool_fcp') True fcp.tm.ltm.virtuals.virtual.exists(name='testvip199') True bigrest # Execute the transaction with br as transaction: data = {} data['name'] = 'testpool_br' br.create('/mgmt/tm/ltm/pool', data) data = {} data['name'] = 'testvip' data['pool'] = 'testpool_br' data['destination'] = '192.168.102.199:80' br.create('/mgmt/tm/ltm/virtual', data) raise RESTAPIError(response, self.debug) # Error because the virtual already exists bigrest.common.exceptions.RESTAPIError: Status: 409 Response Body: { "code": 409, "message": "transaction failed:01020066:3: The requested Virtual Server (/Common/testvip) already exists in partition Common.", "errorStack": [], "apiError": 2 } # Note that the pool was not created br.exist('/mgmt/tm/ltm/pool/testpool_br') False # Execute the transaction with br as transaction: data = {} data['name'] = 'testpool_br' br.create('/mgmt/tm/ltm/pool', data) data = {} data['name'] = 'testvip199' data['pool'] = 'testpool_br' data['destination'] = '192.168.102.199:80' br.create('/mgmt/tm/ltm/virtual', data) # No error, check to see if pool and virtual exist br.exist('/mgmt/tm/ltm/pool/testpool_br') True br.exist('/mgmt/tm/ltm/virtual/testvip199') True Conclusion There’s a lot less setup required with bigrest transactions, so on that front alone I think it’s easier to work with. But bigrest has another ace up it’s sleeve by also supporting specific transaction methods: transaction_create, transaction_validate, and transaction_commit. Here’s an example of their use, and how you can interrogate the commands in a transaction should you need to remove or reorder any. from bigrest.bigip import BIGIP device = BIGIP('ltm3.test.local', 'admin', 'admin') transaction_create = device.transaction_create() transaction_id = transaction_create.properties["transId"] print(f"Transaction ID: {transaction_id}.") pdata = {} pdata["name"] = 'pool_name3' device.create("/mgmt/tm/ltm/pool", pdata) vdata = {} vdata["name"] = 'virtual_name3' vdata['pool'] = 'pool_name3' vdata["destination"] = "10.17.0.3%0:80" device.create("/mgmt/tm/ltm/virtual", vdata) tx1 = device.load(f"/mgmt/tm/transaction/{transaction_id}/commands/1")[0] tx2 = device.load(f"/mgmt/tm/transaction/{transaction_id}/commands/2")[0] print(tx1, tx2) device.transaction_validate() device.transaction_commit() print(f"Transaction {transaction_id} completed.") #output Transaction ID: 1599862284314421. { "method": "POST", "uri": "https://localhost/mgmt/tm/ltm/pool", "body": { "name": "pool_name3" }, "evalOrder": 1, "commandId": 1, "kind": "tm:transaction:commandsstate", "selfLink": "https://localhost/mgmt/tm/transaction/1599862284314421/commands/1?ver=13.1.1.5" } { "method": "POST", "uri": "https://localhost/mgmt/tm/ltm/virtual", "body": { "name": "virtual_name3", "pool": "pool_name3", "destination": "10.17.0.3%0:80" }, "evalOrder": 2, "commandId": 2, "kind": "tm:transaction:commandsstate", "selfLink": "https://localhost/mgmt/tm/transaction/1599862284314421/commands/2?ver=13.1.1.5" } Transaction 1599862284314421 completed. That's a pretty cool differentiator for bigrest! Working with Tasks In iControl REST, a task is an asynchronous operation that allows you to kick off a long-lived process like performing a UCS backup or running a cli script. Not every endpoint has an associated task available, so you’ll need to consult the user guides for the complete list for your version. f5-common-python This SDK does not have task support. bigrest # Import datetime to show task creation/completion times from datetime import datetime data = {} data["command"] = "save" task = br.task_start("/mgmt/tm/task/sys/config", data) print(f'Task {task.properties["_taskId"]} created at {datetime.now().strftime("%H:%M:%S")}') br.task_wait(task) if br.task_completed(task): br.task_result(task) print(f'Task {task.properties["_taskId"]} completed at {datetime.now().strftime("%H:%M:%S")}') else: raise Exception() # Result Task 1599880917597158 created at 22:21:57 Task 1599880917597158 completed at 22:22:07 Conclusion Obviously this is a default win for bigrest since there is no support at all for tasks in f5-common-python, but it’s also super easy. Debugging When facing issues with what is and is not working, it’s helpful to see what the SDK is actually sending to the BIG-IP. f5-common-python fcp = ManagementRoot('ltm3.test.local', 'admin', 'admin', debug=True) fcp.debug True pool_list = fcp.tm.ltm.pools.get_collection() fcp.debug_output ["curl -k -X GET https://ltm3.test.local:443/mgmt/tm/ltm/pool/ -H 'User-Agent: python-requests/2.23.0 f5-icontrol-rest-python/1.3.13' -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'Connection: keep-alive' -H 'Content-Type: application/json' -H 'Cookie: BIGIPAuthCookie=51E86ABC11D59EC24E393DBD608921A4EB9678C9; BIGIPAuthUsernameCookie=admin' -H 'Authorization: Basic YWRtaW46YWRtaW4='"] fcp = ManagementRoot('ltm3.test.local', 'admin', 'admin', debug=True, token=True) pool_list = fcp.tm.ltm.pools.get_collection() fcp.debug_output ["curl -k -X GET https://ltm3.test.local:443/mgmt/tm/ltm/pool/ -H 'User-Agent: python-requests/2.23.0 f5-icontrol-rest-python/1.3.13' -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'Connection: keep-alive' -H 'Content-Type: application/json' -H 'X-F5-Auth-Token: 4KB7K2XUPBPRF3P2GO37A5ZA4D'"] bigrest br = BIGIP('ltm3.test.local', 'admin', 'admin', debug='br_output.txt' Conclusion For the life of me I could not figure out why the debug option in bigrest was not working, until I had a failure with an API call! There are a few differences here. One, f5-common-python stores output in the instantiated object's debug_output attribute (fcp.debug_output in our case,) whereas bigrest uses files for the output. Two, bigrest only writes to the debug file when there is a failure. Three, bigrest logs actual request/response information, whereas f5-common-python output's only what the request would look like in a curl command. This is useful with out-of-band verification of the functionality, but lacks the actual details. I see the value in both approaches and would like to see both as options. Closing Thoughts Whew! We made it through most of the BIG-IP functionality of both libraries. What we didn't cover at all is the BIG-IQ functionality, which bigrest supports much more fully than f5-common-python, which implemented a few endpoints but that's about it. The examples in these intro articles are simple discovery exercises, so my likes and dislikes might not hold when transitioning into real coding assignments. I’m going to take a look in my next article on bigrest at refactoring some existing f5-common-python scripts into bigrest-based scripts, so stay tuned for that! Before my good friend Tim Rupp left F5, we had considered a major overhaul of f5-common-python with a lot of design choices that Leo made with bigrest. But now that that this excellent work has been completed in bigrest, and considering that F5’s strategy going forward will be primarily focused on the declarative model, it hardly makes sense for us to continue to invest development effort in f5-common-python. Go therefore and install bigrest, mess around with it, and post your thoughts in the comments below! Well done, Leo!
