Demystifying iControl REST Part 5: Transferring Files

 iControl REST. It’s iControl SOAP’s baby, brother, introduced back in TMOS version 11.4 as an early access feature but released fully in version 11.5.

Several articles on basic usage have been written on iControl REST so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. This article will cover the details on how to transfer files to/from the BIG-IP using iControl REST and the python programming language. (Note: this functionality requires 12.0+.)

The REST File Transfer Worker

The file transfer worker allows a client to transfer files through a series of GET operations for downloads and POST operations for uploads. The Content-Range header is used for both as a means to chunk the content. For downloads, the worker listens on the following interfaces.

Description	Method	URI	File Location
Download a File	GET	/mgmt/cm/autodeploy/software-image-downloads/	/shared/images/
Upload an Image File	POST	/mgmt/cm/autodeploy/software-image-uploads/	/shared/images/
Upload a File	POST	/mgmt/shared/file-transfer/uploads/	/var/config/rest/downloads/
Download a QKView	GET	/mgmt/shared/file-transfer/qkview-downloads/	/var/tmp/
Download a UCS	GET	/mgmt/shared/file-transfer/ucs-downloads/	/var/local/ucs/
Upload ASM Policy	POST	/mgmt/tm/asm/file-transfer/uploads/	/var/ts/var/rest/
Download ASM Policy	GET	/mgmt/tm/asm/file-transfer/downloads/	/var/ts/var/rest/

Binary and text files are supported. The magic in the transfer is the Content-Range header, which has the following format:

Content-Range: start-end/filesize

Where start/end are the chunk's delimiters in the file and filesize is well, the file size. Any file larger than 1M needs to be chunked with this header as that limit is enforced by the worker. This is done to avoid potential denial of service attacks and out of memory errors. There are benefits of chunking as well: 

• Accurate progress bars
• Resuming interrupted downloads
• Random access to file content possible 

Uploading a File

The function is shown below. Note that whereas normally with the REST API the Content-Type is application/json, with file transfers that changes to application/octet-stream. The workflow for the function works like this (line number in parentheses) :

1. Set the Chunk Size (3)
2. Set the Content-Type header (4-6)
3. Open the file (7)
4. Get the filename (apart from the path) from the absolute path (8)
5. If the extension is an .iso file (image) put it in /shared/images, otherwise it’ll go in /var/config/rest/downloads (9-12)
6. Disable ssl warnings requests (required with my version: 2.8.1. YMMV) (14)
7. Set the total file size for use with the Content-Range header (15)
8. Set the start variable to 0 (17)
9. Begin loop to iterate through the file and upload in chunks (19)
10. Read data from the file and if there is no more data, break the loop (20-22)
11. set the current bytes read, if less than the chunk size, then this is the last chunk, so set the end to the size from step 7. Otherwise, add current bytes length to the start value and set that as the end. (24-28)
12. Set the Content-Range header value and then add that to the header (30-31)
13. Make the POST request, uploading the content chunk (32-36)
14. Increment the start value by the current bytes content length (38)

def _upload(host, creds, fp):

    chunk_size = 512 * 1024
    headers = {
        'Content-Type': 'application/octet-stream'
    }
    fileobj = open(fp, 'rb')
    filename = os.path.basename(fp)
    if os.path.splitext(filename)[-1] == '.iso':
        uri = 'https://%s/mgmt/cm/autodeploy/software-image-uploads/%s' % (host, filename)
    else:
        uri = 'https://%s/mgmt/shared/file-transfer/uploads/%s' % (host, filename)

    requests.packages.urllib3.disable_warnings()
    size = os.path.getsize(fp)

    start = 0

    while True:
        file_slice = fileobj.read(chunk_size)
        if not file_slice:
            break

        current_bytes = len(file_slice)
        if current_bytes < chunk_size:
            end = size
        else:
            end = start + current_bytes

        content_range = "%s-%s/%s" % (start, end - 1, size)
        headers['Content-Range'] = content_range
        requests.post(uri,
                      auth=creds,
                      data=file_slice,
                      headers=headers,
                      verify=False)

        start += current_bytes

Downloading a File

Downloading is very similar but there are some differences. Here is the workflow that is different, followed by the code. Note that the local path where the file will be downloaded to is given as part of the filename.

1. URI is set to downloads worker. The only supported download directory at this time is /shared/images. (8)
2. Open the local file so received data can be written to it (11)
3. Make the request (22-26)
4. If response code is 200 and if size is greater than 0, increment the current bytes and write the data to file, otherwise exit the loop (28-40)
5. Set the value of the returned Content-Range header to crange and if initial size (0), set the file size to the size variable (42-46)
6. If the file is smaller than the chunk size, adjust the chunk size down to the total file size and continue (51-55)
7. Do the math to get ready to download the next chunk (57-62)

def _download(host, creds, fp):
    chunk_size = 512 * 1024

    headers = {
        'Content-Type': 'application/octet-stream'
    }
    filename = os.path.basename(fp)
    uri = 'https://%s/mgmt/cm/autodeploy/software-image-downloads/%s' % (host, filename)
    requests.packages.urllib3.disable_warnings()

    with open(fp, 'wb') as f:
        start = 0
        end = chunk_size - 1
        size = 0
        current_bytes = 0

        while True:
            content_range = "%s-%s/%s" % (start, end, size)
            headers['Content-Range'] = content_range

            #print headers
            resp = requests.get(uri,
                                auth=creds,
                                headers=headers,
                                verify=False,
                                stream=True)

            if resp.status_code == 200:
                # If the size is zero, then this is the first time through the
                # loop and we don't want to write data because we haven't yet
                # figured out the total size of the file.
                if size > 0:
                    current_bytes += chunk_size
                    for chunk in resp.iter_content(chunk_size):
                        f.write(chunk)

                # Once we've downloaded the entire file, we can break out of
                # the loop
                if end == size:
                    break

            crange = resp.headers['Content-Range']

            # Determine the total number of bytes to read
            if size == 0:
                size = int(crange.split('/')[-1]) - 1

                # If the file is smaller than the chunk size, BIG-IP will
                # return an HTTP 400. So adjust the chunk_size down to the
                # total file size...
                if chunk_size > size:
                    end = size

                # ...and pass on the rest of the code
                continue

            start += chunk_size

            if (current_bytes + chunk_size) > size:
                end = size
            else:
                end = start + chunk_size - 1

Now you know how to upload and download files. Let’s do something with it!

A Use Case - Upload Cert & Key to BIG-IP and Create a Clientssl Profile!

This whole effort was sparked by a use case in Q&A, so I had to deliver the goods with more than just moving files around. The complete script is linked at the bottom, but there are a few steps required to get to a clientssl certificate:

1. Upload the key & certificate
2. Create the file object for key/cert
3. Create the clientssl profile

You know how to do step 1 now. Step 2 is to create the file object for the key and certificate. After a quick test to see which file is the certificate, you set both files, build the payload, then make the POST requests to bind the uploaded files to the file object.

def create_cert_obj(bigip, b_url, files):

    f1 = os.path.basename(files[0])
    f2 = os.path.basename(files[1])
    if f1.endswith('.crt'):
        certfilename = f1
        keyfilename = f2
    else:
        keyfilename = f1
        certfilename = f2

    certname = f1.split('.')[0]

    payload = {}
    payload['command'] = 'install'
    payload['name'] = certname

    # Map Cert to File Object
    payload['from-local-file'] = '/var/config/rest/downloads/%s' % certfilename
    bigip.post('%s/sys/crypto/cert' % b_url, json.dumps(payload))

    # Map Key to File Object
    payload['from-local-file'] = '/var/config/rest/downloads/%s' % keyfilename
    bigip.post('%s/sys/crypto/key' % b_url, json.dumps(payload))

    return certfilename, keyfilename

Notice we return the key/cert filenames so they can be used for step 3 to establish the clientssl profile. In this example, I name the file object and the clientssl profile to the name of the certfilename (minus the extension) but you can alter this to allow the objects names to be provided. To build the profile, just create the payload with the custom key/cert and make the POST request and you are done!

def create_ssl_profile(bigip, b_url, certname, keyname):
    payload = {}
    payload['name'] = certname.split('.')[0]
    payload['cert'] = certname
    payload['key'] = keyname
    bigip.post('%s/ltm/profile/client-ssl' % b_url, json.dumps(payload))

Much thanks to Tim Rupp who helped me get across the finish line with some counting and rest worker errors we were troubleshooting on the download function.

Get the Code

• Upload a File
• Download a File
• Upload Cert/Key & Build a Clientssl Profile