iControl REST. It’s iControl SOAP’s baby, brother, introduced back in TMOS version 11.4 as an early access feature but released fully in version 11.5.
Several articles on basic usage have been written on iControl REST so the intent here isn’t basic use, but rather to demystify some of the finer details of using the API. This article will cover the details on how to transfer files to/from the BIG-IP using iControl REST and the python programming language. (Note: this functionality requires 12.0+.)
The REST File Transfer Worker
The file transfer worker allows a client to transfer files through a series of GET operations for downloads and POST operations for uploads. The Content-Range header is used for both as a means to chunk the content. For downloads, the worker listens on the following interfaces.
Description
Method
URI
File Location
Download a File
GET
/mgmt/cm/autodeploy/software-image-downloads/
/shared/images/
Upload an Image File
POST
/mgmt/cm/autodeploy/software-image-uploads/
/shared/images/
Upload a File
POST
/mgmt/shared/file-transfer/uploads/
/var/config/rest/downloads/
Download a QKView
GET
/mgmt/shared/file-transfer/qkview-downloads/
/var/tmp/
Download a UCS
GET
/mgmt/shared/file-transfer/ucs-downloads/
/var/local/ucs/
Upload ASM Policy
POST
/mgmt/tm/asm/file-transfer/uploads/
/var/ts/var/rest/
Download ASM Policy
GET
/mgmt/tm/asm/file-transfer/downloads/
/var/ts/var/rest/
Binary and text files are supported. The magic in the transfer is the Content-Range header, which has the following format:
Content-Range: start-end/filesize
Where start/end are the chunk's delimiters in the file and filesize is well, the file size. Any file larger than 1M needs to be chunked with this header as that limit is enforced by the worker. This is done to avoid potential denial of service attacks and out of memory errors. There are benefits of chunking as well:
Accurate progress bars
Resuming interrupted downloads
Random access to file content possible
Uploading a File
The function is shown below. Note that whereas normally with the REST API the Content-Type is application/json, with file transfers that changes to application/octet-stream. The workflow for the function works like this (line number in parentheses) :
Set the Chunk Size (3)
Set the Content-Type header (4-6)
Open the file (7)
Get the filename (apart from the path) from the absolute path (8)
If the extension is an .iso file (image) put it in /shared/images, otherwise it’ll go in /var/config/rest/downloads (9-12)
Disable ssl warnings requests (required with my version: 2.8.1. YMMV) (14)
Set the total file size for use with the Content-Range header (15)
Set the start variable to 0 (17)
Begin loop to iterate through the file and upload in chunks (19)
Read data from the file and if there is no more data, break the loop (20-22)
set the current bytes read, if less than the chunk size, then this is the last chunk, so set the end to the size from step 7. Otherwise, add current bytes length to the start value and set that as the end. (24-28)
Set the Content-Range header value and then add that to the header (30-31)
Make the POST request, uploading the content chunk (32-36)
Increment the start value by the current bytes content length (38)
def _upload(host, creds, fp):
chunk_size = 512 * 1024
headers = {
'Content-Type': 'application/octet-stream'
}
fileobj = open(fp, 'rb')
filename = os.path.basename(fp)
if os.path.splitext(filename)[-1] == '.iso':
uri = 'https://%s/mgmt/cm/autodeploy/software-image-uploads/%s' % (host, filename)
else:
uri = 'https://%s/mgmt/shared/file-transfer/uploads/%s' % (host, filename)
requests.packages.urllib3.disable_warnings()
size = os.path.getsize(fp)
start = 0
while True:
file_slice = fileobj.read(chunk_size)
if not file_slice:
break
current_bytes = len(file_slice)
if current_bytes < chunk_size:
end = size
else:
end = start + current_bytes
content_range = "%s-%s/%s" % (start, end - 1, size)
headers['Content-Range'] = content_range
requests.post(uri,
auth=creds,
data=file_slice,
headers=headers,
verify=False)
start += current_bytes
Downloading a File
Downloading is very similar but there are some differences. Here is the workflow that is different, followed by the code. Note that the local path where the file will be downloaded to is given as part of the filename.
URI is set to downloads worker. The only supported download directory at this time is /shared/images. (8)
Open the local file so received data can be written to it (11)
Make the request (22-26)
If response code is 200 and if size is greater than 0, increment the current bytes and write the data to file, otherwise exit the loop (28-40)
Set the value of the returned Content-Range header to crange and if initial size (0), set the file size to the size variable (42-46)
If the file is smaller than the chunk size, adjust the chunk size down to the total file size and continue (51-55)
Do the math to get ready to download the next chunk (57-62)
def _download(host, creds, fp):
chunk_size = 512 * 1024
headers = {
'Content-Type': 'application/octet-stream'
}
filename = os.path.basename(fp)
uri = 'https://%s/mgmt/cm/autodeploy/software-image-downloads/%s' % (host, filename)
requests.packages.urllib3.disable_warnings()
with open(fp, 'wb') as f:
start = 0
end = chunk_size - 1
size = 0
current_bytes = 0
while True:
content_range = "%s-%s/%s" % (start, end, size)
headers['Content-Range'] = content_range
#print headers
resp = requests.get(uri,
auth=creds,
headers=headers,
verify=False,
stream=True)
if resp.status_code == 200:
# If the size is zero, then this is the first time through the
# loop and we don't want to write data because we haven't yet
# figured out the total size of the file.
if size > 0:
current_bytes += chunk_size
for chunk in resp.iter_content(chunk_size):
f.write(chunk)
# Once we've downloaded the entire file, we can break out of
# the loop
if end == size:
break
crange = resp.headers['Content-Range']
# Determine the total number of bytes to read
if size == 0:
size = int(crange.split('/')[-1]) - 1
# If the file is smaller than the chunk size, BIG-IP will
# return an HTTP 400. So adjust the chunk_size down to the
# total file size...
if chunk_size > size:
end = size
# ...and pass on the rest of the code
continue
start += chunk_size
if (current_bytes + chunk_size) > size:
end = size
else:
end = start + chunk_size - 1
Now you know how to upload and download files. Let’s do something with it!
A Use Case - Upload Cert & Key to BIG-IP and Create a Clientssl Profile!
This whole effort was sparked by a use case in Q&A, so I had to deliver the goods with more than just moving files around. The complete script is linked at the bottom, but there are a few steps required to get to a clientssl certificate:
Upload the key & certificate
Create the file object for key/cert
Create the clientssl profile
You know how to do step 1 now. Step 2 is to create the file object for the key and certificate. After a quick test to see which file is the certificate, you set both files, build the payload, then make the POST requests to bind the uploaded files to the file object.
Notice we return the key/cert filenames so they can be used for step 3 to establish the clientssl profile. In this example, I name the file object and the clientssl profile to the name of the certfilename (minus the extension) but you can alter this to allow the objects names to be provided. To build the profile, just create the payload with the custom key/cert and make the POST request and you are done!
Much thanks to Tim Rupp who helped me get across the finish line with some counting and rest worker errors we were troubleshooting on the download function.
Hi, thx for very nice article, but I am not able to run it.
Whatever I try I always get 400 as a response and no file, or part of file is copied to LB. I am not able to find what can be wrong. I tried both scripts: "Upload a File" and "Upload Cert/Key & Build a Clientssl Profile". Do you have any clue what I might have wrong? Thanks, Zdenek.
as an addition asm file upload is available as well (not sure about the version minimum)
https://%s/mgmt/tm/asm/file-transfer/uploads/%s
file upload will be done in /ts/var/rest/
and then policy can be imported with :
POST /mgmt/tm/asm/tasks/import-policy
{
"filename":""
"name":""
}
Has someone tested this with token-based authentication? (User is mapped to Administrator)
Line 20 in create_cert_obj throwing:
{"code":400,"message":"Unable to copy (/var/config/rest/downloads/test-vip.crt) into tempfile (/var/system/tmp/tmsh/yrfD5H/test-vip.crt), Permission denied","errorStack":[],"apiError":26214401}
test-vip.crt has root:root 600 permission after upload. Changing file-mode is no option, cause it throws:
{"code":400,"message":"Key management library returned bad status: -4, Invalid Parameter","errorStack":[],"apiError":26214401}
I'll retest with token auth and let you know. I'll also be updating methods from /sys/crypto to /sys/file/ssl-key and /sys/file/ssl-cert as well. It'll be Monday before I have a chance. I'm on PTO today.
Yes, sorry for the delay. With the latest version of the sdk (2.0) I can successfully upload a file that I verified in the BIG-IP's /var/config/rest/downloads directory with remote authentication using tokens (i am testing on 12.1.) user_admin/letmein00 are only defined in my tacacs test server.
thanks for quick response. Uploading is no problem - but when accessing the uploaded file with the same credentials to install it to the f5 key-cert-store it fails (see code 400 error in my previous post). Can you please have a look on this?
ok I'll test with that, but will likely be in the morning. I think I saw the same behavior with tokens when I tested last. You might open a case as well.
Feel free to reach out off-site at j -dot- rahm -at- f5 -dot- com and I can work with you directly on this.
Hi Christian, it doesn't seem to be a problem with tokens, as I can use tokens with the built-in accounts just fine, but if I'm using remote-auth and tokens, that fails to permissions issues. Did you get a case open on this?
This was very helpful. I ended up having my script build a pkcs12 file and import that, since otherwise I couldn't upload a new key/cert pair if the key changed without getting "profile X's key and certificate do not match".
After the file upload then POST to /mgmt/tm/sys/crypto/pkcs12 with a payload of:
Hi all!
This post was very helpful indeed. Also the hint with the central authentication and the requirement to upgrade to 12.1.2.
However, I probably need another little hint...
When I upload a text-file (haven't tried any binary data yet) to the BigIP with the method described above, the file has a leading 0-byte.
Also the original "\n" seems to be trimmed off, which isn't a problem in this case. Just mentioning it for the sake of completeness.
I tried to use Content-Type: text/plain instead of "application/octet-stream", I tried binary-transfer, nothing seems to help.
What am I missing here?
Thanks for your help!
regards, Matt
update:
trace-ascii shows that curl seems to be sending the post data correctly
The following command does NOT work correctly, but DOES follow RFC 7233 4.2:
curl -i -sk -u admin:admin -X POST -H "Expect:" \
-H "Content-Type: application/octet-stream" \
-H "Content-Range: bytes 0-1333/1334" --data-binary "@domain.crt" \
";
The main difference is the use of the HTTP Header
Content-Range
including the word "bytes" with the values. RFC 7233 4.2 indicates that this should be (example from RFC)
Content-Range: bytes 42-1233/1234
However whenever we include the word "bytes" the F5 responds with a HTTP 400, error follows:
Is it possible to install PKCS 12 cert through Python SDK?
I don't see any interface under bigip.tm.sys.cryto.py like certs and keys.
So I tried to use the format where Chris Hiner used above but it didn't work.
A little bit confused.
Then, is there any difference using bigip.tm.sys.crypto.keys.exec_cmd() and b.tm.sys.file.ssl_certs.ssl_key.create()? It seems both install a SSL key from the local key file (/var/config/rest/downloads/)
Also, what if I need to specify a passphrase (password) for pkcs12? Simply add another parameter like the following?
Hi @F5_Digger, the sys/crypto endpoint is still there but deprecated, permissions are not guaranteed to work with that, but should with sys/file/ssl_*.
Yes, the passphrase parameter should be passed if specified.
In TMOS v15 the directory to retrieve uploaded files has obviously changed.
You will find uploaded files under /var/config/rest/downloads/tmp/ now.
Here are Ansible tasks to upload a certificate, to import it from the temp directory to TMOS filestore and to delete the file from the temp directory afterwards:
- name: set certificate path information
set_fact:
crt_file_path: "{{ '%s%s_%s_%s.%s' | format(crt_path,crt_prefix,crt_name,crt_suffix,crt_file_extension) }}"
crt_file_name: "{{ '%s_%s_%s.%s' | format(crt_prefix,crt_name,crt_suffix,crt_file_extension) }}"
- name: register cert file properties
stat:
path: "{{ crt_file_path }}"
register: crt_properties
when: crt_file_path is defined
- name: set certificate file size information
set_fact:
crt_file_size: "{{ crt_properties.stat.size }}"
when: crt_properties is defined
- name: copy certificate to temp directory
uri:
validate_certs: no
url: https://{{ inventory_hostname }}/mgmt/shared/file-transfer/uploads/{{ crt_file_name }}
method: POST
headers:
Content-Range: "0-{{ crt_file_size | int - 2 }}/{{ crt_file_size }}"
Content-Type: "application/octet-stream"
X-F5-Auth-Token: "{{ device_info[inventory_hostname].token }}"
body: "{{ lookup('file', crt_file_path) }}"
- name: copy certificate to TMOS filestore (TMOS v14+)
uri:
validate_certs: no
url: https://{{ inventory_hostname }}/mgmt/tm/sys/crypto/cert
method: POST
headers:
X-F5-Auth-Token: "{{ device_info[inventory_hostname].token }}"
body_format: json
body:
command: install
name: "{{ '%s_%s_%s' | format(crt_prefix,crt_name,crt_suffix) }}"
from-local-file: "/var/config/rest/downloads/tmp/{{ '%s_%s_%s.%s' | format(crt_prefix,crt_name,crt_suffix,crt_file_extension) }}"
- name: cleanup certificate from temp directory
uri:
validate_certs: no
url: https://{{ inventory_hostname }}/mgmt/tm/util/bash
method: POST
headers:
X-F5-Auth-Token: "{{ device_info[inventory_hostname].token }}"
body_format: json
body:
command: run
utilCmdArgs: "-c 'rm /var/config/rest/downloads/tmp/{{ '%s_%s_%s.%s' | format(crt_prefix,crt_name,crt_suffix,crt_file_extension) }}'"
You may want to replace the variables in the code snippet above according to your requirements.
If I want to upload/download I guess the only way to do that is with an admin account I tried with a resource-admin account and no matter what I do I always get:
Would love to have an answer on this as well, the need for admin rights to upload certs. We have a shared F5 where a customer want's to automate cert updateing. But rights are now set by partition. But they need admin rights for uploading files which is a no go, they are only allowed to edit there own partition. They are resource admin on the F5 for there partition and can import certs without issues via the GUI. Why isn't it possible via the rest-api?
Running latest 15.1.6
AS3 is not an option at this time. And that is also missing alot of features that still doesn't make it very usable. For an other F5 we have to do API call's to BIG-IQ/BIG-IP and AS3 to get things working. As F5 is not willing to fix things, so only AS3 can be used.
@EmmanuelCR and @Axel_Boersma, an admin account is required for most iControl REST methods, but there is a workaround where you can assign RBAC to other user accounts. Links to a couple articles that should help pursue that in this post.
I went through the forum but did not find any PowerShell-baed implementation on how to download a UCS file. For some reason, I can't translate Perl to PowerShell and even ChatGPT can't 🙂
I am going to try to build something on my own but will be happy to see a 100% working solution.
Btw, right now I discovered that archive downloaded via GUI has a different size from the /mgmt/shared/file-transfer/ucs-downloads Invoke-WebRequest - 6000 bytes difference. Why? How? Going to dig deeper...
Hi @Roman_Melekh, Joel Newton definetly shared a powershell version, but I can't seem to find it. You might drop a comment on his powershell module in the codeshare to see if he's still listening...it's been a while.
I know about this one. It is very old, outdated and does not have the functions that I need. It looks like building my own approach will be simpler. Working on that.
