API Calls to F5 limited to 1024 KB download

[saidshow]

Hi,

I am interacting with the F5 API in order to download ASM policies for the purpose of automating the backups. The process works fine however policies larger than 1024 KB are cut off at this size of 1024 KB. Initially I suspected that there was a default limit on the curl request however I have not been able to find information on how to increase this with the curl request.

Is this a limitation on the F5 API or the Curl request? wget is not an option as this is not natively supported on the F5 virtual appliance. My script lives on the appliance, downloads the relevant policies and then pushes them to a SMB share. The only issue is that the ASM policies that are larger than 1024 KB are being cut off at 1024 KB.

The API calls are as per the documentation here:

http://cdn.f5.com/websites/devcentral.f5.com/downloads/icontrol-rest-api-user-guide-13-0-0.pdf

specifically:

GET https://x.x.x.x/mgmt/tm/asm/policies

POST https://x.x.x.x/mgmt/tm/asm/tasks/export-policy

GET https://x.x.x.x/mgmt/tm/asm/file-transfer/downloads/$asmPolicy

Excluding the processing in my script the API calls I make are shown below: I expect the issue resides in the download api call. Is there a switch I can add to increase this limit?

curl -ku 'username:password' -X GET https://x.x.x.x/mgmt/tm/asm/policies | jq '.items[] | "pol_name:" + .name + ";api_id:" + .id' >> $wdir/asmDetails.txt
 
 
curl -ku 'username:password' -X POST https://x.x.x.x/mgmt/tm/asm/tasks/export-policy -H 'Content-Type: application/json' -d '{"filename":"'$asmPolicy'","policyReference":{"link":"https://localhost/mgmt/tm/asm/policies/'$asmIDs'"}}'
 
 
curl -ku 'username:password' -X GET https://x.x.x.x/mgmt/tm/asm/file-transfer/downloads/$asmPolicy > $wdir/asmBackup/"$folderName"/$number-$asmPolicy-$hostname-"$dateStamp".xml
 
 

Thanks

[Satoshi_Toyosa1]

See Demystifying iControl REST Part 5: Transferring Files.

It says: "Any file larger than 1M needs to be chunked with this header as that limit is enforced by the worker. "

[saidshow]

Thanks for the reply. I'm working in bash at the moment, from what I have seen, I expected to receive a 206 web response when these policies are cut off at 1024KB however I still receive 200's even then. I was planning on looping based on this web response until I received a 200... I see no other real difference in the data other than content-length...

[Satoshi_Toyosa1]

I have reproduced the symptom: 200 OK is received even the file data is partially transferred.

The following curl call against the file 'large.xml' (12,000,000 bytes) responded with 1 MB (1024 * 1024 bytes) only.

curl -D x -sku admin:<pass> https://<host>/mgmt/tm/asm/file-transfer/downloads/large.xml | wc
      0       1 1048576
 
HTTP/1.1 200 OK
...
Content-Length: 1048576

The behaviour is different from the other downloading endpoint. For example,

curl -D x -sku admin:<pass> https://<host>/mgmt/cm/autodeploy/software-image-downloads/BIGIP-12.1.2.0.0.249.iso -H "Content-Type: application/octet-stream" | wc
   3687   13306 1048576 
 
HTTP/1.1 206 Partial Content
Content-Length: 1048576
Content-Type: application/octet-stream
Content-Range: 0-1048575/2011930624

As you can see, the call returns the first 1 MB along with the 206 Partial Content. You can see the size and the location of the partial content from the Content-Length and Content-Range headers.

The location of the above file transfer (download) is /shared/images. If you want to utilize the 206 response and Content-xxx headers, you can move the asm xml file to that location (use /mgmt/tm/util/unix-mv). Note that the asm file name contains additional user information: e.g., instead of just asmPolicy.xml, it is admin~asmPolicy.xml, so specify the name accordingly in the calls.

Again, please refer to Demystifying iControl REST Part 5: Transferring Files.

[saidshow]

Hi , thank you again for replying. I too have been able to validate that the /software-image-downloads/ endpoint gives the 206 web response when files are larger than 1M while the /downloads/ gives only the 200 even when the file size exceed 1M.

The missing part for me is how to move the files. The "export-policy" api call does not provide a means to specify a different location other than the /downloads/ location that is not the ideal endpoint.

As per your suggestion, I have attempted to use the /mgmt/tm/util/unix-mv api call however I can't find details on what parameters this api call requires. As an example of what I have been trying see below.

curl -ku 'admin:<passwd>' -H 'Content-Type: application/json' -X POST https://x.x.x.x/mgmt/tm/util/unix-mv -d '{"name":"/mgmt/tm/asm/file-transfer/downloads/policyName","target":"/shared/images/policyName"}}'

This results in the following error: "{"code":400,"message":"Found invalid JSON body in the request.","errorStack":[],"apiError":1}" the api document does not show any information on this api call.

Any detail on moving the files would be greatly appreciated, this appears to be the missing part at the moment. Since I can get a 206 web response and specify the byte-range I will be able to loop around the file once I get them in the /shared/images location.

Is there any drawback placing the files in the /shared/images location? eg. Can it affect the GUI? I have not seen any issues in the GUI after creating some test files in this location.

[Satoshi_Toyosa1]

I know. The call is tricky. The API for 'tmsh run util' is shown in the iControl REST User Guide Version 14.1 (PDF). Grab the file and look for the section "Using the run command".

For example, to move /tmp/sat (on the Unix filesystem) to /tmp/sat2, run this.

curl -sku admin:<pass> https://<host>/mgmt/tm/util/unix-mv \
 -X POST -H "Content-Type: application/json" \
 -d '{"command":"run", "utilCmdArgs": "/tmp/sat /tmp/sat2"}'

[saidshow]

You're a champion . That works. Thank you so much. .

I had a working script to automate the asm policy backups - the only catch was that it did not work on files over 1MB. Now to support the couple files that are over 1MB I know what I need to change.

1. export policies
2. move policies to /shared/images
3. within my current download loop, add a nested loop that will check the web response code and adjust the byte range as required.

I'll park this shortly as I have spent more time than intended on this hold up. I will post my solution once i get it working. Thanks again for your assistance.

[Satoshi_Toyosa1]

>>> Is there any drawback placing the files in the /shared/images location?

To my knowledge, none (other than the disk space issue).

The /shared/images directory is typically used for storing the BIG-IP iso images. If you have stored a bogus ISO 9660 file with the extension 'iso', it will show up in System > Software Management > Image List on GUI or 'tmsh list sys software images'. Since your ASM file is an XML, they should be benign. If you are super disk-space conscious, you can of course remove the file any time by calling the /mgmt/tm/util/unix-rm endpoint (specify the name of the file in the "utilCmdArgs" property).

[saidshow]

One last question - any idea where in the file system files are placed with the /mgmt/tm/asm/tasks/export-policy command?

[Satoshi_Toyosa1]

Empirically, exported policy files are stored under /var/ts/var/rest. The file name contains additional information such as the user name or dates. For example, when a file is created by POSTING {"filename":"admPolicy.xml"}, the file name becomes "admin~asmPolicy.xml". The file name has different format when created via GUI. I learnt this from the 'find' Unix command on my 13.1. Please let me know if you can't find your file there.

[saidshow]

Thanks again . I tried the find command however it was coming up blank - there seems to be a clean up taking place. I was able to verify the location of /var/ts/var/rest by specifying the file by the MD5 hash in the export-policy command - running this only once and then listing the contents of this directory.

The naming is as you have noted:

username~policyname

With the naming convention like this, cutting for the f2 after the ~ delimeter will give me a clean policy name.

[saidshow]

  - thank you so very much for your time. My script is now working and able to easily handle files above 1MB.

Originally, I wrote this script to sit on an external linux box that would interact with the API and do everything. Due to concerns around credentials in the script, I decided to move the script to the appliance. This way, if the script is discovered no new privileges are gained since the user finding the file already needs access to the F5's advanced shell.

I was actually quite confined to my original thinking in that I needed to do everything with the API. While I do still need 2 API calls, I was able to swap out 2 api calls for bash. This eliminates the 1MB limit entirely.

By following what you provided I was able to find the answer to another issue I had. When I ran my script for the first time, the downloads would be empty, when I ran it a second time etc, all was well. When I was watching the /var/ts/var/rest location I could see that the policy export actually took a few seconds - so when the download did work, it was the previous policy export command that had completed. I placed a sleep in my loop to cater for this and this part is sorted.

I have a significantly better understanding of how all of this fits together now thanks to your assistance. It is greatly appreciated.

I will clean up my solution, perform some testing and provided it here once ready. I do expect others will benefit greatly from your input here. Thank you again!

[Satoshi_Toyosa1]

Informational: I found a related DevCentral question:

Error code 400 while downloading a file via REST API.

[Satoshi_Toyosa1]

I found my files disappeared (including bogus test files I created) from the /var/ts/var/rest directory too. I haven't been able to find the process responsible for deletion .

[saidshow]

I am trying to post my solution but get an error from the devcentral portal that reads "Maximum number of topic assignments have been exceeded"

[saidshow]

  - my working solution with creds removed is shown below. You were CRUCIAL to getting this to work. Thank you so very much!!!

Unfortunately I get an error when trying to post this as code... hopefully users can zoom in on the image below.

I have an iCall period handler kicking off an iCall script that executes this bash file. Note, this does push the exported asm policies in xml format to an external SMB share and forces a cleanup.

In the event that the unix box has rm pointing to rm -i, I specify /bin/rm to prevent prompting for the clean up.

I will leave this run in TEST every 15 mins for an hour, when I am comfortable all is well, I will have it run every 28 days at 2am starting on Sunday. This is controlled with the iCall period handler.

[JG]

The underlying purpose of the question has been served by a work-around, and thanks for the sharing of the script.

However, the subject issue of curl failing to download beyond the first 1024KB remains.

In my tests, specifying the "Range:" header in the client request was also not successful; the server always returned the first 1024KB no matter what byte range was requested.

[Edit]

Satoshi Toyosawa has just clarified (in his linked post above) that "Content-Length" should be used rather than "Range:", as this is how F5 implemented it.