icontrol - different exported policy via icontrol and GUI

[JustJozef]

Hi,

I trying to export security policy with icontrol but when I compare output with same policy exported via GUI it's different.

/mgmt/tm/asm/tasks/export-policy/", '{"filename": "test_api_new.xml", "minimal":"true", "format":"xml", "policyReference":{"link": "https://localhost/mgmt/tm/asm/policies/' + str(policyId) + '"}}

icontrol exported policy 19684 lines
GUI export has 52857 lines.

For example GUI export contains info about open_api_files but icontrol export not.

<open_api_files>
    <open_api_file>
      <filename>rest-api-security~swagger.json</filename>
      <upload_datetime>2022-07-01T14:01:31Z</upload_datetime>
    </open_api_file>
  </open_api_files>

Should be policy be same? Does not matter from where I export it?

In addition is there detail documentation for icontrol? Currently I use https://cdn.f5.com/websites/devcentral.f5.com/downloads/icontrol-rest-api-user-guide-15-1-0.pdf

I found some options from responses from the requests but maybe I missed some important options?

Thank you.

[JRahm]

I don't know much about the ASM-specific iControl methods, but you might try changing:

minimal: true to minimal: false

and see if that helps.

[JustJozef]

Nope. I tried this at the beggining but still different outputs were generated. Anyway seems that JSON is exported correctly. Same export is from GUI and API.

[JRahm]

so GUI has a lot more metadata around the policy, but the actual policy objects are the same? 

[Piotr_Badach]

There is a limit of 1,048,576 bytes for exporting ASM policies through API. This is the max size returned in a single chunk and the remaining data is truncated.

There is a way to get the rest of the policy using a Content-Range header. I tested this in the lab.
Unfortunately you would need to merge the files manually then.

Policy exported

# restcurl -u admin:admin -X POST https://localhost/mgmt/tm/asm/tasks/export-policy -d '{"filename":"sp_api-gw.xml","policyReference":{"link":"https://localhost/mgmt/tm/asm/policies/_zFj4JiaUN79SDubDLofpQ"}}'
{
"isBase64": false,
"inline": false,
"minimal": false,
"status": "NEW",
"lastUpdateMicros": 1.551281842e+15,
"includeVulnerabilityAssessmentConfigurationAndData": true,
"kind": "tm:asm:tasks:export-policy:export-policy-taskstate",
"selfLink": "https://localhost/mgmt/tm/asm/tasks/export-policy/o11Oq-hmzdWYemX_tBVPdw?ver\u003d13.1.1",
"format": "xml",
"filename": "sp_api-gw.xml",
"policyReference": {
"link": "https://localhost/mgmt/tm/asm/policies/_zFj4JiaUN79SDubDLofpQ?ver\u003d13.1.1"
},
"id": "o11Oq-hmzdWYemX_tBVPdw",
"startTime": "2019-02-27T15:37:22Z"
}

Status checked

# restcurl -u admin:admin -X GET https://localhost/mgmt/tm/asm/tasks/export-policy/o11Oq-hmzdWYemX_tBVPdw?ver\u003d13.1.1
{
"isBase64": false,
"inline": false,
"minimal": false,
"status": "COMPLETED",
"lastUpdateMicros": 1.551281856e+15,
"includeVulnerabilityAssessmentConfigurationAndData": true,
"kind": "tm:asm:tasks:export-policy:export-policy-taskstate",
"selfLink": "https://localhost/mgmt/tm/asm/tasks/export-policy/o11Oq-hmzdWYemX_tBVPdw?ver\u003d13.1.1",
"format": "xml",
"filename": "sp_api-gw.xml",
"policyReference": {
"link": "https://localhost/mgmt/tm/asm/policies/_zFj4JiaUN79SDubDLofpQ?ver\u003d13.1.1"
},
"endTime": "2019-02-27T15:37:37Z",
"id": "o11Oq-hmzdWYemX_tBVPdw",
"startTime": "2019-02-27T15:37:22Z",
"result": {
"fileSize": 1124056 <========= total size
}
}

File downloaded (truncated)

# restcurl -u admin:admin -X GET https://localhost/mgmt/tm/asm/file-transfer/downloads/sp_api-gw.xml > /var/tmp/sp_api-gw.xml

# ls -la /var/tmp/sp_api-gw.xml
-rw-r--r--. 1 root root 1048577 2019-02-27 07:42 /var/tmp/sp_api-gw.xml <=== actual size

The rest of the policy downloaded using Content-Range header.

# curl -sku admin:admin -H "Content-Range: 1048576-1124055/1124055" -X GET https://localhost/mgmt/tm/asm/file-transfer/downloads/sp_api-gw.xml >> /var/tmp/sp_api-gw.xml

But then the last part is added starting from new line.

<signature signature_id="200101106">
<enabled>true</enabled>
<in_staging>false</in_staging>
</signature>
<signature signature_id="200 <======
101107">
<enabled>true</enabled>
<in_staging>false</in_staging>
</signature>

I think the best option would be to save policy parts to separate files and then combine them manually.

You can try also exporting in json format by adding the "format":"json" to the task request.

It may give you some lower size output.

# curl -ksu admin:admin -X POST https://localhost/mgmt/tm/asm/tasks/export-policy -d '{"filename":"test_policy_export.json","format":"json","policyReference":{"link":"http://localhost/mgmt/tm/asm/policies/MrLpFzRHNarvj_zuAOD0fw"}}'

[JRahm · Accepted solution]

Tim Rupp helped me write a file upload/download function for the rest interface years ago. Here's the mixin for the python sdk that was refined a lot on how to manage the data with the range header.

https://github.com/F5Networks/f5-common-python/blob/development/f5/bigip/mixins.py#L303-L395

[JustJozef]

Thank you for link. Will check the script and test it.

[JustJozef]

Thank you for advice. I adapted script for my purpose and export function provide same result as GUI. As export-policy function provide directly size of the file I could adapt download to exact size with specific number of chunks.