Python SDK UCS download takes long time

[asalamatov]

Hello guys,

Did anybody implement file backups using Python SDK? UCS creation works fine for me, however file transfer takes forever.

I'm using example at https://f5-sdk.readthedocs.io/en/latest/userguide/file_transfers.html

I does create local file on my PC, but takes forever (20-40 mins) to download it. I'm on BIG-IP 13.1.1.4 Build 0.0.4

Looking for real world experience from folks.

[Samir]

I have observed delay while downloading via Python SDK to my PC. but all depend on the bandwidth speed.

Try to download UCS/Qkview to same DC where LB reside.

[asalamatov]

Could you share your code? I'm just curious if I'm missing something.

[Satoshi_Toyosa1]

Python SDK or iControl REST, which is the base of the SDK, is not optimal for downloading a large file.

This is because there is a 1 MB size limitation in downloading or uploading a file. Any file larger than the limit must be chunked (split into small pieces and concatenate later on the receiving end) manually. See also Demystifying iControl REST Part 5: Transferring Files.

Python SDK does the chunking for you (0.5 MB per chunk). Due to the chunking, it will create a numerous calls (including the one that gets the information on the file size) for a single file retrieval. For example, for a 1 GB file, it calls the iControl REST endpoint 1,910 times. That is not exactly a cheap overhead. In my test, it took 12s for scp and 92s for Python for a 1 GB file.

Another factor that may affect file downloading performance is the system load. Since Python SDK interacts with a various components on the box, any resource contention (such as CPU and memory) would slow the job down (I guess 20-40 min in your case is this). Avoiding the busy time would be a good option.

If you need speed and do not need to use iControl REST, scp (secure copy) is a good alternative solution for file downloadin.

[asalamatov]

Thanks for your detailed answer.

I noticed that using SDK https://f5-sdk.readthedocs.io/en/latest/userguide/file_transfers.html to download UCS file takes lots of time (almost an hour for 270MB file), but when using f5-backup script https://devcentral.f5.com/s/articles/f5-remote-backup-python-script-1163 it's almost instant. Looking into SDK code I see that it's based on f5-backup script, but can't figure out why is download speed varies so much?

I also created an issue on GitHub for this. Just trying to figure out if there is a bug in SDK code? https://github.com/F5Networks/f5-common-python/issues/1548

[Satoshi_Toyosa1]

I cannot tell why one is faster than the other as I did not see much difference, at least on the iControl REST calls. Both access the same endpoint (/mgmt/shared/file-transfer/ucs-downloads) and do 512 kb chunking. Possibly the F5 SDK performs additional verification or throttling but I am not sure (I haven't read the code). I hope you can get information from the Github.