Forum Discussion

Marc_64323's avatar
Marc_64323
Icon for Nimbostratus rankNimbostratus
Mar 24, 2014

SOAP File transfers for backup .ucs files result in unreadable files that are bigger than they should be

Greetings!

 

I am trying to automate our backup process for .ucs files remotely. Using bigsuds with the System.ConfigSync.download_configuration() method I am getting strange results, in that not only is my resulting file unreadable locally or through re-upload to the BIGIP device, but it is also significantly larger than I am expecting it to be. For example, my "backup.ucs" file local to the BIGIP device might be 2.6 Mb; however, at the conclusion of my write loop, the resulting file size would be 3.5 Mb remotely. I am admittedly new to SOAP's idiosyncrasies with downloading files, and I tried to follow the example pseudo code I found for managing loops to handle file chunks and offsets. My "download progress" output shows that the bytes written are actually pretty close to what I am expecting (though a multiple of the chunk size), but then the resulting file is much larger and corrupt, and I have no idea why. Perhaps I am not handling the file handle properly because of SOAP weirdness? Any help would be greatly appreciated.

 

Here is the code:

 

    !/usr/bin/python

    import bigsuds
    import datetime

    temp_date = str(datetime.datetime.utcnow())
    split_date = string.split(temp_date)
    my_date = split_date[0]
    bigip_ip = '***.***.***.***'
    bigip_user='***********'
    bigip_pwd='************'
    chunk_size = 64 * 1024
    file_offset = 0
    write_continue = 1

    create filename
    filename = 'test_' + date

    instantiate connection to bigip
    obj = bigsuds.BIGIP(hostname=bigip_ip, username=bigip_user, password=bigip_pwd)

    first save config file on bigip device
    obj.System.ConfigSync.save_configuration(filename,'SAVE_FULL')

    open target file for writing
    f = open('/home/bigip_backups/' + filename + '.ucs','a')

    download data to file
    while write_continue == 1:
        temp_config = obj.System.ConfigSync.download_configuration(filename + '.ucs',chunk_size,file_offset)
        file_info = temp_config['return']               
        f.write(file_info['file_data'])

        detect EOF                                    
        if file_info['chain_type']  == 'FILE_LAST' or file_info['chain_type'] == 'FILE_FIRST_AND_LAST':
            write_continue = 0

        set offset                    
        file_offset = file_offset + chunk_size

        track download progress
        print str(file_offset) + " bytes written"

        DEBUG
        print file_info['chain_type']

    cleanup
    f.close()
application delivery
availability
bigsuds
devops
iControl
LTM
management
  • Joe_Pruitt's avatar
    Joe_Pruitt
    Mar 25, 2014

    The only thing I think is different with your code and the ones I wrote in the iControl CodeShare for ConfigSync is that the 3rd parameter in the call to download_configuration for file_offset is actually an IN/OUT value. The method updates the value with the original file_offset plus the length of the data returned in the FileTransferContext return value.

     

    I'm not sure how bigsuds handles in/out variables. In perl/SOAP::Lite you must access it via the paramsout variable. https://devcentral.f5.com/wiki/iControl.PerlConfigSync.ashx.

     

    In PowerShell (https://devcentral.f5.com/wiki/iControl.PsConfigArchiving.ashx) the code looks like this:

     

    while($bContinue -eq 1)
{
  $ctx = (Get-F5.iControl).SystemConfigSync.download_configuration($config_name, $chunk_size, [ref]$file_offset);
  $w.Write($ctx.file_data, 0, $ctx.file_data.Length);

  Write-Host "Bytes Transferred: $file_offset";
  if ( ($ctx.chain_type -eq "FILE_LAST") -or ($ctx.chain_type -eq "FILE_FIRST_AND_LAST") )   {
    $bContinue = 0;
  }
}

    You'll see that in there, the value returned in file_offset is what's passed into the next call.

     

    For debugging on your end, I'd print to the console the values of file_offset and chunk_size before each call and make sure that fill_offset is incrementing by chunk_size correctly each it. So, if chunk_size is 100, the output should looks something like:

     

    0, 100
99, 100
199, 100
299, 100
...

    Hope this helps...

     

    -Joe

     

  • Sparks_86637's avatar
    Sparks_86637
    Icon for Nimbostratus rankNimbostratus
    Aug 01, 2014

    Hi,

    Guessing you've got this working by now, but for anyone else who's interested I got Marc's script to work by simply decoding the file_data as base64. Updated version below is exactly the same except for a few lines....

        instantiate connection to bigip
obj = bigsuds.BIGIP(hostname=bigip_ip, username=bigip_user, password=bigip_pwd)

first save config file on bigip device
obj.System.ConfigSync.save_configuration(filename,'SAVE_FULL')

open target file for writing
f = open('/home/bigip_backups/' + filename + '.ucs','wb')  <---- This line

download data to file
while write_continue == 1:
    temp_config = obj.System.ConfigSync.download_configuration(filename + '.ucs',chunk_size,file_offset)
    file_info = temp_config['return']               
    f.write(base64.b64decode(file_info['file_data']))  <---- And this line

    detect EOF                                    
    if file_info['chain_type']  == 'FILE_LAST' or file_info['chain_type'] == 'FILE_FIRST_AND_LAST':
        write_continue = 0

    set offset                    
    file_offset = file_offset + chunk_size

    track download progress
    print str(file_offset) + " bytes written"

    DEBUG
    print file_info['chain_type']

cleanup
f.close()

    Thanks Marc; your work saved me a lot of time!

    Cheers, Stephen