pyControl download_file

Wow, this twisted my brain for a few minutes, but fortunately it turned out to be pretty easy. So here's what is going on: one of the attributes that's being set on the response data is named 'return', which is obviously a python keyword! To get the data, do this:

 The return data is base64 encoded so we'll need to decode it to see anything useful.
import base64
ret = cs.download_file(file_name='/var/tmp/foo.txt',chunk_size=65536,file_offset=0)

Now, if you look at that 'ret' object, you'll see that it's got two attributes set: ret.file_offset, and ret.return. To get a that second guy, do this:

fdata = getattr(ret,'return').file_data
print base64.b64decode(fdata)

So once you do the b64decode you can write it out like a normal file, etc.

Hope this helps,

-Matt

 

Hi Matt,

 

 

I am just getting around to playing with this again and am hoping you can help me figure out the rest. Basically I figured out the response is chunked so I need to figure out how to put it back together. Here is what I have so far, but it doesn't quite work.

 

 

file = open('c:\\file1.ucs', 'w')

 

ctype = "null"

 

foffset = 0

 

 

while (ctype != "FILE_LAST" ) or (ctype != "FILE_FIRST_AND_LAST"):

 

ret = b.System.ConfigSync.download_file(file_name = "/var/local/ucs/sat.ucs", chunk_size = 65536, file_offset = foffset)

 

fdata = getattr(ret,'return').file_data

 

foffset = getattr(ret, 'file_offset')

 

foffset = foffset

 

ctype = getattr(ret,'return').chain_type

 

unencode = base64.b64decode(fdata)

 

file.write(unencode)

 

file.close()

 

Steve - I will look at this tomorrow and see what I find. How big is the ucs?

 

Matt

 

 

No rush...this just something I am working when I get time. The UCS is rather small as it is just from the VE I run on my laptop.

Steve: Wow this is odd - I'm getting some really puzzling results, all surrounding the chunk sizing. I can pull in and unpack an archive (mostly), but it's not complete. Here's the code, in the hopes that you or someone else can see an obvious error in my ways. If I tickle that chunk_size variable to 8k, 16k, 32, 64, etc. they all seem to return an different final file size. Weird.

 

!/bin/env python
import base64
import array
import binascii


def pc_downloader(b,remote_file,local_file):

    '''
    Handles downloading files via System.ConfigSync
    Args:
    b -- a pycontrol client object
    remote_file -- the name of the file to fetch from BigIP
    local_file -- the local file to write to.
    '''

    dl = b.System.ConfigSync
    local_file = open(local_file, 'ab')
    poll = True
    chunk_size = 8*1024
    foffset = 0
    a = array.array('c')

    while poll:
        res = dl.download_file('/var/tmp/test-ucs.tgz',chunk_size = chunk_size,file_offset = foffset)
        fdata = getattr(res,'return').file_data
        foffset = long(res.file_offset)
        chain_type = getattr(res, 'return').chain_type

        if (chain_type == 'FILE_LAST') or (chain_type == 'FILE_FIRST_AND_LAST'):
            poll = False
            local_file.close()
        else:
            local_file.write(binascii.a2b_base64(fdata))

if __name__ == '__main__':

    import pycontrol.pycontrol as pc
    b = pc.BIGIP(hostname='192.168.1.245',username='admin',password='admin',fromurl=True,wsdls=['System.ConfigSync'])
    pc_downloader(b,'/var/tmp/testucs.tgz','/var/tmp/testucs.tgz')

 

Give this a try and see if you get the same results. If so, I think it's worth opening a case.

 

-Matt

Maybe you all need to pick a "real" programming language! B-)

This is a tricky function because the 3rd "in-out" parameter file_offset. I see a couple of things, that may or may not be a problem (due to my ignorance with Python).

1. The file_name parameter is not named in the call to download_file().

res = dl.download_file('/var/tmp/test-ucs.tgz',chunk_size = chunk_size,file_offset = foffset)

Shouldn't this read:

res = dl.download_file(file_name = '/var/tmp/test-ucs.tgz',chunk_size = chunk_size,file_offset = foffset)

Maybe that is handled underneath in pyControl somewhere.

2. I'd check to make sure that the foffset value is equal to the previous value plus the chunk size (or the difference between the previous offset and the end of file if it's the last request in the chain. In my examples, I throw in a print statement after the writing to file with the current file offset.

I just tested this Perl function on my 10.2 LTM and it worked regardless of the chunk size resulting in identical files.

sub downloadFile()
{
  my ($configName, $localFile, $quiet) = (@_);
  my $success = 0;

  if ( "" eq $localFile )
  {
    $localFile = $configName;
  }

  if ( "" eq $configName )
  {
    &usage("download");
  }

  open (LOCAL_FILE, ">$localFile") or die("Can't open $localFile for output: $!");
  binmode(LOCAL_FILE);

  my $file_offset = 0;
  my $chunk_size = 1024*64;
  my $chain_type = $FILE_UNDEFINED;
  my $bContinue = 1;

  print "\n";
  while ( 1 == $bContinue )
  {
    $soap_response = $ConfigSync->download_file
    (
      SOAP::Data->name(file_name => $configName),
      SOAP::Data->name(chunk_size => $chunk_size),
      SOAP::Data->name(file_offset => $file_offset)
    );

    if ( $soap_response->fault )
    {
      if ( 1 != $quiet )
      {
        print $soap_response->faultcode, " ", $soap_response->faultstring, "\n";
      }
      $bContinue = 0;
    }
    else
    {
      $FileTransferContext = $soap_response->result;
      $file_data = $FileTransferContext->{"file_data"};
      $chain_type = $FileTransferContext->{"chain_type"};
      @params = $soap_response->paramsout;
      $file_offset = @params[ 0 ];

       Append Data to File
      print LOCAL_FILE $file_data;

      print "Bytes Transferred: $file_offset\n";

      if ( ("FILE_LAST" eq $chain_type) or
           ("FILE_FIRST_AND_LAST" eq $chain_type) )
      {
        $bContinue = 0;
        $success = 1;
      }
    }
  }
  print "\n";

  close(LOCAL_FILE);

  return $success;
}

As it runs it prints out the currently total downloaded bytes which should be "chunk_size" apart until the last entry.

Bytes Transferred: 32768
Bytes Transferred: 65536
Bytes Transferred: 98304
Bytes Transferred: 131072
Bytes Transferred: 163840
Bytes Transferred: 196608
Bytes Transferred: 229376
Bytes Transferred: 262144
Bytes Transferred: 294912
Bytes Transferred: 327680
Bytes Transferred: 360448
Bytes Transferred: 393216
Bytes Transferred: 425984
Bytes Transferred: 458752
Bytes Transferred: 491520
Bytes Transferred: 524288
Bytes Transferred: 557056
Bytes Transferred: 589824
Bytes Transferred: 622592
Bytes Transferred: 655360
Bytes Transferred: 688128
Bytes Transferred: 720896
Bytes Transferred: 753664
Bytes Transferred: 786432
Bytes Transferred: 819200
Bytes Transferred: 851968
Bytes Transferred: 884736
Bytes Transferred: 917504
Bytes Transferred: 950272
Bytes Transferred: 957225

My guess is that something is going wonky with the file_offset value.

-Joe

Thanks Joe. The arguments are handled automatically (at least in most cases) so that's not an issue. But regarding your second piece and the file_offset value, I did some investigation. It looks like the file_offset value in my code is OK - it's incrementing exactly as you describe. But that very last chunk of data seems to be incomplete somehow. I'll need to do a bit more investigation.

 

 

-Matt

Sigh. It's amazing what a little time away can do...I had an 'off-by' bug in my code above, and it's working just fine now. In the event that it's useful to someone, here's what happened. If you look at my while loop, I was terminating the loop (by setting poll=False), but I wasn't *actually flushing the last chunk of data to the file*. This caused the problem, and this is why my md5sum output was different every time I changed the chunk size. Essentially, I was discarding the data so I was always missing that last chunk. Here's an improved version- that is, a working one 🙂

 

!/bin/env python
import base64
import binascii
import io

def pc_downloader(b,remote_file,local_file):

    '''
    Handles downloading files via System.ConfigSync
    Args:
    b -- a pycontrol client object
    remote_file -- the name of the file to fetch from BigIP
    local_file -- the local file to write to.
    '''
    stream_io = io.open(local_file,'wb')
    dl = b.System.ConfigSync
    poll = True
    chunk_size = 64*1024
    foffset = 0 initial file offset to increment.
    lines = []

    while poll:
        print "Foffset is: ", foffset

        res = dl.download_file(remote_file,chunk_size = chunk_size,file_offset = foffset)
        foffset = long(res.file_offset)
        fdata = getattr(res,'return').file_data
        chain_type = getattr(res, 'return').chain_type
        print "Chain type is: ", chain_type

        if (chain_type == 'FILE_LAST') or (chain_type == 'FILE_FIRST_AND_LAST'):
            poll = False
             Now write out the last bit of data.
            lines.append(binascii.a2b_base64(fdata))
        else:
            lines.append(binascii.a2b_base64(fdata))

    stream_io.writelines(lines)

if __name__ == '__main__':
    import sys
    import pycontrol.pycontrol as pc
    host = sys.argv[1]
    f = sys.argv[2]
    b = pc.BIGIP(hostname=host,username='admin',password='admin',fromurl=True,wsdls=['System.ConfigSync'])
    pc_downloader(b,'/var/tmp/'+f,'/var/tmp/'+f)

 

By the way, if I recall the io module requires 2.6. If you're running something lower, just use normal file io...

 

-Matt

Thanks for updating this Matt. I hadn't had a chance to play with the last version you posted but I will test this one out as soon as i get a few minutes.

Updated and added to the codeshare: http://devcentral.f5.com/wiki/default.aspx/iControl/pycontrolGetFileFromBIGIP.html Click Here