BIG-IP Geolocation Updates – Part 5

Introduction

Management of geolocation services within the BIG-IP require updates to the geolocation database so that the inquired IP addresses are correctly characterized for service delivery and security enforcement. Traditionally managed device, where the devices are individually logged into and manually configured can benefit from a bit of automation without having to describe to an entire CI/CD pipeline and change in operational behavior. Additionally, a fully fledged CI/CD pipeline that embraces a full declarative model would also need a strategy around managing and performing the updates. This could be done via BIG-IQ; however, many organizations prefer BIG-IQ to monitor rather than manage their devices and so a different strategy is required.

This article series hopes to demonstrate some techniques and code that can work in either a classically managed fleet of devices or fully automated environment. If you have embraced BIG-IQ fully, this might not be relevant but is hopefully worth a cursory review depending on how you leverage BIG-IQ.

Assumptions and prerequisites

There are a few technology assumptions that will be imposed onto the reader that should be mentioned:

The solution will be presented in Python, specifically 3.10.2 although some lower versions could be supported. The use of the ‘walrus operator” ( := ) was made in a few places which requires version 3.8 or greater. Support for earlier versions would require some porting.
Visual Studio Code was used to create and test all the code. A modest level of expertise would be valuable, but likely not required by the reader.
An understanding of BIG-IP is necessary and assumed.
A cursory knowledge of the F5 Automation Toolchain is necessary as some of the API calls to the BIG-IP will leverage their use, however this is NOT a declarative operation.
Github is used to store the source for this article and a basic understanding of retrieving code from a github repository would be valuable.

References to the above technologies are provided here:

Lastly, an effort was made to make this code high-quality and resilient. I ran the code base through pylint until it was clean and handle most if not all exceptional cases. However, no formal QA function or load testing was performed other than my own. The code is presented as-is with no guarantees expressed or implied. That being said, it is hoped that this is a robust and usable example either as a script or slightly modified into a library and imported into the reader’s project.

Credits and Acknowledgements

Mark_Menger , for his continued review and support in all things automation based.

Mark Hermsdorfer, who reviewed some of my initial revisions and showed me the proper way to get http chunking to work. He also has an implementation on github that is referenced in the code base that you should look at.

Article Series

DevCentral places a limit on the size of an article and having learned from my previous submission I will try to organize this series a bit more cleanly. This is an overview of the items covered in each section

Part 1 - Design and dependencies

Basic flow of a geolocation update
The imports list
The API library dictionary
The status_code_to_msg dictionary
Custom Exceptions
Method enumeration

Part 2 – Send_Request()

Function - send_request

Part 3 - Functions and Implementation

Function – get_auth_token
Function – backup_geo_db
Function – get_geoip_version

Part 4 - Functions and Implementation Continued

Function – fix_md5_file

Part 5 (This article) - Functions and Implementation Continued

Function – upload_geolocation_update

Part 6 - Functions and Implementation Conclusion

Function – install_geolocation_update

Part 7 - Pulling it together

Function – compare_versions
Function – validate_file
Function – print_usage
Command Line script

Functions and Implementation Continued

This part of the series will go through upload_geolocation_update which is a lengthy but important routine that will upload the updates as well as verify that they made it there in tact and without corruption.

upload_geolocation_update()

The next function is upload_geolocation_update() and there is a lot going on within this routine. The broad steps are as follows:

Upload the md5 file
Upload the zip file using chunking
Run the md5sum utility to verify that the upload is not corrupt

def upload_geolocation_update(uri, token, zip_file, md5_file):
    """
    Uploads an md5 and zip file for geolocation db update of a BIG-IP

    Parameters
    ----------
    uri : str       Base URL to call api
    token : str     Valid access token for this API endpoint
    zip_file : str  full path to a geolocation zip file
    md5_file : str  full path to a respective md5 file for the zip_file

    Returns
    -------
    True on success
    False on failure
    """

The function takes a uri and authorization token, consistent with the other functions to this point. It also takes the zip and md5 files, which should include the full path, which will be uploaded. The function returns True on success and False on failure.

    assert uri is not None
    assert token is not None
    assert zip_file is not None
    assert os.path.splitext(zip_file)[-1] == '.zip'
    assert md5_file is not None
    assert os.path.splitext(md5_file)[-1] == '.md5'

    with requests.Session() as session:
        session.headers.update({'Content-Type':'application/octet-stream'})
        session.headers.update({'X-F5-Auth-Token' : token})
        session.verify = False

        # Upload md5 file, its small so a simple upload is all thats necessary
        fix_md5_file(md5_file, '/var/config/rest/downloads', savebu=True)
        url = f"{uri}{library['file-xfr']}{md5_file}"
        size = os.stat(md5_file).st_size
        content_range = f"0-{size-1}/{size}"
        session.headers.update({'Content-Range':content_range})

        with open(md5_file, 'rb') as fileobj:
            if(response:=send_request(url, Method.POST, session, data=fileobj)) is None:
                # Fail hard on md5 upload failure?
                return False

The function begins with an extensive number of asserts to ensure there were passed variables and in the case of the zip and md5 file, if the sanity check for the file extensions pass. We then start off with the same ‘with’ clause and creating a Session object that we have seen. Next, we call fix_md5_file(), which was discussed in the previous section, to correct the md5 file as discussed with the path that our target upload will have, /var/config/rest/downloads. The url requires a little more effort this time around in that we need to state in the API call what the destination file name is. This is analogous to a copy command where there is a source and destination. We need to compute the content_range header and we do that by first finding out the size of the file on the local system. This upload is small so we can do it in one bite, so this is just a formality.

Next, we open the md5 file for read/binary and if successful put it into fileobj and continue with the ‘with’ clause. We send everything to send_request() to upload the file and if we get None in return, we consider the operation a failure and the function returns False. You could debate that failure of the md5 upload is not critical and the routine should just continue, albeit you can verify the hash in the way we are doing it here. A fallback option would be to set a flag and then attempt to upload the zip file. If that fails, then you are out as there is nothing left you can do. If the zip upload succeeds, then you check that flag and either run the md5sum as is show here (the md5 upload succeeded) or run md5sum remotely, capture the hash value, and then compare it to the local copy. However, it seemed this level of complexity for the failure scenarios that could cause this was a bit of overkill which is why I did not author it that way.

        # upload zip file, this time it must be chunked
        url = f"{uri}{library['file-xfr']}{zip_file}"
        size = os.stat(zip_file).st_size
        chunk_size = 512 * 1024
        start = 0

        with open(zip_file, 'rb') as fileobj:
            # Read a 'slice' of the file, chunk_size in bytes
            while( fslice := fileobj.read(chunk_size) ):

                # Compute the size, start, end and modify the content-range header
                slice_size = len(fslice)
                if slice_size < chunk_size:
                    end = size
                else:
                    end = start+slice_size

                session.headers.update({'Content-Range':f"{start}-{end-1}/{size}"} )

                # Send the slice, if we get a failure, dump out and return false
                if(response:=send_request(url, Method.POST, session, data=fslice)) is None:
                    return False

                start += slice_size

The next part of the function is to upload the zip file, and this is large enough to where it needs to be chunked. Thanks to Mark Hermsdorfer for sharing how he did this. The trick to this is you need to read the file in chunk sizes, compute a new content-range and then call send_request() with the slice or chunk you are uploading. The target system will stitch or concatenate the file together for you.

We start off by setting the url and obtaining the size of the file as we did before for the md5 file. We setup a chunk size of 512K and initialize start to zero, which will be our index as we upload. We open the zip file in read/binary mode and then enter a while loop. The while loop continues as we have file data to read, and the read() function will keep track of where we are in the file for us. We then need to compute the slice size, as we expect at some point, we are going to have a slice that is likely smaller than the chunk_size, the ‘remainder’ as it is. We account for this, and ‘end’ is either computed as start + slice_size or just size. These calculations are then used to format our Content-Range header and we write that into the session header.

Next, we sent the request and if we get a response of None, we return False. There are other ways this could be managed, accounting for timeouts or connection issues and doing retires which would be valid. The send_request() exception handlers would need to be modified to allow for this, where 4xx and 5xx errors can be processed the manner they are currently. Then connection related errors are either re-raised or passed to the caller so that a routine such as this could try and remediate or retry. For the purposes of this article, we will assume our IT connectivity is dependable and responsive, but there is no debate this is not always the case.

Lastly, we move our index, start, by slice_size and then continue with the while loop until the entire file has been uploaded.

    # Verify the upload was successful by checking the md5sum.
    with requests.Session() as session:
        url = f"{uri}{library['bash']}"
        session.headers.update({'Content-Type': 'application/json'})
        session.headers.update({'X-F5-Auth-Token' : token})
        session.verify = False

        # Verify that the file upload is sane and passes an md5 check
        data = {'command':'run'}
        data['utilCmdArgs'] = f"-c 'md5sum -c {'/var/config/rest/downloads'}/{md5_file}'"

        if (response:=send_request(url, Method.POST, session, json.dumps(data))) is not None:
            retval = response.json()['commandResult'].split()
            if retval[1] != 'OK':
                print("MD5 Failed check.  Uploaded zip integrity is questionable")
                return False

        return True

The final stanza of this code base is to verify that the upload succeeded, and the zip file is not corrupted. F5 performs md5 hashes of its files and provides them on the download site, which we have manipulated and uploaded to the BIG-IP device along with the zip file. The last step is to run a bash command on the BIG-IP that performs an md5 hash of the uploaded file and compares that to the mf5 file. If they are the same, the file is good. If not, then something happened, and the archive is questionable.

We take a slightly different approach this time to building the body of the request just to illustrate a different technique by creating a dictionary called data and initializing it would he “command : run” key/value pair. Then we add an additional key value pair for the utilCmdArgs. We then send the request to send_request(), notice that we use json.dumps to translate the dictionary into json as we pass the argument. The function send_request() places its return value into response and if its not None we extract the value out of response. We then check to see if the bash command’s response was OK and if not print the failure and the routine returns False. Otherwise, this drops out of the if statement and the routine returns True.

Wrap up

This concludes part 5 of the series. Part 6 of the series will get into install_geolocation_update which installs and verifies the installation of the update. You can access the entire series here: