BIG-IP Geolocation Updates – Part 2

Introduction

Management of geolocation services within the BIG-IP require updates to the geolocation database so that the inquired IP addresses are correctly characterized for service delivery and security enforcement. Traditionally managed device, where the devices are individually logged into and manually configured can benefit from a bit of automation without having to describe to an entire CI/CD pipeline and change in operational behavior. Additionally, a fully fledged CI/CD pipeline that embraces a full declarative model would also need a strategy around managing and performing the updates. This could be done via BIG-IQ; however, many organizations prefer BIG-IQ to monitor rather than manage their devices and so a different strategy is required.

This article series hopes to demonstrate some techniques and code that can work in either a classically managed fleet of devices or fully automated environment. If you have embraced BIG-IQ fully, this might not be relevant but is hopefully worth a cursory review depending on how you leverage BIG-IQ.

Assumptions and prerequisites

There are a few technology assumptions that will be imposed onto the reader that should be mentioned:

The solution will be presented in Python, specifically 3.10.2 although some lower versions could be supported. The use of the ‘walrus operator” ( := ) was made in a few places which requires version 3.8 or greater. Support for earlier versions would require some porting.
Visual Studio Code was used to create and test all the code. A modest level of expertise would be valuable, but likely not required by the reader.
An understanding of BIG-IP is necessary and assumed.
A cursory knowledge of the F5 Automation Toolchain is necessary as some of the API calls to the BIG-IP will leverage their use, however this is NOT a declarative operation.
Github is used to store the source for this article and a basic understanding of retrieving code from a github repository would be valuable.

References to the above technologies are provided here:

Lastly, an effort was made to make this code high-quality and resilient. I ran the code base through pylint until it was clean and handle most if not all exceptional cases. However, no formal QA function or load testing was performed other than my own. The code is presented as-is with no guarantees expressed or implied. That being said, it is hoped that this is a robust and usable example either as a script or slightly modified into a library and imported into the reader’s project.

Credits and Acknowledgements

Mark_Menger , for his continued review and support in all things automation based.

Mark Hermsdorfer, who reviewed some of my initial revisions and showed me the proper way to get http chunking to work. He also has an implementation on github that is referenced in the code base that you should look at.

Article Series

DevCentral places a limit on the size of an article and having learned from my previous submission I will try to organize this series a bit more cleanly. This is an overview of the items covered in each section:

Part 1 - Design and dependencies

Basic flow of a geolocation update
The imports list
The API library dictionary
The status_code_to_msg dictionary
Custom Exceptions
Method enumeration

Part 2 (This article) – Send_Request()

Function - send_request

Part 3 - Functions and Implementation

Function – get_auth_token
Function – backup_geo_db
Function – get_geoip_version

Part 4 - Functions and Implementation Continued

Function – fix_md5_file

Part 5 - Functions and Implementation Continued

Function – upload_geolocation_update

Part 6 - Functions and implementation Conclusion

Function – install_geolocation_update

Part 7 - Pulling it together

Function – compare_versions
Function – validate_file
Function – print_usage
Command Line script

Functions and Implementation

In this part of the series, we are going to construct a routine that sends and receives information to the BIG-IP. There are lot of things that can go wrong, like incorrect API endpoints, timeouts, server errors and services unavailability. This routine is intended to concentrate managing those details in one place so that every function that needs to make a call will have less complexity in it.

send_request()

This function evolved through many changes as this code was developed, starting with being a very simplistic call to now being almost too complex. However, an attempt was made to make this the single call to send a request to the BIG-IP and handle the numerous ways in which that could fail. To start off with, and this will be continued throughout the code, the function gets a docstring that explains the inputs and outputs:

def send_request(url, method=Method.GET, session=None, data=None):
    """
    send_request is used to send a REST call to the device.  By default it assumes
    that this is a GET request (through the default enumeration).  The passed
    session and data are also by default set to None.  In the case of data, this
    is ignored as its only relevant for a POST or PATCH call.  However the session
    is checked against the default and raises if its None.  PATCH and DELETE are
    also not implemented yet and raise.

    Parameters
    ----------
    url : str                                 The url endpoint to send the request to
    method : Method, defaults to Method.GET   One of the valid Method enumerations
    session : obj, defaults to None           Active / valid session object
    data : str, defaults to None              JSON formatted string passed as body in request

    Returns
    -------
    response str on success
    None on failure

    Raises
    ------
    notImplemented      For improper methods
    ValidationError     If the session object is None or inactive
                        The url parameter is missing
    """

The arguments ‘url’ and ‘session’ are required arguments and the routine will return a response string on success and None on failure. However, it will raise a couple of exceptions in a couple of cases as well.

   if not url:
        raise InvalidURL("The url is invalid", url)

    error_message = None
    response = None

    try:
        if None is session:
            raise ValidationError("Invalid session provided")

        # Send request and then raise exceptions for 4xx and 5xx issues
        if method is Method.GET:
            response = session.get(url)
        elif method is Method.POST:
            response = session.post(url, data)
        elif method is Method.PATCH:
            raise NotImplementedError("The PATCH method is not implemented yet")
        elif method is Method.DELETE:
            raise NotImplementedError("The DELETE method is not implemented yet")
        else:
            raise NotImplementedError(f"The HTTP method {method} is not supported")

        response.raise_for_status()

The first part of the routine checks if there is no url and if so, raises an InvalidURL exception. This is left outside of the try/except clause that follows as the calling code should handle this and the chain of except clauses are more focused on a failure to send the request.

The next set of routines follows a python try/except block but let us break that down a little first:

try:
    # Something risky
except:
    # Optional handling of exception (if required)
    # Can be more than one
else:
    # Run this if there was NO exception
finally:
    # Always run this no matter what

Most people are familiar with try/except however Python implements some additional blocks that are used here. The ‘try’ block is the code that we want to run that we think might have a failure or raise for some sort of condition. In our case, we will raise which will automatically cause an exception if there was a 4xx or 5xx response, but we will get to that in a minute. The except block is what to do it there is an exception. You can ‘catch’ (a c++ piece of terminology) different types of exceptions and handle them differently which we will do but the important thing to understand here is if there is a problem in your ‘try’ block, the system will run through these except blocks looking for something to handle it. It the code in the ‘try’ block runs fine, or rather doesn’t raise, then the else block will be run. Finally, pun intended, the ‘finally’ block is run regardless of an exception or not. While this might seem complicated it does make it easier to handle multiple failures as a group which was why it was implemented here.

The ‘try’ block starts out by defining error_message and response to None, which we check against later. It then verifies that the session is not None, or in this case that the function caller passed something for session. If its still None it raises and the except block processes this. At one point, there was consideration given to creating a session here for very simple requests, but this code was never refactored in that way. However, if this was used in a larger context, I would refactor it in that manner and then the default argument value would make more sense.

Next, we go through the different possibilities for the method variable, of which method.GET and method.POST are the only options that are implemented. Next, the return value is then used to call response.raise_for_status(). The raise_for_status() call is a convenient built-in routine that sorts out 4xx and 5xx responses and then puts them into an HTTPError exception with the respective message.

    except requests.exceptions.HTTPError:
        # Handle 4xx and 5xx errors here.  Common 4xx and 5xx REST errors here
        if not (error_message := status_code_to_msg.get(response.status_code) ):
            error_message = f"{response.status_code}.  Uncommon REST/HTTP error"

    except requests.exceptions.TooManyRedirects as e_redir:
        # Handle excessive 3xx errors here
        error_message = f"{'TooManyRedirects'}:  {e_redir}"

    except requests.exceptions.ConnectionError as e_conn:
        # Handle connection errors here
        error_message = f"{'ConnectionError'}:  {e_conn}"

    except requests.exceptions.Timeout as e_tout:
        # Handle timeout errors here
        error_message = f"{'Timeout'}:  {e_tout}"

    except requests.exceptions.RequestException as e_general:
        # Handle ambiguous exceptions while handling request
        error_message = f"{'RequestException'}:  {e_general}"

Next, we handle the numerous possible exceptions that might take place. The raise_for_status() call simplifies this for us a little and we handle all the 4xx and 5xx in the HTTPError exception block. This is where we take advantage of the status_code_to_msg dictionary and format a specific error message based on the specific code returned. This saves us numerous if-else statements and elegantly handles the case where we don’t have a specific message handler. The next exception, TooManyRedirects, handles an excessive number of 3xx responses. This block could take action to address this situation but for now we just format error_message with a message and move on. The ConnectionError and Timeout exception handlers also just format a message in error_message. Again, more robust handling could be implemented say to verify that the IP address is pingable in the case of a connection error or do some additional checks or reattempt in the case of a Timeout. These would make more sense in a larger project, and the logic is provided here should it make sense to the reader. The last exception handles any other ambiguous exceptions that might have been raised from requests.

    else:
        return response

    finally:
        # if error message isn't None, there is an error to process and we should return None
        if error_message:
            print(f"send_request() Error:\n{error_message}")
            print(f"url:\t{url}\nmethod:\t{method.value}\ndata:\t{data}")

            if response is not None:
                print(f"response: {response.json()}")

    return None

Next, follows the else block which will be run if no exceptions have occurred. In this case, the function just returns the response. Initially, I had designed some ideas to format the response to json if desired or extract content in the response so that calling functions would require less processing of the response. Pylint had already informed me this function was too complex and didn’t think this project required more ‘cleverness’ so I left it simple. Again, this is where modifications could be valuable and make this more usable in a larger context.

Then the ‘finally’ block is processed. The only thing left to do is print out the failure and provide a little more data. If the response was valid, which would happen in the case of 4xx or 5xx responses, then it can be printed as well for additional information.

Lastly, the function returns None which is the appropriate response if the ‘else’ block is not executed.

Wrap up

This concludes part 2 of the series. In part 3, we will start working on the implementation of our routines and explore get_auth_token, backup_geo_db, and get_geoip_version. You can access the entire series here: