BIG-IP Geolocation Updates – Part 3

Introduction

Management of geolocation services within the BIG-IP require updates to the geolocation database so that the inquired IP addresses are correctly characterized for service delivery and security enforcement. Traditionally managed device, where the devices are individually logged into and manually configured can benefit from a bit of automation without having to describe to an entire CI/CD pipeline and change in operational behavior. Additionally, a fully fledged CI/CD pipeline that embraces a full declarative model would also need a strategy around managing and performing the updates. This could be done via BIG-IQ; however, many organizations prefer BIG-IQ to monitor rather than manage their devices and so a different strategy is required.

This article series hopes to demonstrate some techniques and code that can work in either a classically managed fleet of devices or fully automated environment. If you have embraced BIG-IQ fully, this might not be relevant but is hopefully worth a cursory review depending on how you leverage BIG-IQ.

Assumptions and prerequisites

There are a few technology assumptions that will be imposed onto the reader that should be mentioned:

The solution will be presented in Python, specifically 3.10.2 although some lower versions could be supported. The use of the ‘walrus operator” ( := ) was made in a few places which requires version 3.8 or greater. Support for earlier versions would require some porting.
Visual Studio Code was used to create and test all the code. A modest level of expertise would be valuable, but likely not required by the reader.
An understanding of BIG-IP is necessary and assumed.
A cursory knowledge of the F5 Automation Toolchain is necessary as some of the API calls to the BIG-IP will leverage their use, however this is NOT a declarative operation.
Github is used to store the source for this article and a basic understanding of retrieving code from a github repository would be valuable.

References to the above technologies are provided here:

Lastly, an effort was made to make this code high-quality and resilient. I ran the code base through pylint until it was clean and handle most if not all exceptional cases. However, no formal QA function or load testing was performed other than my own. The code is presented as-is with no guarantees expressed or implied. That being said, it is hoped that this is a robust and usable example either as a script or slightly modified into a library and imported into the reader’s project.

Credits and Acknowledgements

Mark_Menger , for his continued review and support in all things automation based.

Mark Hermsdorfer, who reviewed some of my initial revisions and showed me the proper way to get http chunking to work. He also has an implementation on github that is referenced in the code base that you should look at.

Article Series

DevCentral places a limit on the size of an article and having learned from my previous submission I will try to organize this series a bit more cleanly. This is an overview of the items covered in each section:

Part 1 - Design and dependencies

Basic flow of a geolocation update
The imports list
The API library dictionary
The status_code_to_msg dictionary
Custom Exceptions
Method enumeration

Part 2 – Send_Request()

Function - send_request

Part 3 (This article) - Functions and Implementation

Function – get_auth_token
Function – backup_geo_db
Function – get_geoip_version

Part 4 - Functions and Implementation Continued

Function – fix_md5_file

Part 5 - Functions and implementation Continued

Function – upload_geolocation_update

Part 6 - Functions and implementation Conclusion

Function – install_geolocation_update

Part 7 - Pulling it together

Function – compare_versions
Function – validate_file
Function – print_usage
Command Line script

Functions and Implementation

With send_request out of the way, we can now focus on a few ancillary routines that we will need while we perform the main functions of the library/script.

get_auth_token()

With send_request() completed we can now move on to one of the first steps in our design for this operation. Its not required to uses authorization tokens, as you could send authentication every time you make a REST call, but that wouldn’t be as clean a way to perform these operations and it would tax your target system unnecessarily in the process.

def get_auth_token(uri=None, username='admin', password='admin'):
    """
    takes credentials and attempts to obtain an access token from the target
    system.

    Parameters
    ----------
    uri : str       Base URL to call api
    username : str, defaults to 'admin', username for account on target system
    password : str, defaults to 'admin', password for account on target system

    Returns
    -------
    token : str     on success
    None            on failure
    """

The function get_auth_token() defaults the username and password to ‘admin’ which is default for a BIG-IP prior to 14.x, and an alarming good guess otherwise and the uri to none. If the function succeeds it will extract and return an authorization token, otherwise it will return None.

  assert uri is not None

    url = f"{uri}{library['auth-token']}"
    data = {'username':username, 'password':password, 'loginProviderName':'tmos'}

    with requests.Session() as session:
        session.headers.update({'Content-Type': 'application/json'})
        session.verify = False

        # Get authentication token
        if (response:= send_request(url, method=Method.POST, session=session, data=json.dumps(data)) ) is None:
            print("Error attempting to get access token.")
            return None

        # Save token and double check its good
        token = response.json()['token']['token']
        url = f"{uri}{library['mng-tokens']}/{token}"
        session.headers.update({'X-F5-Auth-Token' : token})

        if (response := send_request(url, Method.GET, session) ) is None:
            print(f"Error attempting to validate access token {token}.")
            return None

        return token

First, we assert that uri is not None. The asserts were more of a developmental tool I used as I was putting this code together, as I wanted to be sure that assumptions about passed parameters were not challenged and create more debugging for myself as I got things working. Arguments could be made to manage this differently in finalized code, and they would be valid as asserts are really should be used for debugging. This drifts into the ‘code religion’ area a bit, of which I will tiptoe carefully but, in my view, asserts are developmental tools as production code (optimized) will remove asserts and this can create ‘but it works in dev’ type defects which are needlessly problematic. I left these in here because they could be optimized away, and I felt that this code was going to evolve and be used for other projects.

Next, we build the target url for this API call from our library. This request requires a POST and some additional data passed in the body of the HTTP request which we put into the variable data as a dictionary.

Next, we build a Session object and if successful the ‘with’ block executes with that Session object in the session variable. This request requires that the data passed be of type json, so we update the headers to reflect the data being passed and then disable verify so a system without valid (self-signed) certificates will not cause a problem. Finally, we pass this data on to send_request(). Notice that data is converted from a dictionary to json using the json.dumps() call. We also use the ‘walrus operator’ here to assign the return value into response and check it against None. If send_request() had a problem, it's return value is None, we print an error and then return None to our caller. If it succeeded, then we convert the response to json and extract the token into the variable token.

Just to be thorough, we change the url to a call that allows us to manage, or check, tokens and update the headers to include our newly acquired token. The same arrangement is made to send the request, assign to response, and check it against None where if this fails, we will assume something is wrong and return None. Otherwise, the function returns the token.

backup_geo_db()

The next function to discuss is backup_geo_db(). This call uses a chain of bash shell calls that allow us to create a backup database and then copy the existing db, if it exists, into that directory. Currently that backup location is in /shared/GeoIP_backup. I picked that location simply to ensure adequate space (/var seems to get crowded). The /tmp directory is another viable target although I figured there would be a significant percentage that would prefer not to delete the backup and the end of this operation and /tmp is not a suitable location unless you feel its ephemeral.

def backup_geo_db(uri, token=None):
    """
    Creates a backup directory on target device and then backs the existing
    geolocation db up to that location.

    Parameters
    ----------
    uri : str       Base URL to call api
    token : str     Valid access token for this API endpoint

    Returns
    -------
    True    on success
    False   on failure
    """

The backup_geo_db() function takes a uri and a token, which is defaulted to None, as its arguments. This function simply returns True on success and False on failure.

    assert uri is not None
    assert token is not None

    with requests.Session() as session:
        session.headers.update({'Content-Type': 'application/json'})
        session.headers.update({'X-F5-Auth-Token' : token})
        session.verify = False

        # Create the backup directory
        url = f"{uri}{library['bash']}"
        data = b'{"command": "run", "utilCmdArgs": "-c \'mkdir /shared/GeoIP_backup\'"}'

        # If the backup directory was created, copy the existing db into the backup directory
        if (send_request(url, Method.POST, session, data)) is not None:
            data = b'{"command": "run", "utilCmdArgs": "-c \'cp -R /shared/GeoIP/* /shared/GeoIP_backup/\'"}'

            if (send_request(url, Method.POST, session, data) ) is None:
                print("Unable to backup existing geolocation database")
                return False

        else:
            print("Unable to create backup directory, geolocation db will not be backed up")
            return False

    return True

The function starts with a few asserts to ensure that uri and token are not None. We then create a Session object, place it into session and if this is valid continue with the ‘with’ block. For thus API call, we need the content type to be json, we need the authorization token, and to set verify to False so that self-signed certificates are ignored. This information is added into the session headers. We then create the url which will call a bash function on the BIG-IP. This call uses a POST method, and the body of the request contains information on the specifics for the command that gets passed to bash. We load the data variable with a binary string that has the command, run, and then the arguments to make the backup directory. This is passed to send_request() and if send_request() returns None, it drops to the else below, prints a message on the backup failure and the function returns False. Otherwise, we continue to change the data variable, which is the request body, to copy the /shared/GeoIP/ directory to /shared/GeoIP_backup/. If you have not referred to the K article, it might be worth reviewing as there were directory changes made that might be relevant to you and to adjust the copy source directory accordingly. This is sent to send_request() and if it returns None, the function returns False. Otherwise, execution drops passed all this and to the final return statement which returns True.

get_geoip_version()

Next, we look at get_geoip_version() which calls a utility on the BIG-IP to look up an IP address and we can extract the version of the database from this information. The function takes a uri and an authorization token that is defaulted to None.

def get_geoip_version(uri, token=None):
    """
    Makes a call to run 'geoip_lookup 104.219.101.154' on the F5 to extract
    the db date/version

    Parameters
    ----------
    uri : str       Base URL to call api
    token : str     Valid access token for this API endpoint

    Returns
    -------
    str     date/version string on success
    None    on failure
    """

The function returns None on failure, and a string containing the date/version of the geolocation database on success. The IP address is hardcoded and was extracted from the K article K11176, step 7. If you prefer, you can change this to something like 8.8.8.8.

    assert uri is not None
    assert token is not None

    with requests.Session() as session:
        session.headers.update({'Content-Type': 'application/json'})
        session.headers.update({'X-F5-Auth-Token' : token})
        session.verify = False
        retval = None

        url = f"{uri}{library['bash']}"
        data = b'{"command": "run", "utilCmdArgs": "-c \'geoip_lookup 104.219.101.154\'"}'

        if( response:=send_request(url, Method.POST, session, data)) is not None:

            # Convert the response to json, find the commandResult string and splitlines it into a list
            for line in response.json()['commandResult'].splitlines():
                # Walk the list until we find the Copyright and then return the last 8 characters
                if "Copyright" in line:
                    retval = line[-8:]

        return retval

The function starts off asserting that the uri and token are not None. We then create a Session object and put it into session. If this succeeds, then the ‘with’ block proceeds where we set the json and authorization headers and setting the session to ignore self-signed certificates. We also set retval to None as they default return value unless the function can set it to something else.

Then, we set the url and data variables to tell the BIG-IP to run a bash command, ‘geoip_lookup 104.219.101.154’. We send this to send_request() and assign the return value into response. If the response is not None, we then go on to process the response.

To process the response, we first convert it to json and then extract out the value of ‘commandResult’ and split this into lines. We then enter a for look for each line setting line to the current line to process, this is all accomplished on one line. We then look to see if ‘Copyright’ is in the current line and if it is we set retval to that last eight characters which should be the version date of the geolocation database.

Finally, retval is returned to the caller.