BIG-IP Geolocation Updates – Part 7

Introduction

Management of geolocation services within the BIG-IP require updates to the geolocation database so that the inquired IP addresses are correctly characterized for service delivery and security enforcement. Traditionally managed device, where the devices are individually logged into and manually configured can benefit from a bit of automation without having to describe to an entire CI/CD pipeline and change in operational behavior. Additionally, a fully fledged CI/CD pipeline that embraces a full declarative model would also need a strategy around managing and performing the updates. This could be done via BIG-IQ; however, many organizations prefer BIG-IQ to monitor rather than manage their devices and so a different strategy is required.

This article series hopes to demonstrate some techniques and code that can work in either a classically managed fleet of devices or fully automated environment. If you have embraced BIG-IQ fully, this might not be relevant but is hopefully worth a cursory review depending on how you leverage BIG-IQ.

Assumptions and prerequisites

There are a few technology assumptions that will be imposed onto the reader that should be mentioned:

The solution will be presented in Python, specifically 3.10.2 although some lower versions could be supported. The use of the ‘walrus operator” ( := ) was made in a few places which requires version 3.8 or greater. Support for earlier versions would require some porting.
Visual Studio Code was used to create and test all the code. A modest level of expertise would be valuable, but likely not required by the reader.
An understanding of BIG-IP is necessary and assumed.
A cursory knowledge of the F5 Automation Toolchain is necessary as some of the API calls to the BIG-IP will leverage their use, however this is NOT a declarative operation.
Github is used to store the source for this article and a basic understanding of retrieving code from a github repository would be valuable.

References to the above technologies are provided here:

Lastly, an effort was made to make this code high-quality and resilient. I ran the code base through pylint until it was clean and handle most if not all exceptional cases. However, no formal QA function or load testing was performed other than my own. The code is presented as-is with no guarantees expressed or implied. That being said, it is hoped that this is a robust and usable example either as a script or slightly modified into a library and imported into the reader’s project.

Credits and Acknowledgements

Mark_Menger , for his continued review and support in all things automation based.

Mark Hermsdorfer, who reviewed some of my initial revisions and showed me the proper way to get http chunking to work. He also has an implementation on github that is referenced in the code base that you should look at.

Article Series

DevCentral places a limit on the size of an article and having learned from my previous submission I will try to organize this series a bit more cleanly. This is an overview of the items covered in each section

Part 1 - Design and dependencies

Basic flow of a geolocation update
The imports list
The API library dictionary
The status_code_to_msg dictionary
Custom Exceptions
Method enumeration

Part 2 – Send_Request()

Function - send_request

Part 3 - Functions and Implementation

Function – get_auth_token
Function – backup_geo_db
Function – get_geoip_version

Part 4 - Functions and Implementation Continued

Function – fix_md5_file

Part 5 - Functions and Implementation Continued

Function – upload_geolocation_update

Part 6 - Functions and Implementation Conclusion

Function – install_geolocation_update

Part 7 (This article) - Pulling it together

Function – compare_versions
Function – validate_file
Function – print_usage
Command Line script

Pulling it together

With the completion of the main functional routines, we are now ready to pull everything together. There are a few additional routines that we will add that will simplify the use of the library/script so we will start there first.

compare_versions()

First up is a simple function that allows us to compare two version strings from our geolocation lookup tool to determine if indeed an update was successful, and the database is in use.

def compare_versions(start, end):
    """
    Helper function to compare two geolocation db version and output message

    Parameters
    ----------
    start : str    Beginning version string of geolocation db
    end : str      Ending version string of geolocation db

    Returns
    -------
    0 on success
    1 on failure
    """

The routine takes a start and end, both strings, that represent the two strings to compare. The function returns a simple 0 on success, meaning that the end string represents a later date than the start string. Otherwise, a 1 is returned for the other cases.

    print(f"Starting GeoIP Version: {start}\nEnding GeoIP Version: {end}")

    if int(start) < int(end):
        print("GeoIP DB updated!")
        return 0

    print("ERROR GeoIP DB NOT updated!")
    return 1

Looking at the body of the function, it will first print out what the starting and ending versions are to the console and then check to see if start is less than end. Notice that the strings are casted to an int in both cases. If this statement is true, it prints out that the DB was updated and returns 0. Otherwise, it prints out that the DB was not updated and returns 1.

validate_file()

stuff

def validate_file(path, file):
    """
    Verifies that the file exists and if in the same directory, keeps the basename.
    If its in a relative or different directory, returns the full path resolving
    links and so on.

    Parameters
    ----------
    path : str
        Argument 0 from sys.argv.. the passed current working directory and exe name
    file : str
        Name of the file to check

    Returns
    -------
    Corrected file with full path

    Raises
    ------
    FileNotFoundError if file doesn't exist
    """

The routine accepts a path and a file name as arguments. The path should be the path passed from sys.argv in most cases although depending on how this is being integrated it may be from a working directory. The file name is the name of the file to check. The routine will return a corrected file with the full path. If the file doesn’t exist, it will raise a FileNotFoundError.

    assert path is not None
    assert file is not None

    # unlikely to raise, but there could be an errno.xx for oddly linked CWDs
    cwd = os.path.dirname(os.path.realpath(path))

    # Verify the zip exists, if its in the same directory, clean up the path
    if not os.path.exists(file):
        raise FileNotFoundError(f"Unable to find file {file}")

    # If cwd and file is in same location, just use the basename for the file
    if cwd == os.path.dirname(os.path.realpath(file)):
        retval = os.path.basename(file)
    # otherwise use the full path (and resolve links) to the file
    else:
        retval = os.path.realpath(file)

    return retval

Moving onto the body of the function, it first asserts the path and file arguments are not None. Next, we get the current working directory by taking the path and running it through realpath() to deal with any oddly linked directories and then return only the directory name.

Next, we check if the file exists, notice it doesn’t matter where it is. If it cannot be found, we raise a FileNotFoundError and the exception leaves the routine. Next, we do the same operation on the file as we did the current working directory, saved in cwd, and compare them. If they are the same, then the file resides in the same current working directory, and we just return the base name of the file (the filename with no path). Otherwise, we return the file ensuring we have the real path which should handle relative paths and ensure we don’t get odd issues when trying to access the file.

We the return the return value.

print_usage()

def print_usage():
    """
    Prints out the correct way to call the script
    """

This routine doesn’t take any arguments and is only meant to simplify returning the usage for the script.

    print("Usage: geolocation-update.py <hostname/ip> <credentials> <zip> <md5>")
    print("\t<hostname/ip> is the resolvable address of the F5 device")
    print("\t<credentials> are the username and password formatted as username:password")
    print("\t<zip> is the name, and path if not in the same directory, to the geolocation zip package")
    print("\t<md5> is the name, and path if not in the same directory, to the geolocation zip md5 file")
    print("\nNOTE:  You can omit the password and instead put it in an env variable named BIGIP_PASS")

There is not much to explain here as it just prints out usage to the console. Obviously, depending on how you integrate these routines, you may need to change this appropriately.

Command Line Script

Now we need a way to wrap all this together. Thus far, this code has been presented in a somewhat library-like fashion, although you would need to do some things to make it a module of course. However, to illustrate how to use it all we can set up this code so it can be executed standalone.

###############################################################################
# main() entry point if run from cmdline as script
###############################################################################
if __name__ == "__main__":
    # Disable/suppress warnings about unverified SSL:
    import urllib3
    requests.packages.urllib3.disable_warnings()
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

For a standalone execution, we check if __name__ is equivalent to the string “__main__”. This is a pythonic “trick” which allows the interpreter to figure out if the module being run is the main program. If it is, its basically being run as a script. Otherwise, __name__ will be set to the modules name and we know its being imported into another piece of code.

We import urllib3 which we need in a moment and then disable some annoying warnings that will tell us we are potentially connecting to unsafe web sources.

    try:
        if len(sys.argv) < 5:
            raise ValueError

        # Extract cmd line arguments and massage them accordingly
        g_path = sys.argv[0]
        g_bigip = f"https://{sys.argv[1]}"
        g_creds = sys.argv[2]
        g_zip_file = validate_file(g_path, sys.argv[3])
        g_md5_file = validate_file(g_path, sys.argv[4])

        # Handle username/password from creds or environment variable
        if ('BIGIP_PASS' in os.environ ) and (os.environ['BIGIP_PASS'] is not None) and (not ":" in g_creds):
            g_username = g_creds
            g_password = os.environ['BIGIP_PASS']
        else:
            creds = g_creds.split(':',1)
            g_username = creds[0]
            g_password = creds[1]

    except ValueError:
        print("Wrong number of arguments.")
        print_usage()
        sys.exit(-1)

    except FileNotFoundError as e:
        print(f"{e}.  Exiting..")
        print_usage()
        sys.exit(-1)

The first part of the script, we wand to verify the command line arguments and massage a few things. We do a quick sanity check on the number of arguments passed and if they are less than 5 we know we are missing some data to run correctly. Next, we extract the command line arguments into some global variables. The variable g_bigip is formatted slightly to save us some time putting the protocol later on. Some better checking could be performed to ensure its not already formatted that way. The username and potentially password is put into g_creds, which we will clean up in a moment. The last two, g_zip_file and g_md5_file hold the file names for the respective after being processed by validate_file().

Next, we check and see if the password for the passed username is in an environment variable. If there is no “:” in the string, meaning the caller did not pass <username>:<password> to us and an environment variable is set then we can set g_username and g_password. Otherwise, we extract g_username and g_password from g_creds and move forward. We handle exceptional cases for not enough arguments and if validate_file raises FileNotFoundError and exit in both cases.

    # Get the access token
    print("Getting access token")
    if( g_token := get_auth_token(g_bigip, g_username, g_password) ) is None:
        print("Problem getting access token, exiting")
        sys.exit(-1)

    # Attempt to backup existing db
    print("Backing up existing db")
    backup_geo_db(g_bigip, g_token)

    # Get starting date/version of geolocation db for comparison
    startVersion = get_geoip_version(g_bigip, g_token)

    # Upload geolocation update zip file
    print("Uploading geolocation updates")
    if False is upload_geolocation_update(g_bigip, g_token, g_zip_file, g_md5_file):
        print("Unable to upload zip and/or md5 file.  Exiting.")
        sys.exit(-1)

    # Install geolocation update
    print("Installing geolocation updates")
    if False is install_geolocation_update(g_bigip, g_token, g_zip_file):
        print("Unable to install the geolocation updates.  Exiting.")
        sys.exit(-1)

    # Get end date/version of geolocation db for comparison
    endVersion = get_geoip_version(g_bigip, g_token)
    sys.exit( compare_versions(startVersion, endVersion) )

Finally, we construct what could be considered the “main loop” which, because of all the code we have written thus far, is quite pithy. First, we attempt to get an access token which we will need for authorization going forward. Next, we attempt to back up the db on the BIG-IP. Notice we don’t verify that and simply go forward if that were to fail. A more conservative approach would want to handle this differently. Next, we get the starting version so we can compare that after we process the update.

Then, we upload the geolocation update files. If this fails, we do catch it and exit with an error. Next, we install the geolocation update and again, if it fails exit the script with an error.

Lastly, we get the version string again and then compare versions as we exit the routine. And that, finally, concludes the project.

Wrap up

This concludes part 7 of the series, and the conclusion of the series overall. Hopefully, this provides a suitable framework for performing geolocation updates that you can either use as is or incorporate into your toolset or CI/CD pipeline. It should pass a lint check and was vigorously tested by myself, but I would encourage a more rigorous and formal review for production purposes. Hopefully this has provided some insight and ideas to solving geolocation database maintenance in your environment. You can access the entire series here:

Updated May 12, 2022

Version 2.0

application delivery

devops