BIG-IP Geolocation Updates – Part 1

Introduction

Management of geolocation services within the BIG-IP require updates to the geolocation database so that the inquired IP addresses are correctly characterized for service delivery and security enforcement. Traditionally managed device, where the devices are individually logged into and manually configured can benefit from a bit of automation without having to describe to an entire CI/CD pipeline and change in operational behavior. Additionally, a fully fledged CI/CD pipeline that embraces a full declarative model would also need a strategy around managing and performing the updates. This could be done via BIG-IQ; however, many organizations prefer BIG-IQ to monitor rather than manage their devices and so a different strategy is required.

This article hopes to demonstrate some techniques and code that can work in either a classically managed fleet of devices or fully automated environment. If you have embraced BIG-IQ fully, this might not be relevant but is hopefully worth a cursory review depending on how you leverage BIG-IQ.

Assumptions and prerequisites

There are a few technology assumptions that will be imposed onto the reader that should be mentioned:

The solution will be presented in Python, specifically 3.10.2 although some lower versions could be supported. The use of the ‘walrus operator” ( := ) was made in a few places which requires version 3.8 or greater. Support for earlier versions would require some porting.
Visual Studio Code was used to create and test all the code. A modest level of expertise would be valuable, but likely not required by the reader.
An understanding of BIG-IP is necessary and assumed.
A cursory knowledge of the F5 Automation Toolchain is necessary as some of the API calls to the BIG-IP will leverage their use, however this is NOT a declarative operation.
Github is used to store the source for this article and a basic understanding of retrieving code from a github repository would be valuable.

References to the above technologies are provided here:

Lastly, an effort was made to make this code high-quality and resilient. I ran the code base through pylint until it was clean and handle most if not all exceptional cases. However, no formal QA function or load testing was performed other than my own. The code is presented as-is with no guarantees expressed or implied. That being said, it is hoped that this is a robust and usable example either as a script or slightly modified into a library and imported into the reader’s project.

Credits and Acknowledgements

Mark_Menger, for his continued review and support in all things automation based.

Mark Hermsdorfer, who reviewed some of my initial revisions and showed me the proper way to get http chunking to work. He also has an implementation on github that is referenced in the code base that you should look at.

Article Series

DevCentral places a limit on the size of an article and having learned from my previous submission I will try to organize this series a bit more cleanly. This is an overview of the items covered in each section:

Part 1 (This article) Design and dependencies

Basic flow of a geolocation update
The imports list
The API library dictionary
The status_code_to_msg dictionary
Custom Exceptions
Method enumeration

Part 2 – Send_Request()

Function - send_request

Part 3 - Functions and Implementation

Function – get_auth_token
Function – backup_geo_db
Function – get_geoip_version

Part 4 - Functions and Implementation Continued

Function – fix_md5_file

Part 5 - Functions and Implementation Continued

Function – upload_geolocation_update

Part 6 - Functions and Implementation Conclusion

Function – install_geolocation_update

Part 7 - Pulling it together

Function – compare_versions
Function – validate_file
Function – print_usage
Command Line script

Design and dependencies

For this article, the design of the code will roughly follow the steps outlined in the F5 article located here F5 - K11176. It is worth looking through this article and its suggestions if your intentions are to use this article and its code as a reference and build your own solution from scratch. The rough design process is as follows:

This flow is extracted from the article reference as a rough outline of the steps we want to take. Since we are using Python and we know that REST calls will be made in order to facilitate interaction with the BIG-IP device, we can also will in some of the dependencies we will need:

from enum import Enum
from datetime import datetime
import os
import sys
import json
import shutil
import requests

Skipping ahead to a bit of implementation details, we will use enums to control the type of API call, we want to make, GET, POST, etc. to a general API calling function. There are other ways to approach this, and I admit this is a C/C++ centric idea, so I’ll accept the charge as being slightly un-pythonic. The datetime library will have a couple of uses, mostly to stamp backup files and, if we should decide, logging (spoiler: the latter was not implemented). The os and sys imports are routine library imports for various utilities and functions. The shutil import is used once for making a backup of a file. The json and requests imports are critical for manipulating json content and for interacting with the BIG-IP via REST calls, vis-à-vis HTTP.

Global Variables

There are a few global variables defined at the top of the source file that are namely used to simplify and shorten the code that follows. The first of these is a dictionary library:

# Library of REST API end points for frequently used automation calls
library = {'auth-token':'/mgmt/shared/authn/login',
           'mng-tokens':'/mgmt/shared/authz/tokens',
           'pass-policy':'/mgmt/tm/auth/password-policy',
           'pass-change':'/mgmt/tm/auth/user/',
           'get-version':'/mgmt/tm/sys/version',
           'file-xfr':'/mgmt/shared/file-transfer/uploads/',    # /var/config/rest/downloads/
           'iso-xfr':'/cm/autodeploy/software-image-uploads/',  # /shared/images/
           'mgmt-tasks':'/mgmt/shared/iapp/package-management-tasks',
           'do-info':'/mgmt/shared/declarative-onboarding/info',
           'as3-info':'/mgmt/shared/appsvcs/info',
           'do-upload':'/mgmt/shared/declarative-onboarding/',
           'do-tasks':'/mgmt/shared/declarative-onboarding/task/',
           'as3-upload':'/mgmt/shared/appsvcs/declare?async=true',
           'as3-tasks':'/mgmt/shared/appsvcs/task/',
           'bash':'/mgmt/tm/util/bash',
}

When building a request call, we will need to define the specific API endpoint which in a REST call is defined by the URL we are calling. To make this a more usable, the domain name will be obtained as the target system and then the path will be one of these dictionary values. The scheme or protocol will always be ‘https’ as is necessary for making the call to a BIG-IP. These will be connected to create the API endpoint for our call. The library dictionary ensures that this is done the same way for every call and if we should want to change, or must change, we only have one place to do it as opposed to numerous places in the file.

The next dictionary, status_code_to_msg, was created to refactor some error handling code:

# Dictionary to translate a status code to an error message
status_code_to_msg = {400:"400 Bad request.  The url is wrong or malformed\n",
                      401:"401 Unauthorized.  The client is not authorized for this action or auth token is expired\n",
                      404:"404 Not Found.  The server was unable to find the requested resource\n",
                      415:"415 Unsupported media type.  The request data format is not supported by the server\n",
                      422:"422 Unprocessable Entity.  The request data was properly formatted but contained invalid or missing data\n",
                      500:"500 Internal Server Error.  The server threw an error while processing the request\n",
}

When we send a request, there are several different possibilities which may result. We can raise an exception which will allow us to handle 4xx and 5xx failures all at once, which are results that need to be addressed differently than various 2xx responses. However, this can then create a large chain of if-else clauses which mostly just capture the error. This dictionary allows us to refactor many lines of code into just 2 or 3 and provides for easy expansion should it be deemed necessary.

Custom Exceptions

We add a couple of custom exceptions that are specific for the code:

# Define some exception classes to handle failure cases and consolidate some of the errors
class ValidationError(Exception):
    """
    ValidationError for some of the necessary items in send_request
    """

class InvalidURL(Exception):
    """
    For raising exception if invalid on null urls are passed to send_request
    """
    def __init__(self, message, errors):
        super().__init__(message)
        self.errors = errors

The ValidationError is arguably not needed as a suitable built-in exception exists, however while refactoring the code there was some cases where more information was being added and reported and thus it was never expunged. The InvalidURL exception does capture a bit of information that we use when reporting an error but still remains a very ‘vanilla’ custom exception.

Method enumeration

Last is the Method enumeration:

class Method(Enum):
    """
    Class Method(Enum) provides simple enumeration for controlling way
    send_request communicates to target
    """
    GET = 1
    POST = 2
    PATCH = 3
    DELETE = 4

This enumeration controls which type of REST call the send_request, which has not been discussed, call will send a request. During construction of the code there were some alternative ideas on how to implement this that might have made more sense for this additional complexity but where then refactored out to be more straightforward and readable. The enumeration still felt elegant from a function calling perspective, so it remains.

Wrap up

This concludes part 1 of the series. In part 2 we will construct a routine to send and receive calls to the big-ip as well as handle the numerous issues that can arise. You can access the entire series here: