How I did it - "Visualizing Data with F5 TS and the Elastic ELK Stack"
With the F5 BIG-IP and Telemetry Streaming I have the ability to send BIG-IP metrics to a variety of third-party analytics vendors. One of the more popular of these is Elastic. Elastic's ELK Stack, (acronym for Elasticsearch, Logstash, Kibana) provides a platform where I can store, search, analyze and visualize my BIG-IP telemetry data. With said, here's an overview of "How I did it"; integrating and visualizing data with the ELK Stack. P.S. Make sure to stay for the movie. Application Services 3 Extension (AS3) There are several resources, (logging profiles, log publishers, iRules, etc.) that must be configured on the BIG-IP to enable remote logging. I utilized AS3 to deploy and manage these resources. I used Postman to apply a single REST API declaration to the AS3 endpoint. Telemetry Streaming (TS) F5's Telemetry Streaming, (TS) service enables the BIG-IP to stream telemetry data to a variety of third-party analytics providers. Aside from the aforementioned resources, configuring TS to stream to a consumer, (Logstash in this instance), is simply a REST call away. Just as I did for AS3, I utilized Postman to post a single declaration to the BIG-IP. Elastic (ELK) Stack Configuring the ELK stack to receive and ingest BIG-IP telemetry is a fairly simple process. Logstash, (the "L" in ELK) is the data processor I used to ingest data into the stack. To accomplish this, I applied the sample Logstash configuration file. The configuration file specifies, (among other items) the listener port, message format, and the Elasticsearch index naming format. Dashboards Getting telemetry data into Elasticsearch is great but only if you can make use of it. If I'm going to utilize the data, I need to visualize the data; (should probably trademark that). For visualization, i created a couple sample dashboards. The dashboards, (community-supported and perhaps not suitable for framing) report various relevant BIG-IP performance metrics and WAF incident information. F5 BIG-IP Advanced WAF Insights F5 BIG-IP Performance Metrics Check it Out Rather than walk you through the entire configuration, how about a movie? Click on the link (image) below for a brief walkthrough demo integrating F5's BIG-IP with Elastic's ELK stack using F5 Telemetry Streaming. Try it Out Liked what you saw? If that's the case, (as I hope it was) try it out for yourself. Checkout F5's CloudDocs for guidance on configuring your BIG-IP(s) with the F5 Automation Toolchain.The various configuration files, (including the above sample dashboards) used in the demo are available on the GitHub solution repository Enjoy!F5 Automation Toolchain: Upload the Components with BIGREST!
If you've ever flown in earshot of a pilot or been handed a to-do list from your mommy or significant other, then you are quite familiar with eleventy billion step processes. I may be rounding up there a little. But that is similar to using an imperative approach to automation. The F5 Automation Toolchain, however, is a declarative approach to automation. You throw a big blob (technical term) of JSON data at the BIG-IP in a single declaration and BIG-IP does all the work for you. We will not use the components of the toolchain in this article, but we will get them all installed for use. Prerequisites In order to install the components of the toolchain, you need to download them first. You can find them in the releases of each respective repository on GitHub. I chose the latest release, but there are LTS versions as well if that’s important to your organization. f5-declarative-onboarding f5-appsvcs-extension f5-telemetry-streaming Process The upload and installation procedures are defined in each respective components clouddocs documentation. f5-declarative-onboarding f5-appsvcs-extension f5-telemetry-streaming The process is made super simple for you though: upload, install, verify. Upload To upload each component, you need to send the data to the /mgmt/shared/file-transfer/uploads endpoint in iControl REST. This puts the files in /var/config/rest/downloads on the BIG-IP host system. With the python BIGREST library, I’m using the following function to achieve this. def upload_file(obj, file_name): try: obj.upload(f"/mgmt/shared/file-transfer/uploads/", file_name) except Exception as e: print(f"Failed to upload the component due to {type(e).__name__}:\n") print(f"{e}") sys.exit() Install Installing each component requires a call to the /mgmt/shared/iapp/package-management-tasks endpoint with a JSON blob stating the INSTALL operation and the path to the component we uploaded. This is the function for this: def install_package(obj, file_name): try: data = {"operation": "INSTALL", "packageFilePath": f"/var/config/rest/downloads/{Path(file_name).name}"} obj.command("/mgmt/shared/iapp/package-management-tasks", data) except Exception as e: print(f"Failed to install the component due to {type(e).__name__}:\n") print(f"{e}") sys.exit() Verify Verification takes a little more customization as each is different and one of the component’s data format for the version is different as well. But the basic endpoint that needs to be queried to validate that the package is installed and available for service is /mgmt/shared/$component/info, where $component is updated accordingly like so: /mgmt/shared/declarative-onboarding/info /mgmt/shared/appsvcs/info /mgmt/shared/telemetry/info The function for verification: def verify_package(obj, file_name): component = Path(file_name).name if "f5-declarative-onboarding" in component: try: result = obj.load("/mgmt/shared/declarative-onboarding/info") if result.properties[0].get('version') in component: return True else: return False except RESTAPIError: return False elif "f5-appsvcs" in component: try: result = obj.load("/mgmt/shared/appsvcs/info") if result.properties.get('version') in component: return True else: return False except RESTAPIError: return False elif "f5-telemetry" in component: try: result = obj.load("/mgmt/shared/telemetry/info") if result.properties.get('version') in component: return True else: return False except RESTAPIError: return False else: return False Success! Putting it all together, when I run the script, I get the following feedback (venv) me@mine scripts % python toolchain_prep.py 10.0.2.26 admin admin Instantiating BIG-IP (host 10.0.2.26) Uploading packages Uploading f5-declarative-onboarding-1.20.0-2.noarch.rpm Uploading f5-appsvcs-3.27.0-3.noarch.rpm Uploading f5-telemetry-1.19.0-3.noarch.rpm Installing packages Installing f5-declarative-onboarding-1.20.0-2.noarch.rpm Installing f5-appsvcs-3.27.0-3.noarch.rpm Installing f5-telemetry-1.19.0-3.noarch.rpm Quick break to register packages... Verifying packages f5-declarative-onboarding-1.20.0-2.noarch.rpm installed and verified f5-appsvcs-3.27.0-3.noarch.rpm installed and verified f5-telemetry-1.19.0-3.noarch.rpm installed and verified ---complete--- The full source for this script is in the codeshare. Note that you'll need to be at least at version 1.3.3 of BIGREST for this to work. Conclusion You are now free to move about the cabin. And start declaring your BIG-IP configurations like a boss. And bonus news: the AS3 Configuration Converter, or ACC, was recently released and is available on GitHub. This takes the guess work out of converting your configuration to the proper AS3 declaration schema. Join us on DevCentral Connects this Thursday for a conversation with Ben Gordon on how to accomplish all this in VS Code.F5 Automation Toolchain Upload
Problem this snippet solves: This script uploads, installs, and verifies the three toolchain components: declarative onboarding, as3, and telemetry streaming. Note: bip.py in BIGREST must be modified to allow 202 response codes. The author will have this fixed in the next release, but you will need to modify line 183 in big.py: if response.status_code != 200 and response.status_code != 201 and response.status_code != 202: How to use this snippet: python toolchain_prep.py 10.0.2.26 admin admin Instantiating BIG-IP (host 10.0.2.26) Uploading packages Uploading f5-declarative-onboarding-1.20.0-2.noarch.rpm Uploading f5-appsvcs-3.27.0-3.noarch.rpm Uploading f5-telemetry-1.19.0-3.noarch.rpm Installing packages Installing f5-declarative-onboarding-1.20.0-2.noarch.rpm Installing f5-appsvcs-3.27.0-3.noarch.rpm Installing f5-telemetry-1.19.0-3.noarch.rpm Quick break to register packages... Verifying packages f5-declarative-onboarding-1.20.0-2.noarch.rpm installed and verified f5-appsvcs-3.27.0-3.noarch.rpm installed and verified f5-telemetry-1.19.0-3.noarch.rpm installed and verified ---complete--- Tested this on version: No Version FoundLightboard Lessons: F5 Automation Toolchain
In this episode of Lightboard Lessons, Jason introduces the F5 Automation Toolchain, a suite of tools that enable programmatic declarative interfaces for BIG-IP device onboarding, application services, and telemetry for integration into your CI/CD flows. Resources F5 Automation Toolchain (F5.com introduction) F5 Declarative Onboarding (Clouddocs) F5 Application Services (Clouddocs) F5 Telemetry Streaming (Clouddocs) How Does F5 AS3 Really Work Under the Hood (article) F5 Services in a CI/CD Pipeline (demo) Automate Application Delivery with F5 Hashicorp Terraform and Consul (article)
