F5 Distributed Cloud Telemetry (Metrics) - ELK Stack

Introduction: 

This article is a part of the F5 Distributed Cloud (F5 XC) telemetry series. Here we will explore how to export metrics from the XC console to ELK Stack using XC’s service graph API and visualize them on the ELK Dashboard. 

 

Overview: 

As we are looking into exporting metrics data to the ELK stack using Python script, let's first get a high-level overview of the same. 

Metrics are numerical values that provide actionable insights into the performance, health and behavior of systems or applications over time, allowing teams to monitor and improve the reliability, stability and performance of modern distributed systems. 

ELK Stack (Elasticsearch, Logstash, and Kibana) is a powerful open-source platform. It enables organizations to collect, process, store, and visualize telemetry data such as logs, metrics, and traces from remote systems in real-time. 

In this article, we will demonstrate how to use a custom Python-based exporter script to export metrics from XC console to ELK stack using XC’s Service Graph API. We will then visualize them on the Kibana dashboard. 

 

Prerequisites: 

• Access to F5 Distributed Cloud (XC) SaaS console 

• VM with Python installed to run exporter script 

• VM with ELK stack configured (To bring up the ELK stack using docker, follow the docker-elk GitHub readme) 

 

Architecture diagram: 

 

Demonstration: 

Follow this GitHub Repo Link for more information related to execution steps: 

Step 1: Run the python exporter script `metrics.py`: 

• This Python script has 3 main functions: 

• • Fetch the metrics from XC’s service graph API
  • Transform metrics into Elasticsearch-compatible format
  • Send Metrics to Elasticsearch 

• Update `user_inputs.json` file: 

• • Add XC API Token 

• • Add HTTP LB name 

• • Add XC tenant name 

• • Add XC namespace to deploy the HTTP LB 

• • Add ELK index name for exporting metrics 

• • Add Elastic Search URL 

• Execute exporter script: `python3 exporter.py` 

 

Note: The above step will initiate a Flask web server listening on port 8888, with an endpoint /send-metrics.

 

Step 2: Run the shell script `metrics.sh`. This script will periodically call the /send-metrics endpoint, which will internally call fetch, transform and export/send functions mentioned in Step 1. 

 

Step 3: Open ELK Stack Dashboard. Select the index and use the fetched values to create a visualization graph. 

 

Fig: Below’s image shows a graph plot of collected request throughput values over the last few minutes from the XC console 

 

Fig: Below image shows multiple graph plots of collected metrics values from the XC console 

 

Conclusion: 

F5 Distributed Cloud offers Service Graph API, allowing users to fetch metrics data from XC console and export it to SIEM tools. The custom Python exporter script acts as a bridge between the service graph API and the ELK stack; it fetches the metrics, transforms it, and sends it to Elasticsearch. Later, using the Kibana dashboard, the metrics data can be visualized as graphs, allowing teams to gain insights into the performance, health and behavior of remote systems and enabling them to take quick decisions in real time.