F5 Distributed Cloud Telemetry (Logs) - Loki

Scope 

This article walks through the process of integrating log data from F5 Distributed Cloud’s (F5 XC) Global Log Receiver (GLR) with Grafana Loki.  By the end, you'll have a working log pipeline where logs sent from F5 XC can be visualized and explored through Grafana. 

 

Introduction 

Observability is a critical part of managing modern applications and infrastructure. F5 XC offers the GLR as a centralized system to stream logs from across distributed services. 

Grafana Loki, part of the Grafana observability stack, is a powerful and efficient tool for aggregating and querying logs. To improve observability, you can forward logs from F5 XC into Loki for centralized log analysis and visualization. 

This article shows you how to implement a lightweight Python webhook that bridges F5 XC GLR with Grafana Loki. The webhook acts as a log ingestion and transformation service, enabling logs to flow seamlessly into Loki for real-time exploration via Grafana. 

 

Prerequisites 

• Access to F5 Distributed Cloud (XC) SaaS tenant with GLR setup
• VM with Python3 installed  
• Running Loki instance (If not, check "Configuring Loki and Grafana" section below)
• Running Grafana instance (If not, check "Configuring Loki and Grafana" section below)

Note – In this demo, an AWS VM is used with Python3 installed and running webhook (port - 5000), Loki (port - 3100) and Grafana (port - 3000) running as docker instance, all in the same VM. 

Fig 1 : Docker instances

 

 

Architecture Overview 

• F5 XC GLR → Python Webhook → Loki → Grafana

Fig 2 : F5 XC Log Collection Architecture Overview

 

 

F5 XC GLR Configuration 

Follow the steps mentioned below to set up and configure Global Log Receiver (GLR). 

F5 XC GLR

Fig 3 : F5 XC GLR Configuration

 

 

Building the Python Webhook 

To send the log data from F5 Distributed Cloud Global Log Receiver (GLR) to Grafana Loki, we used a lightweight Python webhook implemented using the Flask framework. This webhook acts as a simple transformation and relay service. It receives raw log entries from F5 XC, repackages them in the structure Loki expects, and pushes them to a Loki instance running on the same virtual machine. 

Fig 4 : Python3 code to accept logs from GLR, format and push to Loki

 

 

Key Functions of the Webhook 

• Listens for Log Data: The webhook exposes an endpoint (/glr-webhook) on port 5000 that accepts HTTP POST requests from the GLR. Each request can contain one or more newline-separated log entries. 

• Parses and Structures the Logs: Incoming logs are expected to be JSON-formatted. The webhook parses each line individually and assigns a consistent timestamp (in nanoseconds, as required by Loki). 

• Formats the Payload for Loki: The logs are then wrapped in a structure that conforms to Loki’s push API format. This includes organizing them into a stream, which can be labeled (e.g., with a job name like f5-glr) to make logs easier to query and group in Grafana. 

• Pushes Logs to Loki: Once formatted, the webhook sends the payload to the Loki HTTP API using a standard POST request. If the request is successful, Loki returns a 204 No Content status. 

• Handles Errors Gracefully: The webhook includes basic error handling for malformed JSON, network issues, or unexpected failures, returning appropriate HTTP responses. 

 

Running the Webhook 

• python3 webhook.py > python.log 2>&1 & 

This command runs webhook.py using Python3 in the background and redirects all standard output and error messages to python.log for easier debugging. 

 

Configuring Loki and Grafana

• docker run -d --name=loki -p 3100:3100 grafana/loki:latest

• docker run -d --name=grafana -p 3000:3000 grafana/grafana:latest 

Loki and Grafana are running as docker instance in the same VM, private IP of the Loki docker instance along with port is used as data source in Grafana configuration.

Fig 5 : Configuring Loki data source in Grafana

Once Loki is configured under Grafana Data sources, follow the below steps: 

1. Navigate to Explore menu
2. Select “Loki” in data source picker
3. Choose appropriate label and value, in this case label=job and value=f5-glr
4. Select desired time range and click “Run query”
5. Observe logs will be displayed based on “Log Type” selected in F5 XC GLR Configuration
   Note: Some requests need to be generated for logs to be visible in Grafana based on Log Type selected. 

Fig 6 : Verifying F5 XC GLR logs from Loki in Grafana

 

Conclusion 

F5 Distributed Cloud's (F5 XC) Global Log Receiver (GLR) unlocks real-time observability by integrating with open-source tools like Grafana Loki. This reflects F5 XC's commitment to open source, enabling seamless log management with minimal overhead. A customizable Python webhook ensures adaptability to evolving needs. Centralized logs in Loki and visualized in Grafana empower teams with actionable insights, accelerating troubleshooting and optimization. F5 XC GLR's flexibility future-proofs observability strategies. This integration showcases F5’s dedication to interoperability and empowering customers with community-driven solutions.