Automation Toolchain - Telemetry Streaming - Grafana StatsD Graphite

Introduction

This article explains how to use the Telemetry Streaming component (TS) of the Automation Tool chain (ATC) for integration with Grafana through StatsD and Graphite. To get more information on the Push Consumer supported by the F5 Networks ATC, more in particular the TS component, please refer to the official documentation on CloudDocs here.

BIG-IP Configuration

In order to configure the TS component of the ATC correctly for integration with Grafana, we will need to post the following JSON blob to your BIG-IP TS API endpoint at https://<BIG-IP-ADDRESS>:8443/mgmt/shared/telemetry/declare

{
 "class": "Telemetry",
 "MyTelemetrySystem": {
   "class": "Telemetry_System",
   "allowSelfSignedCert": true,
   "systemPoller": {
     "interval": 60
   }
 },
 "GraphiteConsumer": {
   "class": "Telemetry_Consumer",
   "type": "Graphite",
   "host": "10.0.0.55",
   "protocol": "http",
   "port": 80
 },
 "StatsdConsumer": {
  "class": "Telemetry_Consumer",
  "type": "Statsd",
  "host":"10.0.0.55",
  "protocol": "udp",
  "port": 8125
 },
 "MyTelemetryListener": {
   "class": "Telemetry_Listener",
   "port": 6514
 }
}

The above 4 JSON stanzas are the following

A Telemetry System class, that sets up the system poller. More information here.
Two Push Consumers classes, that will push the metrics or data externally. In this case to Graphite and StatsD. More information here.
A Telemetry Listener class, that sets an Event Listener )both TCP and UDP protocols) and can accept events in a specific format and process them. More information here

Note that in this example, Graphite and StatsD are running on the same host, because we used a docker container to host them as follows

# docker run -d \
  --name graphite \
  --restart=always \
  -p 80:80 \
  -p 2003-2004:2003-2004 \
  -p 2023-2024:2023-2024 \
  -p 8125:8125/udp \
  -p 8126:8126 \
  graphiteapp/graphite-statsd

Telemetry data

Let's have a look at the TS telemetry data being produced and send over to Graphite. StatsD is used for metrics, Graphite is being used for events

StatsD metrics

StatsD is supporting 3 main metric types: gauges, timers and counters. The TS StatsD integration is using Gauges. We can use Netcat to have a look at the format of these gauge based metrics

# echo "gauges" | nc 10.0.0.55 8126
{
 'statsd.timestamp_lag': 0,
 'f5telemetry.ip-10-0-0-130-eu-west-1-compute-internal.system.networkInterfaces.1-0.counters-bitsIn': 297895992,
 'f5telemetry.ip-10-0-0-130-eu-west-1-compute-internal.system.networkInterfaces.1-0.counters-bitsOut': 0,
 'f5telemetry.ip-10-0-0-130-eu-west-1-compute-internal.system.networkInterfaces.mgmt.counters-bitsIn': 248764520,
 'f5telemetry.ip-10-0-0-130-eu-west-1-compute-internal.system.networkInterfaces.mgmt.counters-bitsOut': 134973160,
 'f5telemetry.ip-10-0-0-130-eu-west-1-compute-internal.system.tmmTraffic.clientSideTraffic-bitsIn': 62854192,
 'f5telemetry.ip-10-0-0-130-eu-west-1-compute-internal.system.tmmTraffic.clientSideTraffic-bitsOut': 229153456,
 'f5telemetry.ip-10-0-0-130-eu-west-1-compute-internal.system.tmmTraffic.serverSideTraffic-bitsIn': 62432120,
 'f5telemetry.ip-10-0-0-130-eu-west-1-compute-internal.system.tmmTraffic.serverSideTraffic-bitsOut': 228977008,
 ...

We can also see the same gauge metrics inside the Graphite admin UI

The structure and path of this telemetry data is important when you create you own dashboards

Graphite events

As mentioned earlier, the TS Graphite integration uses events to send the data to Graphite. You can observe those events by going to the /events endpoint on your Graphite admin UI

The details of such an event are as follows

Grafana

In order to be able to use and display the data now collected in Graphite, one needs to set-up Graphite as a data source and import a Grafana dashboard that uses this data

Graphite data source

Let's add Graphite as a data source

Grafana BIG-IP TS dashboard

Let's import an example dashboard that used the data available. This sample dashboard is also available in the Grafana dashboard collection online here

This sample dashboard will make use of dashboard variables, so users can filter on parameters like Device (which BIG-IP), Tenant (which BIG-IP partition), Application, Virtual Server and Pool. For the sake of demonstration, there is also a filter for Profile

For more information and screenshots on the dashboard itself, refer to the Grafana website where the dashboard is downloadable. The dashboard contains separate rows for

application health status: 4xx and 5xx responses. You can add slow responses as well as a matter of excercise
device system statistics: CPU, memory, TTM traffic in/out, interface traffic in/out
virtual server traffic in/out and server connections
pool traffic in/out and server connections
members traffic in/out and server connections
profile details statistics

The variable queries used for this dashboard are as follows, based on the structure of the metrics data you will find in the Graphite admin UI

Conclusion

In this article we have demonstrated how the F5 Automation Tool Chain, and more in particular also its Telemetry Streaming component, is a perfect match for integration into popular DevOps telemetry solutions. For a fully automated scenario, demonstrating the usage of Declarative Onboarding (DO), Application Services 3 (AS3) and Telemetry Streaming (TS) with automated Grafana integration, you can refer to the following Github repo.

Published May 07, 2020

Version 1.0