Configuring Apache Kafka as push consumer for real time F5 BIG-IP Telemetry Streaming (TS)

Apache Kafka is a distributed event streaming platform designed for real-time data streaming such as telemetry from F5 BIG-IP devices.

Kafka is composed of producers, consumers, topics, partitions, brokers and can be configured using ZooKeeper (traditional architecture) or KRaft mode (newer architecture)

In this technical article, the ZooKeeper mode will be used to configure the broker. We will also install the TS extension on F5 BIG-IP.

First, let’s configure the BIG-IP using the installation steps detailed in the following links:

Install Telemetry Streaming Extension on BIG-IP:

Steps to install Kafka on Ubuntu server

Kafka requires installation of java 11 or higher
Choose a version that includes Zookeeper see ref: https://kafka.apache.org/downloads

Kafka server configuration

root@kafkaDev:~# apt install -y openjdk-21-jdk
root@kafkaDev:~# java -version
root@kafkaDev:~# cd /opt
root@kafkaDev:/opt# wget https://dlcdn.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz
root@kafkaDev:/opt# tar -xvzf kafka_2.13-3.9.0.tgz
root@kafkaDev:/opt# mv kafka_2.13-3.9.0 kafka

Once Kafka is installed configure the broker address using the following steps
Open config/server.properties file using vim find section below and edit to include the server hostname/IP
Also note location where streamed telemetry data will be stored on the broker
Create your own directory if you do not need to use the default location

Kafka configuration settings

root@kafkaDev:~# cd /opt/kafka 

listeners=PLAINTEXT://0.0.0.0:9092 

# Listener name, hostname and port the broker will advertise to clients. 
# If not set, it uses the value for "listeners". 
#advertised.listeners=PLAINTEXT://your.host.name:9092 
advertised.listeners=PLAINTEXT://<server_hostname_IPaddress>:9092 

############################# Log Basics ############################# 

# A comma separated list of directories under which to store log files 
log.dirs=/tmp/kafka-logs

Start ZooKeeper in a separate bash terminal and start Kafka in the current session

Start ZooKeeper service

root@kafkaDev:/opt/kafka# bin/zookeeper-server-start.sh config/zookeeper.properties

In a separate Linux terminal start Kafka

Start Kafka server

root@kafkaDev:/opt/kafka# bin/kafka-server-start.sh config/server.properties

Open a new terminal and create a topic

Create Kafka topic

root@kafkaDev:/opt/kafka# bin/kafka-topics.sh --bootstrap-server serverIP:9092 --create --topic test --partitions 1 --replication-factor 1

Next we produce a message
Type messages in the terminal and press Enter to send

Produce Kafka message

root@kafkaDev:/opt/kafka# bin/kafka-console-producer.sh --bootstrap-server serverIP:9092 --topic test >2019, 2023, 2024

Then consume a message
We should see the messages sent by the producer

Consume message produced

root@kafkaDev:/opt/kafka# bin/kafka-console-consumer.sh --bootstrap-server serverIP:9092 --topic test --from-beginning 2019, 2023, 2024

BIG-IP TS configuration

Validate TS installed correctly on BIG-IP

Example to validate TS installed correctly

[root@bigip.local:TimeLimitedModules::Active:Standalone] config # curl -sku admin:admin https://mgmt_IP/mgmt/shared/telemetry/info | jq .
{
  "branch": "v1.37.0",
  "buildID": "8aec8953",
  "buildTimestamp": "20241017213926",
  "fullVersion": "1.37.0-1",
  "nodeVersion": "v8.11.1",
  "release": "1",
  "schemaCurrent": "1.37.0",
  "schemaMinimum": "0.9.0",
  "version": "1.37.0"
}

Example declaration to collect pool member and virtual server traffic stats from BIG-IP

Example TS declaration

root@kafkaDev:~# cat defaultTelemetry.js
{
  "class": "Telemetry",
  "controls": {
    "class": "Controls",
    "logLevel": "info"
  },
  "My_System": {
    "class": "Telemetry_System",
    "systemPoller": {
      "interval": 60,
      "chunkSize": 1,
      "actions": [
            {
                "includeData": {},
                "locations": {
                    "system": {
                        "hostname": true
                    },
                    "pools": {
                        "/Common/poolA": {
                            "members": {
                                "/Common/10.68.213.2:80": true,
                                "/Common/10.68.213.7:80": true
                            }
                        }
                    },
                    "virtualServers": {
                        ".*": {
                            "name": true,
                            "clientside.bitsIn": true
                        }
                    }
                }
            }
        ]

    }
  },
   "My_Listener": {
        "class": "Telemetry_Listener",
        "trace": [
            {
            "type": "input"
        },
        {
            "type": "output"
        }
    ]
    },
  "Kafka_Consumer": {
    "class": "Telemetry_Consumer",
    "type": "Kafka",
    "host": "10.68.213.34",
    "port": 9092,
    "protocol": "binaryTcp",
    "topic": "defaultTelemetry",
    "format": "default",
    "compressionType": "None"
  }
}

A successful post to the JSON endpoint should return success message

TS declaration to BIG-IP client endpoint

root@kafkaDev:~# curl -ksu admin:admin -d "@defaultTelemetry.js" -H "Content-Type: application/json" https://mgmt_IP/mgmt/shared/telemetry/declare | jq .    {
  "message": "success",
  "declaration": {
    "class": "Telemetry",
    "controls": {
      "class": "Controls",
      "logLevel": "info",
      "debug": false,
      "memoryThresholdPercent": 90
    },
    "My_System": {
      "class": "Telemetry_System",
      "systemPoller": {
        "interval": 60,
        "chunkSize": 1,
        "actions": [
          {
            "includeData": {},
            "locations": {
              "system": {
                "hostname": true
              },
              "pools": {
                "/Common/poolA": {
                  "members": {
                    "/Common/10.68.213.2:80": true,
                    "/Common/10.68.213.7:80": true
                  }
                }
              },
              "virtualServers": {
                ".*": {
                  "name": true,
                  "clientside.bitsIn": true
                }
              }
            },
            "enable": true
          }
        ],
        "enable": true,
        "workers": 5
      },
      "enable": true,
      "host": "localhost",
      "port": 8100,
      "protocol": "http",
      "allowSelfSignedCert": false
    },
    "My_Listener": {
      "class": "Telemetry_Listener",
      "trace": [
        {
          "type": "input"
        },
        {
          "type": "output"
        }
      ],
      "enable": true,
      "port": 6514,
      "match": "",
      "actions": [
        {
          "setTag": {
            "tenant": "`T`",
            "application": "`A`"
          },
          "enable": true
        }
      ]
    },
    "Kafka_Consumer": {
      "class": "Telemetry_Consumer",
      "type": "Kafka",
      "host": "Kafka_server_IP",
      "port": 9092,
      "protocol": "binaryTcp",
      "topic": "defaultTelemetry",
      "format": "default",
      "compressionType": "None",
      "enable": true,
      "trace": false,
      "allowSelfSignedCert": false,
      "authenticationProtocol": "None",
      "partitionerType": "default"
    },
    "schemaVersion": "1.37.0"
  }
}

on the Kafka broker we should see the streamed data based on the topic

topic: defaultTelemetry

root@kafkaDev:/tmp/kafka-logs/defaultTelemetry-0# strings *.log | jq .
{
  "system": {
    "hostname": "bigip.local"
  },
  "virtualServers": {
    "/Common/virtualA": {
      "clientside.bitsIn": 13848,
      "name": "/Common/virtualA"
    }
  },
  "pools": {
    "/Common/poolA": {
      "members": {
        "/Common/10.68.213.2:80": {
          "addr": "10.68.213.2",
          "monitorStatus": "up",
          "mr.msgIn": 0,
          "mr.msgOut": 0,
          "mr.reqIn": 0,
          "mr.reqOut": 0,
          "mr.respIn": 0,
          "mr.respOut": 0,
          "poolName": "/Common/poolA",
          "port": 80,
          "serverside.bitsIn": 3784,
          "serverside.bitsOut": 30240,
          "serverside.curConns": 0,
          "serverside.maxConns": 1,
          "serverside.pktsIn": 6,
          "serverside.pktsOut": 6,
          "serverside.totConns": 1,
          "availabilityState": "available",
          "enabledState": "enabled",
          "status.statusReason": "Pool member is available",
          "totRequests": 1
        },
        "/Common/10.68.213.7:80": {
          "addr": "10.68.213.7",
          "monitorStatus": "up",
          "mr.msgIn": 0,
          "mr.msgOut": 0,
          "mr.reqIn": 0,
          "mr.reqOut": 0,
          "mr.respIn": 0,
          "mr.respOut": 0,
          "poolName": "/Common/poolA",
          "port": 80,
          "serverside.bitsIn": 7568,
          "serverside.bitsOut": 60480,
          "serverside.curConns": 0,
          "serverside.maxConns": 2,
          "serverside.pktsIn": 12,
          "serverside.pktsOut": 12,
          "serverside.totConns": 2,
          "availabilityState": "available",
          "enabledState": "enabled",
          "status.statusReason": "Pool member is available",
          "totRequests": 2
        }
      }
    }
  }
}

Published Feb 03, 2026

Version 1.0

Employee

Escalation Engineer @F5Networks. I specialize in F5 CIS deployments/troubleshooting, automating BIG-IP on AWS/Azure deployments using cloud CLI tools, infrastructure-as-code with Terraform, Go web development, and providing API-driven solutions Tech stack: F5 CIS | Kubernetes | Terraform | Go | AWS/Azure CLI | REST APIs | Telemetry/AS3

View Profile

Configuring Apache Kafka as push consumer for real time F5 BIG-IP Telemetry Streaming (TS)

BIG-IP TS configuration

ABOUT DEVCENTRAL

RESOURCES

SUPPORT

PARTNERS