Telemetry Streaming: getting HTTP statistics via SNMP
Hi F5 community, I am looking to get HTTP statistics (total count, and broken by response code) metrics from Telemetry Streaming via SNMP (seems to be the most viable option). F5-BIGIP-LOCAL-MIB::ltmHttpProfileStat oid: .1.3.6.1.4.1.3375.2.2.6.7.6 However, the stats don't seem to come out correct at all: I do see deltas happening, but they don't match at all the traffic rate I expect to see. Furthermore, I have done some tests where I would start a load testing tool (vegeta) to fire concurrent HTTP requests, for which I do see the logs from the virtual server, but no matching increment in the above SNMP OID entries on none of the profiles configured. What am I doing wrong? does something need to be enabled on the HTTP profile in use to collect those stats? Best, Owayss
GTM - Lingo translated from an LTM perspective (and monitoring)
Dear experts I've set out to find typically good stuff to monitor on the GTM and to do that I had to re-evaluate some terms and things I took for granted from the LTM world. I thought I'd share what I have written down. Hopefully it will help some other poor soul coming from the LTM side to get the grip of the basics. Please do comment if (or actually rather when) I've misunderstood something? A diagram of a basic set-up Lingo Server This is the equivalent of an F5 unit (or a generic server). Data center This is more or less just a container with Servers in them. Data center availability is determined by doing an aggregate checks of virtual server statuses. If all virtual servers are down, the data-center is down, otherwise not. Virtual server These are the classical virtual servers we know from the LTM. Wide IP This is the actual dns record and it's aliases. The wide IP uses a pool containing one or more LTM virtual servers (or, a manually defined fallback server IP). The GTM uses the pool to select the best server based on criteria defined by the admin (GEO-IP based, connections, just plain round-robin, irules etc.). Link A link is a possible path to the internet. In case a GTM is connected to multiple routers it can also use multiple links in case one, or more of the routers is unavailable. If all links are down, the objects monitored via those links also goes down. If no link is specified the GTM would use the configured routing. Listeners This is where the clients would send their DNS queries. You must have at least one in order for GTM to work, but more than one is also possible. Monitoring of virtual servers The GTM would not monitor F5 devices like the LTM monitors its pool members and nodes. Instead it establishes a trust with the devices by running the bigip_add (a bit like device trust) and then the monitoring is done through iQuery. Monitoring In order to get a basic monitoring going I've thought out the following parts. Please let me know if something should be added? Availability: Server availability Data center availability Virtual server availablity Wide IP availability* Link availability Status of iQuery Statistics: Memory, CPU, throughput etc Requests per Wide IP Unhandled requests per Wide IP Total requests per GTM Total unhandled requests per GTM Any comments/feedback very much appreciated!Show iRule SNAT Statistics
I have a number of LTM wildcard Virtual Servers defined which have iRules applied that NAT the source address. Here is an example of an iRule: text when CLIENT_ACCEPTED { snat 1.1.1.1 } text I am seeing intermittent dropped connections and I am trying to determine if I am exhausting the ephemeral ports. I have tried running 'tmsh show ltm snat detail', but that only appears to display the statistics for defined SNATs and not SNAT functions generated from an iRule. Is there a way to show the statistics for a SNAT that is defined in an iRule, but not defined in the SNAT list?Get system compression statistics from command line?
Hi! Does anyone know a way of getting system wide compression throughput from the command line? I need to get certain metrics limited by licenses, but have only found SSL TPS and throughput via tmsh, not compression. Tmsh and bash is what I'm allowed to use and I can't use SNMP as it has to be compatible with all systems, even those with SNMP disabled (doing this for a client with very specific needs and methods). Example: show sys performance all-stats Sys::Performance System ------------------------------------------------------------------- System CPU Usage(%) Current Average Max(since 11/06/16 12:37:17) ------------------------------------------------------------------- Utilization 28 28 43 ------------------------------------------------------------------- Memory Used(%) Current Average Max(since 11/06/16 12:37:17) ------------------------------------------------------------------- TMM Memory Used 11 11 11 Other Memory Used 63 63 63 Swap Used 0 0 0 Sys::Performance Connections --------------------------------------------------------------------------- Active Connections Current Average Max(since 11/06/16 12:37:17) --------------------------------------------------------------------------- Connections 45.1K 43.2K 49.4K --------------------------------------------------------------------------- Total New Connections(/sec) Current Average Max(since 11/06/16 12:37:17) --------------------------------------------------------------------------- Client Connections 916 962 1.1K Server Connections 745 782 922 --------------------------------------------------------------------------- HTTP Requests(/sec) Current Average Max(since 11/06/16 12:37:17) --------------------------------------------------------------------------- HTTP Requests 2.8K 2.8K 3.3K Sys::Performance Throughput ----------------------------------------------------------------------------- Throughput(bits)(bits/sec) Current Average Max(since 11/06/16 12:37:17) ----------------------------------------------------------------------------- Service 644.2M 622.5M 807.2M In 659.1M 638.1M 828.2M Out 303.5M 300.6M 375.6M ----------------------------------------------------------------------------- SSL Transactions Current Average Max(since 11/06/16 12:37:17) ----------------------------------------------------------------------------- SSL TPS 599 640 790 ----------------------------------------------------------------------------- Throughput(packets)(pkts/sec) Current Average Max(since 11/06/16 12:37:17) ----------------------------------------------------------------------------- Service 78.7K 77.1K 97.6K In 78.5K 77.1K 97.8K Out 68.4K 67.8K 84.1K Sys::Performance Ramcache ------------------------------------------------------------------------ RAM Cache Utilization(%) Current Average Max(since 11/06/16 12:37:17) ------------------------------------------------------------------------ Hit Rate 61 63 70 Byte Rate 67 67 75 Eviction Rate 8 14 28 Any input appreciated! /PatrikBIG-IQ statistics in dashboard missing/interrupted
Dear all, any idea, why statistics data is missing/interrupted for a few days like in the following screenshot? Or how I can verify any issues? In which logs can I found any hints? The missing data is visible for device statistics as well as for Local Traffic. Am I right, that these statistics are the ones, which are stored on the DCDs? And why is there no data before 27.11.2022? Is there any limit configured that just the last 30 days (or so) will be stored? Does this indicate to an connectivity issue between the DCDs and the BIG-IPs? As of now I could only verify that the ucs backups were successfully executed during these days, so connectivity between the BIG-IQ and the BIG-IPs was fine? Thanks for any useful ideas or hints. Regards Stefan 🙂Orchestrated Infrastructure Security - BIG-IQ
The F5 Beacon capabilities referenced in this article hosted on F5 Cloud Services are planning a migration to a new SaaS Platform - Check out the latesthere. Introduction This article is part of a series on implementing Orchestrated Infrastructure Security. It includes High Availability, Central Management with BIG-IQ, Application Visibility with Beacon and the protection of critical assets using F5 Advanced WAF and Protocol Inspection (IPS) with AFM.It is also assumed that BIG-IQ is deployed, and basic network connectivity is working. If you need help setting up BIG-IQ for the first time, refer to the Dev/Central article series Implementing SSL Orchestrator here.That article covers SSL Orchestrator but the procedure to add Advanced WAF and AFM to BIG-IQ is the same. This article focuses on configuring BIG-IQ version 7.1.0 to manage F5 Advanced WAF, AFM and SSL Orchestrator.It covers management of BIG-IP running version 15.1.0.4 and SSL Orchestrator version 7.4.9, and version 16.0.0 with AFM and Advanced WAF. Please forgive me for using SSL and TLS interchangeably in this article. This article is divided into the following high level sections: Import BIG-IP Devices into BIG-IQ Service Import Error Resolution Schedule regular backups of BIG-IP devices Push backups to BIG-IP device Import BIG-IP Devices into BIG-IQ From the BIG-IQ GUI go to Devices > BIG-IP Devices.This is where you add new devices to be managed by BIG-IQ.You should add the two SSL Orchestrator’s using the Dev/Central article above.Click Add Device(s) to add Advanced WAF and AFM devices. Select the option to Add BIG-IP device(s) and automatically discover and import services.Then click Add Devices. Enter the IP Addresses of the Devices you want to add, 192.168.41.3 and 192.168.41.4 in this example (use the Plus sign to add another IP address field).These are the two AFM devices.Enter the username and password to access these devices.Under Services check the box for Network Security (AFM) then scroll down. Check the box to enable Statistics Collection.You can configure a Zone and/or Cluster Display Name if desired.Click Save and Close. Your screen should look like the following.Click Add Devices so we can add the two Advanced WAFs. Enter the IP Addresses of the Devices you want to add, 192.168.41.21 and 192.168.41.22 in this example (use the Plus sign to add another IP address field).These are the two Advanced WAF devices.Enter the username and password to access these devices.Under Services check the box for Web Application Security (ASM) then scroll down. Check the box to enable Statistics Collection.You can configure a Zone and/or Cluster Display Name if desired.Click Save and Close. Click Discover and Import. You should see a Progress screen.Click Close. When complete, your screen should look similar to the following.= Service Import Error Resolution Some devices had errors during Import.Click the first one to resolve it. There was a conflict importing SSM.Check the box to create a snapshot of the configuration then click Import. The following items were changed on the BIG-IP.You can choose to import these into the BIG-IQ by selecting Set all BIG-IP.Click Continue. A dialog screen will present you with more information about what you’re doing.Click Resolve. Click Import to complete the import process.You may want to create a Snapshot of the configuration by checking the box. The BIG-IP Devices screen should look like this.The Advanced WAF device has been successfully imported.Repeat this process for any devices with an import error. When all Devices are successfully imported the screen should look like this. Schedule regular backups of BIG-IP Devices Now is a good time to schedule regular Backups.Check the box next to Status to select all the BIG-IPs.Click the down Arrow next to More and select Schedule Backup. Give the Backup a name, Backup_all in this example.There are several options here that you may wish to enable.For Local Retention Policy, it’s not a bad idea to keep multiple backups, 3 in this example.The Start Date and time can be adjusted to suit your needs. The Devices should automatically be selected.You can optionally enable the Archiving of Backups to an external SCP or SFTP server.Click Save & Close. Push backups to BIG-IP Device At some point you may need to restore one of your BIG-IP devices from a backup.To do this select the Devices tab > Back Up & Restore > Backup Files. From here you can view the different backup files.You can also Compare, Download, Restore or Delete backup files.Select the backup you would like to restore then click Restore. You will be presented with a confirmation message warning you that the configuration of the device is about to be overwritten from the backup.Click Restore to proceed. While the device is being restored you will see the following. Select BIG-IP Devices to check the status of the device when the restore is complete. Summary In this article you learned how to import BIG-IP devices into BIG-IQ, import the BIG-IP Services and schedule regular backups of the BIG-IP devices. Next Steps Click Next to proceed to the next article in the series.
New Connections - missing statistics
Hello, I am wondering why there is no data for new connections in the analytics screen Statistics ›› Analytics : Virtual Servers : TCP : Connections I can see avg active and concurrent connections (Traffic Details:Connections) just fine, but I am interested in the connection attempts per second and not the avg active. Is there any logging profile that needs to be activated for that? Running v16Add a Data Collection Device to your BIG-IQ Cluster
Gathering and analyzing data helps organizations make intelligent decisions about their IT infrastructure. You may need a data collection device (DCD) to collect BIG-IP data so you can manage that device with BIG-IQ. BIG-IQ is a platform that manages your devices and the services they deliver. Let’s look at how to discover and add a data collection device in BIG-IQ v5.2. You can add a new data collection device to your BIG-IQ cluster so that you can start managing it using the BIG-IP device data. In addition to Event and Alert Log data, you can view and manage statistical data for your devices. From licensing to policies, traffic to security, you’ll see it all from a single pane of glass. But you need a DCD to do that. So, we start by logging in to a BIG-IQ. Then, under the System tab, go to BIG-IQ Data Collection and under that, click BIG-IQ Data Collection Devices. The current DCD screen shows no devices in this cluster. To add a DCD, click Add. This brings us to the DCD Properties screen. For Management Address field, we add the management IP address of the BIG-IP/DCD we want to manage. We’ll then add the Admin username and password for the device. For Data Collection IP Address, we put the transport address which is usually the internal Self-IP address of the DCD and click Add. The process can take a little while as the BIG-IQ authenticates with the BIG-IQ DCD and adds it to the BIG-IQ configuration. But once complete, you can see the devices has been added successfully. Now you’ll notice that the DCD has been added but there are no Services at this point. To add Services, click Add Services. In this instance, we’re managing a BIG-IP with multiple services including Access Policies so we’re going to activate the Access services. The listener address already has the management address of the DCD populated so we’ll simply click Activate. Once activated, you can see that it is Active. When we go back to the Data Collection Devices page, we can see that the Access Services have been added and the activation worked. Congrats! You’ve added a Data Collection Device! You can also watch a video demo of How to Add a data collection device to your BIG-IQ cluster. ps Related: Lightboard Lesson: What is BIG-IQ?
Analytics : HTTP : Transactions - Export Top URL
Hello, I'm using AVR (Application Visibility Reporting)for my F5 Big IP LTM traffic. When i'm going to my statistics, I can see 29 975 total entries for my URL. But When I try to export all of these URLs in CSV format, my CSV file contains only the first 10 entries. Do I need to apply a particular method to export all of my 29,975 entries to the same CSV file to process this data? Thanks for your help, Hugo
