GTM Pool Members Gone After Maintenance? It's Probably This One Setting
You finish a maintenance window, everything looks good on LTM, and then someone notices Wide IPs are resolving to fewer destinations than before. You check the GTM pools and the members are just... gone. The virtual servers are fine on LTM. GTM just doesn't know about them anymore — and more importantly, it doesn't remember if they were ever pool members. This happens more often than it should, and it almost always comes back to the same thing: virtual-server-discovery enabled doing exactly what it was designed to do, at exactly the wrong moment. What's Actually Going On When virtual-server-discovery is set to enabled on a GTM server object, GTM keeps its view of LTM virtual servers in sync via iQuery. It automatically adds new virtual servers, updates existing ones, and — this is the part that causes problems — deletes virtual servers that LTM stops reporting on. That delete behavior is the issue. Any time iQuery reports zero virtual servers, even temporarily, GTM treats it as a mass deletion event. The virtual servers get pulled from the server object, and with them, their pool memberships. When LTM eventually reports on those virtual servers again, GTM re-discovers them as brand new objects with no memory of which pools they belonged to. Two scenarios trigger this consistently. Scenario 1: LTM Software Upgrade This is the one that catches most people. During an upgrade, LTM reboots and goes through a phase where iQuery can connect but the full configuration hasn't finished loading yet. From GTM's perspective, LTM is reachable but reporting no virtual servers. GTM interprets that as a deletion event, clears out the discovered virtual servers, and empties the pools. When LTM finishes loading and the virtual servers come back, GTM re-discovers them — but the pool memberships are gone. You're left manually rebuilding what was there before the maintenance window started. The telltale sign is pool members coming back in blue/CHECKING state. That only happens to newly discovered objects. GTM treated a returning virtual server as a brand new one — because as far as it's concerned, it is. The GTM log won't show a deletion event, only the re-add. That gap in the logs is a known blind spot with virtual-server-discovery enabled, and it's exactly why the problem is hard to diagnose after the fact. What you'll typically see in /var/log/gtm after the LTM comes back: alert gtmd[xxxxx]: 011a1005:1: SNMP_TRAP: Pool your_pool state change green --> red (No enabled pool members available) alert gtmd[xxxxx]: 011a3004:1: SNMP_TRAP: Wide IP your.wideip.example.com state change green --> red (No enabled pools available) And then shortly after, the virtual servers re-appear in CHECKING state as GTM re-discovers them — but with no pool bindings. Scenario 2: LTM HA Failover This one surprises people because the LTM pair is still running — it's just switching active units. After a failover, the new active device may not have its iQuery connections fully re-established yet. GTM sees the iQuery state as inconsistent, virtual server status updates stop coming through, and members disappear from the discovered list. What makes this harder to diagnose is that tmsh show gtm iquery may show "connected" — but connected doesn't mean the config sync is working correctly. In a GTM sync group, only the device assigned local ID 0 (the GTM with the lowest IP address) is responsible for writing auto-discovery results to the configuration. If that specific device loses its iQuery connection during the failover window, discovery events are missed entirely — even if every other GTM in the group can still reach the LTM. So you can have a situation where five out of six GTMs look perfectly healthy, iQuery shows connected everywhere, and yet pool members are still disappearing — because the one device that matters for discovery is the one with the broken connection. You can check which device in your sync group holds local ID 0 with: tmsh list sys db gtm.peerinfolocalid If that device's iQuery connection to the LTM is the one that dropped during the failover window, that's your answer — even if everything else looks fine. The Fix: enabled-no-delete Both scenarios share the same root cause: GTM's auto-delete behavior treating a temporary iQuery disruption as a permanent deletion event. The fix is the same for both: gtm server /Common/site1-ltm { addresses { 10.1.1.1 { device-name site1-ltm } } datacenter /Common/dc1 monitor /Common/bigip virtual-server-discovery enabled-no-delete } With enabled-no-delete, GTM still auto-discovers new virtual servers and keeps existing ones updated. The only thing that changes is that it will never delete a virtual server just because LTM temporarily stopped reporting it. Your pool memberships survive both scenarios above. Mode Adds new VS Updates VS Deletes VS Pool memberships survive iQuery disruption? disabled No No No Yes — nothing changes enabled Yes Yes Yes No — any disruption can empty pools enabled-no-delete Yes Yes No Yes — preserved The Trade-Off enabled-no-delete won't clean up after you when you intentionally decommission a virtual server on LTM. The stale GTM object stays in the discovered list until you remove it manually. In environments with a lot of VS churn, this can accumulate over time. The question is which failure mode you'd rather manage: pool members silently disappearing during a maintenance window, or occasionally needing to clean up stale objects after a planned decommission. For most production environments, the latter is far easier to deal with — and far less likely to wake someone up at 2am. How to Make the Change Via tmsh: tmsh modify gtm server /Common/site1-ltm \ virtual-server-discovery enabled-no-delete tmsh save sys config Via GUI: Go to DNS → GSLB → Servers Select the server object Set Virtual Server Discovery to Enabled (No Delete) Click Update This takes effect immediately and does not affect existing discovered virtual servers or current pool memberships. Cleaning Up Stale Objects When you intentionally decommission a virtual server on LTM, remove the leftover GTM object manually: # List virtual servers under a GTM server object tmsh list gtm server /Common/site1-ltm virtual-server # Remove a specific stale entry tmsh modify gtm server /Common/site1-ltm \ virtual-servers delete { /Common/old-vs-name } tmsh save sys config Make this part of your standard VS decommission runbook and stale objects will never pile up. Quick Diagnostic When Members Go Missing Before assuming it's a discovery issue, check iQuery health across all GTM devices first: tmsh show gtm iquery Look for: State: should be connected to all entries Reconnects: A high count suggests instability even if the connection looks up Configuration Time: None means the config has never successfully synced from that LTM Then confirm which GTM holds local ID 0 and verify its connectivity specifically: tmsh list sys db gtm.peerinfolocalid If the local ID 0 device is the one with the broken iQuery connection, that's your answer — regardless of what the other devices are showing. Wrapping Up Whether it's an LTM upgrade or an HA failover, the pattern is the same: iQuery goes quiet for a moment, GTM interprets silence as deletion, and your pool memberships are gone. It's working as designed — just not in a way that's useful to you. enabled-no-delete is a one-line change that stops this from happening. The cleanup overhead it introduces is predictable and manageable. The alternative — rebuilding pool memberships after an unplanned event — is not. Have you run into either of these scenarios in your environment? Drop a comment below, especially if you've seen the local ID 0 shift cause issues during a rolling GTM upgrade.
irule to insert http header with values based on received radius accounting avp
I have a requirement to insert msisdn gotten from radius logs into https virtual server (http irule). It is pretty similar to this. insert HTTP header according to a value received in Radius accounting | DevCentral " Radius accounting arrives with following attributes: Acct-Status-Type=1 Calling-Station-Id=123456789 Framed-IP-Address=1.1.1.1 An entry is inserted to a table 1.1.1.1 (key) -> 123456789 (value) HTTP request arrives from source IP 1.1.1.1 The F5 will insert a header "MSISDN: 123456789" to the HTTP request before forwarding. Radius accounting arrives with following attributes: Acct-Status-Type=2 Calling-Station-Id=123456789 Framed-IP-Address=1.1.1.1 The value is removed 1.1.1.1 (key) -> (empty) HTTP request arrives from source IP 1.1.1.1 The F5 will not insert a header (not "MSISDN: 123456789" and not "MSISDN: ") " I have created two virtual servers. one , a radius accounting virtual server (with no pool members) on port 1813. it uses this irule. when CLIENT_DATA { set avp40 [RADIUS::avp 40 integer] set client_ip [RADIUS::avp 8 ip4] set msisdn [RADIUS::avp 31 string] if { $avp40 == 1 || $avp40 == 3 } { # Start or Interim-Update → refresh TTL table set -subtable "ip2msisdn" $client_ip $msisdn 540 } elseif { $avp40 == 2 } { table delete -subtable "ip2msisdn" $client_ip } } The HTTPs virtual server with ssl termination, has following irule. when HTTP_REQUEST { set client_ip [IP::client_addr] set msisdn [table lookup -subtable "ip2msisdn" $client_ip] if { $msisdn ne "" } { HTTP::header insert "X-MSISDN" $msisdn HTTP::header insert "X-Auth-Status" "AUTHORIZED" } else { HTTP::header insert "X-Auth-Status" "UNKNOWN" HTTP::header insert "X-MSISDN" "NOT_FOUND" } } we see several logs when logs are added on radius irule. Yet, we never get to see ip-msisdn match, when user browser attempts to access the website. We can confirm radius feeds are being sent by GGSN. example. Aug 20 13:14:02 slot1/OAU info tmm6[19301]: Rule /Common/awo <CLIENT_DATA>: RADIUS START: 10.63.27.96 -> 2347038201355 Aug 20 13:14:02 slot1/OAU info tmm16[19302]: Rule /Common/awo <CLIENT_DATA>: RADIUS STOP: removed 10.36.203.125 Aug 20 13:14:02 slot1/OAU info tmm6[19301]: Rule /Common/awo <CLIENT_DATA>: RADIUS STOP: removed 10.36.203.125 Aug 20 13:14:02 slot1/OAU info tmm16[19302]: Rule /Common/awo <CLIENT_DATA>: RADIUS START: 10.63.27.62 -> 2347069332877 Aug 20 13:14:02 slot1/OAU info tmm8[19301]: Rule /Common/awo <CLIENT_DATA>: RADIUS START: 10.63.27.62 -> 2347069332877 Aug 20 13:14:02 slot1/OAU info tmm17[19302]: Rule /Common/awo <CLIENT_DATA>: RADIUS STOP: removed 10.29.44.232 Aug 20 13:14:02 slot1/OAU info tmm17[19302]: Rule /Common/awo <CLIENT_DATA>: RADIUS STOP: removed 10.38.116.115 Aug 20 13:14:02 slot1/OAU info tmm17[19302]: Rule /Common/awo <CLIENT_DATA>: RADIUS START: 10.63.15.231 -> 2347070645170 Aug 20 13:14:02 slot1/OAU info tmm17[19302]: Rule /Common/awo <CLIENT_DATA>: RADIUS STOP: removed 10.29.44.232 Aug 20 13:14:02 slot1/OAU info tmm17[19302]: Rule /Common/awo <CLIENT_DATA>: RADIUS STOP: removed 10.38.116.115 Aug 20 13:14:02 slot1/OAU info tmm17[19302]: Rule /Common/awo <CLIENT_DATA>: RADIUS START: 10.63.15.231 -> 2347070645170 Aug 20 13:14:02 slot1/OAU info tmm6[19301]: Rule /Common/awo <CLIENT_DATA>: RADIUS START: 10.63.15.190 -> 2348137835053 Aug 20 13:14:02 slot1/OAU info tmm[19301]: Rule /Common/awo <CLIENT_DATA>: RADIUS START: 10.63.15.224 -> 2348140105877 . How do we force write the radius logs to memory on F5. The F5 is a VIPRION.
Unable to use a single CRL on BIG-IP rSeries tenant (works on BIG-IP VE)
Hello everyone, I'm experiencing an issue with CRL validation on the BIG-IP tenant. I have a single CRL file (approximately 35 MB) downloaded directly from our CA. The CRL imports successfully into BIG-IP (either .pem or .crl format) without any errors. However, when I assign this CRL to a Client SSL profile configured for client certificate authentication, an error occurs. The following message is logged in /var/log/ltm: "Profile /Common/<clientssl_profile> - cannot load <CRL file> CRL file error: Unknown error." After that, as expected, access to the application published through this Client SSL profile no longer works correctly. The same CRL imports successfully on a BIG-IP VE running the same TMOS version. Has anyone experienced similar behavior or knows whether this could be a platform-specific limitation or a known issue? Any suggestions would be appreciated. Thank you.Finding the Primary Replica of an Availability Group
Hi All, I've configured VIPs and a WIP for a SQL Always On service and would like to ensure traffic is directed to the current Primary Replica wherever possible. The configuration is as follows: DC1 hosts a SQL server and a corresponding VIP. DC2 hosts a SQL server and a corresponding VIP. The WIP currently directs traffic to the DC1 VIP using Global Availability, with a fallback to Round Robin if required. Does anyone know of a way for the F5 infrastructure to identify which SQL Always On replica is currently the Primary Replica and automatically direct clients to the associated VIP across the two data centres? Any advice or experience with this would be greatly appreciated. Thanks in advance.
An Irule for Client Ssl Profile that Allows Unassigned TLS Extension Values (17516)
Hello Community, I have a requirement to allow enriched https header enrichment. The SSL negotiation (I'm doing ssl termination on F5) fails because the enriched header from client contains reserved tls extension values. (https://www.iana.org/assignments/tls-extensiontype-values/tls-extensiontype-values.xhtmltls-extensiontype-values-1). The Client Hello request in the SSL Handshake was captured and contained an Extensions list, which included a reserved TLS Extension value (17156), which the F5 isn't presenting in Server Hello. I need an irule that can allow that Extension to be added on the client ssl profile so the ssl handshake doesn't fail.
Weblogic JSessionID Persistence
Problem this snippet solves: Contributed by: unRuleY, Summarized by: deb Note: The previous version of this iRule contained escaped newlines following the session command, which in versions 10.0 - 10.2.0 causes TMM to core as documented in CR135937 / SOL11427. This was fixed in 10.2.1. See this related Codeshare example for details on how to take advantage of session replication on the WebLogic servers with targeted node failover in an iRule. Provides persistence on the jsessionid value found in either the URI or a cookie. When a request is received, the iRule first looks for a "jsessionid" cookie, and if not found, for a "jsessionid" parameter in the requested URI. If either is found, a persistence record is created if it doesn't already exist, or followed if it does. If neither is found, the request is load balanced according to the load balancing method applied to the virtual server and persisted based on the client's IP address. In order to ensure the second and subsequent requests follow the first, LTM must create a persistence record indicating the pool member to which the first request was load balanced. If the server is setting the jsessionid in a cookie, the persistence key value may be extracted from the server response to create the persistence record. If the server is setting the jsessionid in the URLs, source address persistence with a short timeout is recommended to track the original destination until the jsessionid is sent. How to use this snippet: To ensure a new persistence record is followed when a request is re-load balanced in a client-side Keep-Alive connection, apply a OneConnect profile to the virtual server. The iRule assumes the jsessionid is in upper case when used as a cookie name. If this isn't the case, please update the example. To persist on jsessionid, create the iRule below and create a custom Universal persistence profile, with Match Across Services enabled, that uses the iRule. Then use this custom Universal persistence profile as the Default Persistence profile on your Virtual Server. Applying a Fallback Persistence profile of type Source Address Affinity with a host mask and a short timeout (the default source_addr persistence profile will do the trick) to your Virtual Server is also recommended. Attention, if you are running firmware 11.0 - 11.2.1 and enabled "Match Across Services"! There is a bug inside. SOL14061 This iRule requires LTM v10. or higher. Code : when HTTP_REQUEST { # Log details for the request set log_prefix "[IP::client_addr]:[TCP::client_port]" log local0. "$log_prefix: Request to [HTTP::uri] with cookie: [HTTP::cookie value JSESSIONID]" # Check if there is a JSESSIONID cookie if { [HTTP::cookie "JSESSIONID"] ne "" }{ # Persist off of the cookie value with a timeout of 1 hour (3600 seconds) persist uie [string tolower [HTTP::cookie "JSESSIONID"]] 3600 # Log that we're using the cookie value for persistence and the persistence key if it exists. log local0. "$log_prefix: Used persistence record from cookie. Existing key? [persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]]]" } else { # Parse the jsessionid from the path. The jsessionid, when included in the URI, is in the path, # not the query string: /path/to/file.ext;jsessionid=1234?param=value set jsess [findstr [string tolower [HTTP::path]] "jsessionid=" 11] # Use the jsessionid from the path for persisting with a timeout of 1 hour (3600 seconds) if { $jsess != "" } { persist uie $jsess 3600 # Log that we're using the path jessionid for persistence and the persistence key if it exists. log local0. "$log_prefix: Used persistence record from path: [persist lookup uie $jsess]" } } } when HTTP_RESPONSE { # Check if there is a jsessionid cookie in the response if { [HTTP::cookie "JSESSIONID"] ne "" }{ # Persist off of the cookie value with a timeout of 1 hour (3600 seconds) persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 3600 log local0. "$log_prefix: Added persistence record from cookie: [persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]]]" } }An Irule that Stops TLS (tcp) Fragmentation on Client Side
There is a requirement to implement a service that extracts Inserted traffic in extension header from subscribers. the insertion is done by PGW on packet core, and works on server side, which can successfully extract the information from the extension header. When the PGW Succesfully inserts the added extension, the solution on serverside works well. The challenge is that the PGW frequently truncates the TLS Header (TCP), and the serverside (F5), is unable to see the added extension header, since it has been cut off. A technical explanation is goes thus. "The EPG is configured for Heuristic TLS Enrichment, its Deep Packet Inspection (DPI) engine scans incoming TCP payloads on port 443. The EPG searches for the TLS ClientHello record header. It calculates the extension array offsets, dynamically shifts the remaining TLS bytes downstream, and injects Extension 17516 contains 4 bytes of TLS headers and 16 bytes of data). EPG platforms enforce strict MTU boundaries. If a subscriber's initial ClientHello packet is already 1,445 bytes, adding the 20-byte extension pushes it to 1,465 bytes. The EPG's internal network stack will instantly transmit the first 1,460 bytes and create a second TCP fragment for the remaining 5 bytes." Is there an irule extension that can detect and stop this fragmentation of TCP/TLS Data on the Client side, to ensure the TLS CLIENTSSL_HELLO is never fragmented.F5 BIG-IP Multi-Site Dashboard
Code is community submitted, community supported, and recognized as ‘Use At Your Own Risk’. A comprehensive real-time monitoring dashboard for F5 BIG-IP Application Delivery Controllers featuring multi-site support, DNS hostname resolution, member state tracking, and advanced filtering capabilities. A 170KB modular JavaScript application runs entirely in your browser, served directly from the F5's high-speed operational dataplane. One or more sites operate as Dashboard Front-Ends serving the dashboard interface (HTML, JavaScript, CSS) via iFiles, while other sites operate as API Hosts providing pool data through optimized JSON-based dashboard API calls. This provides unified visibility across multiple sites from a single interface without requiring even a read-only account on any of the BIG-IPs, allowing you to switch between locations and see consistent pool, member, and health status data with almost no latency and very little overhead. Think of it as an extension of the F5 GUI: near real-time state tracking, DNS hostname resolution (if configured), advanced search/filtering, and the ability to see exactly what changed and when. It gives application teams and operations teams direct visibility into application pool state without needing to wait for answers from F5 engineers, eliminating the organizational bottleneck that slows down troubleshooting when every minute counts. https://github.com/hauptem/F5-Multisite-DashboardBIG-IP Report
Problem this snippet solves: Overview This is a script which will generate a report of the BIG-IP LTM configuration on all your load balancers making it easy to find information and get a comprehensive overview of virtual servers and pools connected to them. This information is used to relay information to NOC and developers to give them insight in where things are located and to be able to plan patching and deploys. I also use it myself as a quick way get information or gather data used as a foundation for RFC's, ie get a list of all external virtual servers without compression profiles. The script has been running on 13 pairs of load balancers, indexing over 1200 virtual servers for several years now and the report is widely used across the company and by many companies and governments across the world. It's easy to setup and use and only requires auditor (read-only) permissions on your devices. Demo/Preview Interactive demo http://loadbalancing.se/bigipreportdemo/ Screen shots The main report: The device overview: Certificate details: How to use this snippet: Installation instructions BigipReport REST This is the only branch we're updating since middle of 2020 and it supports 12.x and upwards. Downloads: https://github.com/net-utilities/BigIPReport/releases Documentation, installation instructions and troubleshooting: https://net-utilities.github.io/BigIPReport/ BIG-IP Report (Legacy) Older version of the report that only runs on Windows and is depending on a Powershell plugin originally written by Joe Pruitt (F5). The documentation for this will stay on loadbalancing.se. BIG-IP Report (only download this if you have v10 devices): https://github.com/net-utilities/BigIPReport/releases/download/v5.8.0/bigipreport-5.4.0-beta.zip iControl Snapin https://github.com/net-utilities/BigIPReport/releases/download/v5.8.0/f5-icontrol.zip Got issues/problems/feedback? Still have issues? Drop a comment below. We usually reply quite fast. Any bugs found, issues detected or ideas contributed makes the report better for everyone, so it's always appreciated. --- Join us on Discord: https://discord.gg/7JJvPMYahA Tested this on versions: 12, 13, 14, 15, 16, 17 (probably works on later versions too)
Unlocking the power of AI with Model Context Protocol (MCP): Key features in F5 BIG-IP v21 & v21.1
Model Context Protocol (MCP), now supported in F5 BIG-IP v21, is a groundbreaking standard that revolutionizes AI systems by enabling seamless, dynamic discovery and integration of contextual data across tools, databases, and MCP servers. F5 enhances MCP workflows with optimized load balancing, dynamic traffic management, and secure integration capabilities, ensuring scalable and reliable AI-driven operations across hybrid and multicloud environments. With features like intelligent routing, adaptive context discovery, and agentic AI support, F5 BIG-IP v21 empowers organizations to confidently scale AI solutions while maintaining high performance and security.
