Unlocking the power of AI with Model Context Protocol (MCP): Key features in F5 BIG-IP v21 & v21.1
Model Context Protocol (MCP), now supported in F5 BIG-IP v21, is a groundbreaking standard that revolutionizes AI systems by enabling seamless, dynamic discovery and integration of contextual data across tools, databases, and MCP servers. F5 enhances MCP workflows with optimized load balancing, dynamic traffic management, and secure integration capabilities, ensuring scalable and reliable AI-driven operations across hybrid and multicloud environments. With features like intelligent routing, adaptive context discovery, and agentic AI support, F5 BIG-IP v21 empowers organizations to confidently scale AI solutions while maintaining high performance and security.
SSL Orchestrator Advanced Use Cases: Client Certificate Constrained Delegation (C3D) Support
Introduction F5 BIG-IP is synonymous with "flexibility". You likely have few other devices in your architecture that provide the breadth of capabilities that come native with the BIG-IP platform. And for each and every BIG-IP product module, the opportunities to expand functionality are almost limitless. In this article series we examine the flexibility options of the F5 SSL Orchestrator in a set of "advanced" use cases. If you haven't noticed, the world has been steadily moving toward encrypted communications. Everything from web, email, voice, video, chat, and IoT is now wrapped in TLS, and that's a good thing. The problem is, malware - that thing that creates havoc in your organization, that exfiltrates personnel records to the Dark Web - isn't stopped by encryption. TLS 1.3 and multi-factor authentication don't eradicate malware. The only reasonable way to defend against it is to catch it in the act, and an entire industry of security products are designed for just this task. But ironically, encryption makes this hard. You can't protect against what you can't see. F5 SSL Orchestrator simplifies traffic decryption and malware inspection, and dynamically orchestrates traffic to your security stack. But it does much more than that. SSL Orchestrator is built on top of F5's BIG-IP platform, and as stated earlier, is abound with flexibility. SSL Orchestrator Use Case: Client Certificate Constrained Delegation (C3D) Using certificates to authenticate is one of the oldest and most reliable forms of authentication. While not every application supports modern federated access or multi-factor schemes, you'll be hard-pressed to find something that doesn't minimally support authentication over TLS with certificates. And coupled with hardware tokens like smart cards, certificates can enable one of the most secure multi-factor methods available. But certificate-based authentication has always presented a unique challenge to security architectures. Certificate "mutual TLS" authentication requires an end-to-end TLS handshake. When a server indicates a requirement for the client to submit its certificate, the client must send both its certificate, and a digitally-signed hash value. This hash value is signed (i.e. encrypted) with the client's private key. Should a device between the client and server attempt to decrypt and re-encrypt, it would be unable to satisfy the server's authentication request by virtue of not having access to the client's private key (to create the signed hash). This makes encrypted malware inspection complicated, often requiring a total bypass of inspection to sites that require mutual TLS authentication. Fortunately, F5 has an elegant solution to this challenge in the form of Client Certificate Constrained Delegation, affectionally referred to as "C3D". The concept is reasonably straightforward. In very much the same way that SSL forward proxy re-issues a remote server certificate to local clients, C3D can re-issue a remote client certificate to local servers. A local server can continue to enforce secure mutual TLS authentication, while allowing the BIG-IP to explicitly decrypt and re-encrypt in the middle. This presents an immediate advantage in basic load balancing, where access to the unencrypted data allows the BIG-IP greater control over persistence. In the absence of this, persistence would typically be limited to IP address affinity. But of course, access to the unencrypted data also allows the content to be scanned for malicious malware. C3D actually takes this concept of certificate re-signing to a higher level though. The "constrained delegation" portion of the name implies a characteristic much like Kerberos constrained delegation, where (arbitrary) attributes can be inserted into the re-signed token, like the PAC attributes in a Kerberos ticket, to inform the server about the client. Servers for their part can then simply filter on client certificates issued by the BIG-IP (to prevent direct access), and consume any additional attributes in the certificate to understand how better to handle the client. With C3D you can maintain strong mutual TLS authentication all the way through to your servers, while allowing the BIG-IP to more effectively manage availability. And combined with SSL Orchestrator, C3D can enable decryption and inspection of content for malware inspection. This article describes how to configure SSL Orchestrator to enable C3D for inbound decrypted inspection. Arguably, most of what follows is the C3D configuration itself, as the integration with SSL Orchestrator is pretty simple. Note that Client Certificate Constrained Delegation (C3D) is included with Local Traffic Manager (LTM) 13.0 and beyond, but for integration with SSL Orchestrator you should be running 14.1 or later.To learn more about C3D, please see the following resources: K14065425: Configuring Client Certificate Constrained Delegation (C3D): https://support.f5.com/csp/article/K14065425 Manual Chapter: SSL Traffic Management: https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-system-ssl-administration/ssl-traffic-management.html#GUID-B4D2529E-D1B0-4FE2-8C7F-C3774ADE1ED2 SSL::c3d iRule reference - not required to use C3D, but adds powerful functionality https://clouddocs.f5.com/api/irules/SSL__c3d.html The integration of C3D with SSL Orchestrator involves effectively replacing the client and server SSL profiles that the SSL Orchestrator topology creates, with C3D SSL profiles. This is done programmatically with an iRule, so no "non-strict" customization is required at the topology. Also note that an inbound (reverse proxy) SSL Orchestrator topology will take the form of a "gateway mode" deployment (a routed path to multiple applications), or "application mode" deployment (a single application instance hosted at the BIG-IP). See section 2.5 of the SSL Orchestrator deployment guide for a deeper examination of gateway and application modes: https://clouddocs.f5.com/sslo-deployment-guide/ The C3D integration is only applicable to application mode deployments. Configuration C3D itself involves the creation of client and server SSL profiles: Create a new Client SSL profile: Configuration Certificate Key Chain: public-facing server certificate and private key. This will be the certificate and key presented to the client on inbound request. It will likely be the same certificate and key defined in the SSL Orchestrator inbound topology. Client Authentication Client Certificate: require Trusted Certificate Authorities: bundle that can validate client certificate. This is a certificate bundle used to verify the client's certificate, and will contain all of the client certificate issuer CAs. Advertised Certificate Authorities: optional CA hints bundle. Not expressly required, but this certificate bundle is forwarded to the client during the TLS handshake to "hint" at the correct certificate, based on issuer. Client Certificate Constrained Delegation Client Certificate Constrained Delegation: Enabled Client Fallback Certificate (new in 15.1): option to select a default client certificate if client does not send one. This option was introduced in 15.1 and provides the means to select an alternate (local) certificate if the client does not present one. The primary use case here might be to select a "template" certificate, and use an iRule function to insert arbitrary attributes. OCSP: optional client certificate revocation control. This option defines an OCSP revocation provider for the client certificate. Unknown OCSP Response Control (new in 15.1): determines what happens when OCSP returns Unknown. If an OCSP revocation provider is selected, this option defines what to do if the response to the OCSP query is "unknown". Create a new Server SSL profile: Configuration Certificate: default.crt. The certificate and key here are used as "templates" for the re-signed client certificate. Key: default.key Client Certificate Configuration Delegation Client Certificate Constrained Delegation: Enabled CA Certificate: local forging CA cert. This is the CA certificate used to re-sign the client certificate. This CA must be trusted by the local servers. CA Key: local forging CA key CA Passphrase: optional CA passphrase Certificate Extensions: extensions from the real client cert to be included in the forged cert. This is the list of certificate extensions to be copied from the original certificate to the re-issued certificate. Custom Extension: additional extensions to copy to forged cert from real cert (OID). This option allows you to insert additional extensions to be copied, as OID values. Additional considerations: Under normal conditions, the F5 and backend server attempt to resume existing SSL sessions, whereby the server doesn’t send a Certificate Request message. The effect is that all connections to the backend server use the same forged client cert. There are two ways to get around this: Set a zero-length cache value in the server SSL profile, or Set server authentication frequency to ‘always’ in the server SSL profile CA certificate considerations: A valid signing CA certificate should possess the following attributes. While it can work in some limited scenarios, a self-signed server certificate is generally not an adequate option for the signing CA. keyUsage: certificate extension containing "keyCertSign" and "digitalSignature" attributes basicConstraints: certificate extension containing "CA = true" (for Yes), marked as "Critical" With the client and server SSL profiles built, the C3D configuration is basically done. To integrate with an inbound SSL Orchestrator topology, create a simple iRule and add it to the topology's Interception Rule configuration. Modify the SSL profile paths below to reflect the profiles you created earlier. ### Modify the SSL profile paths below to match real C3D SSL profiles when CLIENT_ACCEPTED priority 250 { ## set clientssl set cmd1 "SSL::profile /Common/c3d-clientssl" ; eval $cmd1 } when SERVER_CONNECTED priority 250 { ## set serverssl SSL::profile "/Common/c3d-serverssl" } In the SSL Orchestrator UI, either from topology workflow, or directly from the corresponding Interception Rule configuration, add the above iRule and deploy. The above iRule programmatically overrides the SSL profiles applied to the Interception Rule (virtual server), effectively enabling C3D support. At this point, the virtual server will request a client certificate, perform revocation checks if defined, and then mint a local copy of the client certificate to pass to the backend server. Optionally, you can insert additional certificate attributes via the server SSL profile configuration, or more dynamically through additional iRule logic: ### Added in 15.1 - allows you to send a forged cert to the server ### irrespective of the client side authentication (ex. APM SSO), ### and insert arbitrary values when SERVERSSL_SERVERCERT { ### The following options allow you to override/replace a submitted ### client cert. For example, a minted client certificate can be sent ### to the server irrespective of the client side authentication method. ### This certificate "template" could be defined locally in the iRule ### (Base64-encoded), pulled from an iFile, or some other certificate source. # set cert1 [b64decode "LS0tLS1a67f7e226f..."] # set cert1 [ifile get template-cert] ### In order to use a template cert, it must first be converted to DER format # SSL::c3d cert [X509::pem2der $cert1] ### Insert arbitrary attributes (OID:value) SSL::c3d extension 1.3.6.1.4.1.3375.3.1 "TEST" } If you've configured the above, a server behind SSL Orchestrator that requires mutual TLS authentication can receive minted client certificates from external users, and SSL Orchestrator can explicitly decrypt and pass traffic to the set of malware inspection tools. You can look at the certificate sent to the server by injecting a tcpdump packet between the BIG-IP and server, then open in Wireshark. tcpdump -lnni [VLAN] -Xs0 -w capture.pcap [additional filters] Finally, you might be asking what to do with certificate attributes injected by C3D, and really it depends on what the server can support. The below is a basic example in an Apache config file to block a client certificate that doesn't contain your defined attribute. <Directory /> SSLRequire "HTTP/%h" in PeerExtList("1.3.6.1.4.1.3375.3.1") RewriteEngine on RewriteCond %{SSL::SSL_CLIENT_VERIFY} !=SUCCESS RewriteRule .? - [F] ErrorDocument 403 "Delegation to SPN HTTP/%h failed. Please pass a valid client certificate" </Directory> And there you have it. In just a few steps you've configured your SSL Orchestrator to integrate with Client Certificate Constrained Delegation to support mutual TLS authentication, and along the way you have hopefully recognized the immense flexibility at your command. Updates As of F5 BIG-IP 16.1.3, there are some new C3D capabilities: C3D has been updated to encode and return the commonName (CN) found in the client certificate subject field in printableString format if possible, otherwise the value will be encoded as UTF8. C3D has been updated to support inserting a subject commonName (CN) via 'SSL::c3d subject commonName' command: when CLIENTSSL_HANDSHAKE { if {[SSL::cert count] > 0} { SSL::c3d subject commonName [X509::subject [SSL::cert 0] commonName] } } C3D has been updated to support inserting a Subject Alternative Name (SAN) and Authority Info Access (AIA) via 'SSL::c3d extension' commands: when CLIENTSSL_HANDSHAKE { ## Insert Subject Alternative Name (SAN) value SSL::c3d extension SAN "DNS:*.test-client.com, IP:1.1.1.1" ### Insert Authority Info Access (AIA) value SSL::c3d extension AIA "ocsp,https://ocsp.entrust.net.com; caIssuer, https://aia.entrust.net/l1m-chain256.cer" } C3D has been updated to add the Authority Key Identifier (AKI) extension to the client certificate if the CA certificate has a Subject Key Identifier (SKI) extension. Another interesting use case is copying the real client certificate Subject Key Identifier (SKI) to the minted client certificate. By default, the minted client certificate will not contain an SKI value, but it's easy to configure C3D to copy the origin cert's SKI by modifying the C3D server SSL profile. In the "Custom extension" field of the C3D section, add 2.5.29.14 as an available extension. __________________________________________ As of F5 BIG-IP 17.1.0 (SSL Orchestrator 11.0), C3D has been integrated natively. Now, for a deployed Inbound topology, the C3D SSL profiles are listed in the Protocol Settings section of the Interception Rules tab. You can replace the client and server SSL profiles created by SSL Orchestrator, with C3D SSL profiles in the Interception Rules tab to support C3D. The C3D support is now extended to both Gateway and Application modes. __________________________________________ As of F5 BIG-IP 21.1, there are some new C3D capabilities: C3D adds a new option in the server SSL profile to enable overriding the certificate start date. This overrides the notBefore (valid-from) attribute of forged certificates to dynamically set the certificate's start time to align with the exact time of certificate forging. C3D adds a new option in the server SSL profile to enable caching of forged certificates. This reduces the need to repeatedly generate them for similar transactions, thereby improving efficiency during high-traffic scenarios, and reducing computational overhead in TLS proxy flows. C3D adds a new iRule command to keep or override the notBefore date in the forged certificate. when CLIENTSSL_HANDSHAKE { ## Update forged certificate lifespan SSL::c3d cert_lifespan [1-8760] } C3D adds a new iRule command to override the forged certificate lifespan. when CLIENTSSL_HANDSHAKE { ## Override the forged certificate notBefore date SSL::c3d cert_start_date [ override | original ] } C3D adds a new iRule command to inject X.509 keyUsage attributes into the forged certificate. when CLIENTSSL_HANDSHAKE { ## Injects KeyUsage constraints into the forged certificate SSL::c3d extension KU "digitalSignature, keyEncipherment" } C3D adds a new iRule command to inject X.509 extendedKeyUsage attributes into the forged certificate. when CLIENTSSL_HANDSHAKE { ## Injects extendedKeyUsage constraints into the forged certificate SSL::c3d extension EKU "serverAuth, clientAuth" } C3D adds a new iRule command to inject X.509 subjectDirectoryAttributes into the forged certificate. when CLIENTSSL_HANDSHAKE { ## Injects subjectDirectoryAttributes constraints into the forged certificate SSL::c3d extension SDA "dateOfBirth:20000101000000Z, countryOfCitizenship:US" } Note that certificate extensions enabled in the C3D server SSL profile configuration are copy operations. When enabled they copy the selected attributes from the origin certificate. The iRule commands are write operations. They allow you to specify new attribute values.Enhancing AI Data Pipelines with BIG-IP v21: Discover S3 Integration
F5 BIG-IP v21 revolutionizes AI data pipelines with advanced support for S3-compatible object storage, enabling enterprises to optimize, secure, and scale AI and analytics workflows seamlessly. By introducing S3-tuned traffic profiles, intelligent load balancing, and robust health monitoring, BIG-IP ensures predictable performance, resiliency, and protection against protocol-specific threats. This transformative delivery layer empowers businesses to handle complex workloads efficiently, making AI-driven innovation faster, smoother, and more reliable than ever.
GTM Pool Members Gone After Maintenance? It's Probably This One Setting
You finish a maintenance window, everything looks good on LTM, and then someone notices Wide IPs are resolving to fewer destinations than before. You check the GTM pools and the members are just... gone. The virtual servers are fine on LTM. GTM just doesn't know about them anymore — and more importantly, it doesn't remember if they were ever pool members. This happens more often than it should, and it almost always comes back to the same thing: virtual-server-discovery enabled doing exactly what it was designed to do, at exactly the wrong moment. What's Actually Going On When virtual-server-discovery is set to enabled on a GTM server object, GTM keeps its view of LTM virtual servers in sync via iQuery. It automatically adds new virtual servers, updates existing ones, and — this is the part that causes problems — deletes virtual servers that LTM stops reporting on. That delete behavior is the issue. Any time iQuery reports zero virtual servers, even temporarily, GTM treats it as a mass deletion event. The virtual servers get pulled from the server object, and with them, their pool memberships. When LTM eventually reports on those virtual servers again, GTM re-discovers them as brand new objects with no memory of which pools they belonged to. Two scenarios trigger this consistently. Scenario 1: LTM Software Upgrade This is the one that catches most people. During an upgrade, LTM reboots and goes through a phase where iQuery can connect but the full configuration hasn't finished loading yet. From GTM's perspective, LTM is reachable but reporting no virtual servers. GTM interprets that as a deletion event, clears out the discovered virtual servers, and empties the pools. When LTM finishes loading and the virtual servers come back, GTM re-discovers them — but the pool memberships are gone. You're left manually rebuilding what was there before the maintenance window started. The telltale sign is pool members coming back in blue/CHECKING state. That only happens to newly discovered objects. GTM treated a returning virtual server as a brand new one — because as far as it's concerned, it is. The GTM log won't show a deletion event, only the re-add. That gap in the logs is a known blind spot with virtual-server-discovery enabled, and it's exactly why the problem is hard to diagnose after the fact. What you'll typically see in /var/log/gtm after the LTM comes back: alert gtmd[xxxxx]: 011a1005:1: SNMP_TRAP: Pool your_pool state change green --> red (No enabled pool members available) alert gtmd[xxxxx]: 011a3004:1: SNMP_TRAP: Wide IP your.wideip.example.com state change green --> red (No enabled pools available) And then shortly after, the virtual servers re-appear in CHECKING state as GTM re-discovers them — but with no pool bindings. Scenario 2: LTM HA Failover This one surprises people because the LTM pair is still running — it's just switching active units. After a failover, the new active device may not have its iQuery connections fully re-established yet. GTM sees the iQuery state as inconsistent, virtual server status updates stop coming through, and members disappear from the discovered list. What makes this harder to diagnose is that tmsh show gtm iquery may show "connected" — but connected doesn't mean the config sync is working correctly. In a GTM sync group, only the device assigned local ID 0 (the GTM with the lowest IP address) is responsible for writing auto-discovery results to the configuration. If that specific device loses its iQuery connection during the failover window, discovery events are missed entirely — even if every other GTM in the group can still reach the LTM. So you can have a situation where five out of six GTMs look perfectly healthy, iQuery shows connected everywhere, and yet pool members are still disappearing — because the one device that matters for discovery is the one with the broken connection. You can check which device in your sync group holds local ID 0 with: tmsh list sys db gtm.peerinfolocalid If that device's iQuery connection to the LTM is the one that dropped during the failover window, that's your answer — even if everything else looks fine. The Fix: enabled-no-delete Both scenarios share the same root cause: GTM's auto-delete behavior treating a temporary iQuery disruption as a permanent deletion event. The fix is the same for both: gtm server /Common/site1-ltm { addresses { 10.1.1.1 { device-name site1-ltm } } datacenter /Common/dc1 monitor /Common/bigip virtual-server-discovery enabled-no-delete } With enabled-no-delete, GTM still auto-discovers new virtual servers and keeps existing ones updated. The only thing that changes is that it will never delete a virtual server just because LTM temporarily stopped reporting it. Your pool memberships survive both scenarios above. Mode Adds new VS Updates VS Deletes VS Pool memberships survive iQuery disruption? disabled No No No Yes — nothing changes enabled Yes Yes Yes No — any disruption can empty pools enabled-no-delete Yes Yes No Yes — preserved The Trade-Off enabled-no-delete won't clean up after you when you intentionally decommission a virtual server on LTM. The stale GTM object stays in the discovered list until you remove it manually. In environments with a lot of VS churn, this can accumulate over time. The question is which failure mode you'd rather manage: pool members silently disappearing during a maintenance window, or occasionally needing to clean up stale objects after a planned decommission. For most production environments, the latter is far easier to deal with — and far less likely to wake someone up at 2am. How to Make the Change Via tmsh: tmsh modify gtm server /Common/site1-ltm \ virtual-server-discovery enabled-no-delete tmsh save sys config Via GUI: Go to DNS → GSLB → Servers Select the server object Set Virtual Server Discovery to Enabled (No Delete) Click Update This takes effect immediately and does not affect existing discovered virtual servers or current pool memberships. Cleaning Up Stale Objects When you intentionally decommission a virtual server on LTM, remove the leftover GTM object manually: # List virtual servers under a GTM server object tmsh list gtm server /Common/site1-ltm virtual-server # Remove a specific stale entry tmsh modify gtm server /Common/site1-ltm \ virtual-servers delete { /Common/old-vs-name } tmsh save sys config Make this part of your standard VS decommission runbook and stale objects will never pile up. Quick Diagnostic When Members Go Missing Before assuming it's a discovery issue, check iQuery health across all GTM devices first: tmsh show gtm iquery Look for: State: should be connected to all entries Reconnects: A high count suggests instability even if the connection looks up Configuration Time: None means the config has never successfully synced from that LTM Then confirm which GTM holds local ID 0 and verify its connectivity specifically: tmsh list sys db gtm.peerinfolocalid If the local ID 0 device is the one with the broken iQuery connection, that's your answer — regardless of what the other devices are showing. Wrapping Up Whether it's an LTM upgrade or an HA failover, the pattern is the same: iQuery goes quiet for a moment, GTM interprets silence as deletion, and your pool memberships are gone. It's working as designed — just not in a way that's useful to you. enabled-no-delete is a one-line change that stops this from happening. The cleanup overhead it introduces is predictable and manageable. The alternative — rebuilding pool memberships after an unplanned event — is not. Have you run into either of these scenarios in your environment? Drop a comment below, especially if you've seen the local ID 0 shift cause issues during a rolling GTM upgrade.Enforcing a Single Connection Max to Pool Members
I like finding jewels and nuggets of clarity in problems presented to the community at large, whether it’s here on DevCentral or in third party communities like Reddit, where member macallen posed the following problem in r/sysadmin a couple months back (paraphrased here, check the link for full context). Problem Statement I have a pool of five servers, and I need a maximum of one connection per server strictly enforced. When I set the connection limit to 1 at the node level, I’m still seeing a second connection offered when the 6th active request comes in. Any ideas on how I can accomplish this? Diagnosing the Problem First, I’ll mock this up in my lab, only on a smaller scale of two servers rather than five, and setting the connection limit on each server to one. Using curl from two virtual machines, I run curl 192.168.102.50/ several times and notice that I am seeing a max of two per server being enforced, not one as anticipated. The problem here is not that TMM is failing to honor the connection limits. The problem, at least on my test system, is that there are two TMMs present. Each TMM is limiting the servers to a maximum of one connection, so in this case, two connections are allowed instead of the required one. And just like the statistical representation of a family consisting of 2.3 kids, well, there’s no such thing as .3 of a kid, and there’s no such thing as .5 of a connection, so setting that doesn’t make much sense and isn’t allowed anyway. The good news is that for almost all use cases at scale the BIG-IP does the math, taking maximum configured connections and dividing by the number of TMMs. Note that this can lead to unexpected issues if for some reason the disaggregator (DAG) has an uneven connection distribution, and it is generally recommended NOT to have a connection maximum less than the active number of TMM instances. See K8457 for additional details. But now that the problem is known, what do I do about it? Solutions Option #1 - Duct Tape & Chewing Gum! In the Reddit thread, the original poster solved his own problem by, in his words, "I created a duct tape solution. I wrote a service that opens a port. When the user connects, it closes the port, when they disconnect it opens it back up. Then I created a contract in F5 for that port so it disables the node when the port is down. Cheap and dirty, but works." Glad to hear that works, but not a process I’d recommend. If someone else takes over ownership of that application and has no idea why that service exists and thus removes it…outage city! Option #2 - Configure BIG-IP VE for a Single Core I call this the machete mode, where I just whack some compute cycles away to solve the problem. That’s an easy one! Shut down the image, strip it down to a single core, fire it back up, and presto! And if this was the only application in service, that would be fantastic. But that’s not likely, and so punishing the rest of the application delivery needs to meet this need is not a great solution. Option #3 - Pin the Virtual Server to a Single Core with an iRule This option requires no system changes at all, just a simple iRule using a global variable, as they are not CMP compatible and thus will demote any virtual server to a single TMM, effectively pinning it and solving the problem. The iRule could look something like this: when RULE_INIT { set ::global_pin_tmm } This iRule is clean and compact, with no impact to traffic since its only engagement is at initialization. It also has a useful name, indicating it’s a global variable and its purpose is to pin the virtual server to a single TMM. Effective, but it feels a little icky to use an iRule with global variables in any version after 11.4 and one of my biggest messages when I speak at user groups is that “iRules are great! But don’t use them!” I always suggest the use of a configuration option when available, and only when iRules are necessary should they be utilized. Option #4 - Pin the Virtual Server to a Single Core with a TMSH Command That brings me to the final option I’ll explore, and that is to use a TMSH command to pin the virtual server. It’s an option on the virtual server (not available in the GUI) to disable CMP: tmsh modify ltm virtual <virtual name> cmp-enabled no Super simple, crystal clear in the configuration, no Tcl-machine necessary. That sounds like a winner to me and is evident now in a new screen capture. Conclusion With BIG-IP, there are often many ways to approach a problem. Sometimes there are no clear advantages amongst solutions, but this problem has a clear winner and that is the final option presented here: using the tmsh command to disable CMP.SNI Routing with BIG-IP
In the previous article, The Three HTTP Routing Patterns, Lori MacVittie covers 3 methods of routing. Today we will look at Server Name Indication (SNI) routing as an additional method of routing HTTPS or any protocol that uses TLS and SNI. Using SNI we can route traffic to a destination without having to terminate the SSL connection. This enables several benefits including: Reduced number of Public IPs Simplified configuration More intelligent routing of TLS traffic Terminating SSL Connections When you have a SSL certificate and key you can perform the cryptographic actions required to encrypt traffic using TLS. This is what I refer to as “terminating the SSL connection” throughout this article. When you want to route traffic this is a chicken and an egg problem, because for TLS traffic you want to be able to route the traffic by being able to inspect the contents, but this normally requires being able to “terminate the SSL connection”. The goal of this article is to layer in traffic routing for TLS traffic without having to require having/knowing the original SSL certificate and key. Server Name Indication (SNI) SNI is a TLS extension that makes it possible to "share" certificates on a single IP address. This is possible due to a client using a TLS extension that requests a specific name before the server responds with a SSL certificate. Prior to SNI, the other options would be a wildcard certificate or Subject Alternative Name (SAN) that allows you to specify multiple names with a single certificate. SNI with Virtual Servers It has been possible to use SNI on F5 BIG-IP since TMOS 11.3.0. The following KB13452 outlines how it can be configured. In this scenario (from the KB article) the BIG-IP is terminating the SSL connection. Not all clients support SNI and you will always need to specify a “fallback” profile that will be used if a SNI name is not used or matched. The next example will look at how to use SNI without terminating the SSL connection. SNI Routing Occasionally you may have the need to have a hybrid configuration of terminating SSL connections on the BIG-IP and sending connections without terminating SSL. One method is to create two separate virtual servers, one for SSL connections that the BIG-IP will handle (using clientssl profile) and one that it will not handle SSL (just TCP). This works OK for a small number of backends, but does not scale well if you have many backends (run out of Public IP addresses). Using SNI Routing we can handle everything on a single virtual server / Public IP address. There are 3 methods for performing SNI Routing with BIG-IP iRule with binary scan a. Article by Colin Walker code attribute to Joel Moses b. Code Share by Stanislas Piron iRule with SSL::extensions Local Traffic Policy Option #1 is for folks that prefer complete control of the TLS protocol. It only requires the use of a TCP profile. Options #2 and #3 only require the use of a SSL persistence profile and TCP profile. SNI Routing with Local Traffic Policy We will skip option #1 and #2 in this article and look at using a Local Traffic Policy for SNI Routing. For a review of Local Traffic Policies check out the following DevCentral articles: LTM Policy Jan 2015 Simplifying Local Traffic Policies in BIG-IP 12.1 June 2016 In previous articles about Local Traffic Policies the focus was on routing HTTP traffic, but today we will use it to route SSL connections using SNI. In the following example, using a Local Traffic Policy named “sni_routing”, we are setting a condition on the SSL Extension “servername” and sending the traffic to a pool without terminating the SSL connection. The pool member could be another server or another BIG-IP device. The next example will forward the traffic to another virtual server that is configured with a clientssl profile. This uses VIP targeting to send traffic to another virtual server on the same device. In both examples it is important to note that the “condition”/“action” has been changed from occurring on “request” (that maps to a HTTP L7 request) to “ssl client hello”. By performing the action prior to any L7 functions occurring, we can forward the traffic without terminating the SSL connection. The previous example policy, “sni_routing”, can be attached to a Virtual Server that only has a TCP profile and SSL persistence profile. No HTTP or clientssl profile is required! This method can also be used to solve the issue of how to consolidate multiple SSL virtual servers behind a single virtual server that have different APM and/or ASM policies. This is similar to the architecture that is used by the Container Connector for Cloud Foundry; in creating a two-tier load balancing solution on a single device. Routed Correctly? TLS 1.3 has interesting proposals on how to obscure the servername (TLS in TLS?), but for now this is a useful and practical method of handling multiple SSL certs on a single IP. In the future this may still be possible as well with TLS 1.3. For example the use of a HTTP Fronting service could be a tier 1 virtual server (this is just my personal speculation, I have not tried, at the time of publishing this was still a draft proposal). In other news it has been demonstrated that a combination of using SNI and a different host header can be used for “domain fronting”. A method to enforce consistent policy (prevent domain fronting) would be to layer in additional conditions that match requested SNI servername (TLS extension) with requested HOST header (L7 HTTP header). This would help enforce that a tenant is using a certificate that is associated with their application and not “borrowing” the name and certificate that is being used by an adjacent service. We don’t think of a TLS extension as an attribute that can be used to route application traffic, but it is useful and possible on BIG-IP.BIG-IP LTM VE: Transfer your iRules in style with the iRule Editor
The new LTM VE has opened up the possibilities for writing, testing and deploying iRules in a big way. It’s easier than ever to get a test environment set up in which you can break things develop to your heart’s content. This is fantastic news for us iRulers that want to be doing the newest, coolest stuff without having to worry about breaking a production system. That’s all well and good, but what the heck do you do to get all of your current stuff onto your test system? There are several options, ranging from copy and paste (shudder) to actual config copies and the like, which all work fine. Assuming all you’re looking for though is to transfer over your iRules, like me, the easiest way I’ve found is to use the iRule editor’s export and import features. It makes it literally a few clicks and super easy to get back up and running in the new environment. First, log into your existing LTM system with your iRule editor (you are using the editor, right? Of course you are…just making sure). You’ll see a screen something like this (right) with a list of a bagillionty iRules on the left and their cool, color coded awesomeness on the right. You can go through and select iRules and start moving them manually, but there’s really no need. All you need to do is go up to the File –> Archive –> Export option and let it do its magic. All it’s doing is saving text files to your local system to archive off all of your iRuley goodness. Once that’s done, you can then spin up your new LTM VE and get logged in via the iRule editor over there. Connect via the iRule editor, and go to File –> Archive –> Import, shown below. Once you choose the import option you’ll start seeing your iRules popping up in the left-hand column, just like you’re used to. This will take a minute depending on how many iRules you have archived (okay, so I may have more than a few iRules in my collection…) but it’s generally pretty snappy. One important thing to note at his point, however, is that all of your iRules are bolded with an asterisk next to them. This means they are not saved in their current state on the LTM. If you exit at this point, you’ll still be iRuleless, and no one wants that. Luckily Joe thought of that when building the iRule editor, so all you need to do is select File –> Save All, and you’ll be most of the way home. I say most of the way because there will undoubtedly be some errors that you’ll need to clean up. These will be config based errors, like pools that used to exist on your old system and don’t now, etc. You can either go create the pools in the config or comment out those lines. I tend to try and keep my iRules as config agnostic as possible while testing things, so there aren’t a ton of these but some of them always crop up. The editor makes these easy to spot and fix though. The name of the iRule that’s having a problem will stay bolded and any errors in that particular code will be called out (assuming you have that feature turned on) so you can pretty quickly spot them and fix them. This entire process took me about 15 minutes, including cleaning up the code in certain iRules to at least save properly on the new system, and I have a bunch of iRules, so that’s a pretty generous estimate. It really is quick, easy and painless to get your code onto an LTM VE and get hacking coding. An added side benefit, but a cool one, is that you now have your iRules backed up locally. Not only does this mean you’re double plus sure that they won’t be lost, but it means the next time you want to deploy them somewhere, all you have to do is import from the editor. So if you haven’t yet, go download your BIG-IP LTM VE and get started. I can’t recommend it enough. Also make sure to check out some of the really handy DC content that shows you how to tweak it for more interfaces or Joe’s supremely helpful guide on how to use a single VM to run an entire client/LTM/server setup. Wicked cool stuff. Happy iRuling. #Colin
Virtual-wire Configuration and Troubleshooting
A virtual wire(vWire) logically connects two interfaces or trunks, in any combination, to each other, enabling the BIG-IP system to forward traffic from one interface to the other, in either direction. This type of configuration is typically used for security monitoring, where the BIG-IP system inspects ingress packets without modifying them in any way. To deploy a BIG-IP system without making changes to other devices on your network, you can configure the system to operate strictly at Layer 2. By deploying a virtual wire configuration, you transparently add the device to the network without having to create self IP addresses or change the configuration of other network devices that the BIG-IP device is connected to. Topology Before vWire Deployment After vWire Deployment Few points about virtual wire configurations in general: vWire works in transparent mode,which means there is no packet modification The system bridges both tagged and untagged packets Neither VLANs nor MAC addresses change in Symmetry mode propagate virtual wire link: When enabled, the BIG-IP system changes the peer port state to down when the corresponding interface is disabled or down. If disabled, the BIG-IPsystem does not change the peer port state Configuring vWire in UI BIG-IP Navigate to Network>>Virtual Wire Select Create (upper right) Enter the values for interfaces added to the virtual wire Enter VLAN information and click on Add for every VLAN object created Recommended- Enable propagate virtual wire link status for detecting link failure Once all the selections are made and you are ready to implement, click on "Commit Changes to System": The resulting screen will look like the following: The resulting VLAN configuration will look as follows: Note: Be sure to configure an untagged VLAN on the relevant virtual wire interface to enable the system to correctly handle untagged traffic. Note that many Layer 2 protocols, such as Spanning Tree Protocol (STP), employ untagged traffic in the form of BPDUs. Configuring vWire in cli mode Configure interfaces to support virtual wire: tmsh modify net interface 1.1 port-fwd-mode virtual-wire tmsh modify net interface 1.2 port-fwd-mode virtual-wire Create all VLAN tag VLAN objects: tmsh create net vlan Direct_all_vlan_4096_1 tag 4096 interfaces add { 1.1 { tagged } } tmsh create net vlan Direct_all_vlan_4096_2 tag 4096 interfaces add { 1.2 { tagged } } Create specific (802.1Q tag 512) VLAN objects: tmsh create net vlan Direct_vlan_512_1 tag 512 interfaces add { 1.1 { tagged } } tmsh create net vlan Direct_vlan_512_2 tag 512 interfaces add { 1.2 { tagged } } Create VLAN Groups: tmsh create net vlan-group Direct_all_vlan members add { Direct_all_vlan_4096_1 Direct_all_vlan_4096_2 } mode virtual-wire tmsh create net vlan-group Direct_vlan_512 members add { Direct_vlan_512_1 Direct_vlan_512_2 } mode virtual-wire config save: tmsh save sys config partitions all vWire config with trunk with LACP and LACP Pass Through LACP Pass through feature tunnels LACP packets through trunks between switches. Configure an untagged VLAN on the virtual wire interface to tunnel LACP packets. Note: Propagate virtual wire link status should be enabled for LACP pass through mode.LACP Pass through and Propagate virtual wire link status is supported from 16.1.x Configuring LACP Pass Through Configuring interface to support in vwire mode: tmsh modify net interface 1.1 port-fwd-mode virtual-wire tmsh modify net interface 2.1 port-fwd-mode virtual-wire tmsh modify net interface 1.2 port-fwd-mode virtual-wire tmsh modify net interface 2.2 port-fwd-mode virtual-wire Configure trunk : tmsh create net trunk left_trunk_1 interfaces add { 1.1 2.1 } qinq-ethertype 0x8100 link-select-policy auto tmsh create net trunk right_trunk_1 interfaces add { 1.2 2.2 } qinq-ethertype 0x8100 link-select-policy auto Configure VLAN tagged and untagged interface: tmsh create net vlan left_vlan_1_4k tag 4096 interfaces add {left_trunk_1 {tagged}} tmsh create net vlan left_vlan_1 tag 31 interfaces add {left_trunk_1 {tagged}} tmsh create net vlan left_vlan_333 tag 333 interfaces add {left_trunk_1 {untagged}} tmsh create net vlan right_vlan_1_4k tag 4096 interfaces add {right_trunk_1 {tagged}} tmsh create net vlan right_vlan_1 tag 31 interfaces add {right_trunk_1 {tagged}} tmsh create net vlan right_vlan_333 tag 333 interfaces add {right_trunk_1 {untagged}} Create VLAN Group and enabled propagate-linkstatus: tmsh create net vlan-group vg_1_4k bridge-traffic enabled mode virtual-wire members add { left_vlan_1_4k right_vlan_1_4k } vwire-propagate-linkstatus enabled tmsh create net vlan-group vg_untagged bridge-traffic enabled mode virtual-wire members add { left_vlan_333 right_vlan_333 } vwire-propagate-linkstatus enabled tmsh create net vlan-group vg_1 bridge-traffic enabled mode virtual-wire members add { left_vlan_1 right_vlan_1 } vwire-propagate-linkstatus enabled configuring LACP(Active-Active) mode Configuring interface to support in vwire mode: tmsh modify net interface 1.1 port-fwd-mode virtual-wire tmsh modify net interface 1.2 port-fwd-mode virtual-wire tmsh modify net interface 2.1 port-fwd-mode virtual-wire tmsh modify net interface 2.2 port-fwd-mode virtual-wire Configure trunk in LACP Active mode : tmsh create net trunk left_trunk_1 interfaces add { 1.1 1.2 } qinq-ethertype 0x8100 link-select-policy auto lacp enabled lacp-mode active tmsh create net trunk right_trunk_1 interfaces add { 2.1 2.2 } qinq-ethertype 0x8100 link-select-policy auto lacp enabled lacp-mode active Configure VLAN tagged interface: tmsh create net vlan left_vlan_1_4k tag 4096 interfaces add {left_trunk_1 {tagged}} tmsh create net vlan left_vlan_1 tag 31 interfaces add {left_trunk_1 {tagged}} tmsh create net vlan right_vlan_1_4k tag 4096 interfaces add {right_trunk_1 {tagged}} tmsh create net vlan right_vlan_1 tag 31 interfaces add {right_trunk_1 {tagged}} Create VLAN Group and enabled propagate-linkstatus: tmsh create net vlan-group vg_1_4k bridge-traffic enabled mode virtual-wire members add { left_vlan_1_4k right_vlan_1_4k } vwire-propagate-linkstatus enabled tmsh create net vlan-group vg_1 bridge-traffic enabled mode virtual-wire members add { left_vlan_1 right_vlan_1 } vwire-propagate-linkstatus enabled DB variables for vWire: Trouble shooting vWire : 1 . Verify that traffic flowing through default Virtual Server(_vlangroup) Tcpdump cmd: tcpdump -nne -s0 -i 0.0:nnn 22:00:53.398116 00:00:00:00:01:31 > 33:33:00:00:00:05, ethertype 802.1Q (0x8100), length 139: vlan 31, p 0, ethertype IPv6, fe80::200:ff:fe00:131 > ff02::5: OSPFv3, Hello, length 40 out slot1/tmm9 lis=_vlangroup 22:00:53.481645 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 91: vlan 31, p 0, ethertype IPv4, 10.31.0.3 > 224.0.0.18: VRRPv3, Advertisement, vrid 1, prio 150, intvl 100cs, length 12 out slot1/tmm4 lis=_vlangroup 2. Now create Virtual Server based on requirements like TCP, UDP and ICMP with Virtual Server name as test.Verify traffic is hitting Virtual Server Tcpdump cmd: tcpdump -nne -s0 -i 0.0:nnn 22:04:54.161197 3c:41:0e:9b:01:31 > 00:00:00:00:03:31, ethertype 802.1Q (0x8100), length 145: vlan 31, p 0, ethertype IPv4, 10.20.0.10 > 10.13.0.10: ICMP echo request, id 30442, seq 2, length 64 out slot4/tmm2 lis=/Common/test 22:05:14.126544 3c:41:0e:9b:01:31 > 00:00:00:00:03:31, ethertype 802.1Q (0x8100), length 121: vlan 31, p 0, ethertype IPv4, 10.20.0.10.41692 > 10.13.0.10.80: Flags [S], seq 2716535389, win 64240, options [mss 1460,sackOK,TS val 685348731 ecr 0,nop,wscale 7], length 0 out slot3/tmm8 lis=/Common/test 22:05:14.126945 3c:41:0e:9b:03:31 > 00:00:00:00:01:31, ethertype 802.1Q (0x8100), length 121: vlan 31, p 0, ethertype IPv4, 10.13.0.10.80 > 10.20.0.10.41692: Flags [S.], seq 1173350299, ack 2716535390, win 65160, options [mss 1460,sackOK,TS val 4074187325 ecr 685348731,nop,wscale 7], length 0 in slot3/tmm8 lis=/Common/test 3 . Trouble Shooting steps Get the tcpdump and check the traffic hitting Virtual Server or not If traffic is dropped, enable “tmsh modify sys db vlangroup.forwarding.override value enable” with destination as catch all and check whether traffic is hitting _vlangroup and going out or not. If traffic is going without any issue, then there is an issue with created virtual server. Even after enabling vlangroup.forwarding.override db variable, then take the output of below commands tmctl ifc_stats - displays interface traffic statistics tmctl ip_stat - displays ip traffic statistics tmctl ip6_stat - displays ipv6 traffic statistics vWire Behavior This table describes how the BIG-IP system handles certain conditions when the relevant interfaces are configured to use a virtual wire. The table also shows what actions you can take, if possible Notable Effects-Caveats When deploying a pair of BIG-IP’s in HA mode, the virtual wire configuration will create objects with different names on each BIG-IP. So for example, the creation of vwire_lab01 will result in the creation of VLAN objects vwire_lab01_1_567 and vwire_lab01_2_567 on one BIG-IP, while the other BIG-IP will have vwire_lab01_1_000 and vwire_lab01_2_000 in its configuration. For modules like SSL Orchestrator, or in cases where a Virtual Server needs to be associated with a specific VLAN, the numbering is problematic. The administrator will not be able to associate the topology or Virtual Server to one VLAN object (vwire_lab01_2_567) on the first BIG-IP and the other VLAN object (vwire_lab01_2_000) on the peer BIG-IP. (this is not possible for a number of reasons, one of which is the way configurations are synchronized between BIG-IP devices) Q-in-Q is not supported in a virtual wire configuration Virtual wire feature is not supported on Virtual Clustered Multiprocessing (vCMP) Active/Active deployment is not supported vWire is not supported on virtual edition(VE) Conclusion BIG-IP in Virtual Wire can be deployed in any network without any network design or configuration changes, as it works in L2 transparent mode. L2Transparency Caveats There are few caveats with respect to L2 Transparency OSPF neighborship struck in exstart state. BGP neighborship won’t come with MD5 authentication OSPF neighborship struck in exstart state In transparent mode when standard Virtual server is configured, the VS will process the DBD packet with this the TTL value become zero and the OSPF neighborship will struck at Exstart state. To solve the above problem, we need to configure a profile to preserve the TTL value and attach the profile to the virtual server. Below are the steps to configure the profile and the virtual server. Same steps can be configured for both vwire and vlangroup Create a profile to preserve TTL Click on create and enter the profile name as TTL and select to preserve Now attach the profile under ipother After attaching the profile the OSPF neighborship will come up. BGP neighborship won’t come with MD5 authentication In transparent mode when standard Virtual server is configured, the VS will process the BGP packet and will reply back to the tcp connection without MD5 with BGP wont come up between two devices To solve the above problem, we need to configure a profile to support Md5 authentication and attach the profile to the virtual server. Below are the steps to configure the profile and the virtual server. Same steps can be configured for both Vwire and Vlan-group Create a profile to support md5 authentication Create a profile with name md5. Enable md5 authentication and provide md5 authentication password. Attach the MD5 profile to virtual server After attaching the md5 policy the BGP neighborship will up.
7. SYN Cookie: Troubleshooting Stats
Introduction In this article I will explain what SYN Cookie stats you can consult and their meaning. There are more complex stats than the explained in this article but they are intended for helping F5 engineers when cases become complex. SYN Cookie stats First at all, in order to troubleshoot SYN Cookie you need to know how you can check SYN Cookie stats easily and understand what you are reading. The easiest way to see these stats in a device running LTM module based SYN Cookie is by running below command and focusing in SYN Cookies section of the output: # tmsh show ltm virtual <virtual> SYN Cookies Status full-hardware Hardware SYN Cookie Instances 6 Software SYN Cookie Instances 0 Current SYN Cache 2.0K SYN Cache Overflow 24 Total Software 4.3M Total Software Accepted 0 Total Software Rejected 0 Total Hardware 21.7M Total Hardware Accepted 3 Let’s go through each field to know what they means: Status: [full-software/full-hardware|not-activated]. This value describes what type of SYN Cookie mode has been activated, software or hardware. Once an attack has finished it is normal that LTM takes some time to deactivate SYN Cookie mode after attack traffic stops, see the second article in this series (SYN Cookie Operation). Note that we do not want SYN Cookie entering in an activating/deactivating loop in case TCP SYN packets per second reaching device are near to the configured SYN cache threshold. So delay of 30-60 seconds before SYN Cookie being deactivated is in normal range. Hardware SYN Cookie Instances: How many TMMs are under Hardware SYN Cookie mode. Software SYN Cookie Instances: How many TMMs are under Software SYN Cookie mode. It indicates if software is currently generating SYN cookies. Current SYN Cache: Indicates how many embryonic connections are handled by BIG-IP (refreshed every 2 seconds). SYN cache is always counting embryonic connections, regardless if SYN Cookie is activated or not. So even if SYN Cookies is completely turned off, we still have embryonic flows that have not been promoted to full flows yet (SYNs which has not completed 3WHS), this means that cache counter will still be used regularly, so is normal having a value different than 0. Disabling SYN Cookie will not avoid this counter increase but thresholds will not be taken into account. Since SYN cache is no longer a cache, as explained in prior articles, and the stat is merely a counter of embryonic flows, it does not consume memory resources. As the embryonic flows are promoted or time out, this value will decrease. SYN Cache Overflow (per TMM): It is incremented whenever the SYN cache threshold is exceeded and SYN cookies need to be generated. It increments in one per each TMM instance and this value is a counter that only increases, so we can consult the value to know how many times SYN Cookie has been activated since last stats reset, remember, per TMM. Total Software: Number of challenges generated by Software. This is increased regardless if client sends a response, does not send any response or the response is not correct. Total Software Accepted: Number of TCP handshakes that were correctly handled with clients. Total Software Rejected: Number of wrong responses to challenges. Remember that ALL rejected SYN cookies are in software, there is no hardware rejected. Total Hardware: Number of challenges generated by Hardware. Total Hardware Accepted: Number of TCP handshakes that were correctly handled with clients. In case you have configured AFM based SYN Cookie then you can use two easy sources of information, the already explained LTM command above, but also you can check stats for TCP half open DoS vector, at device or virtual server context, as you would do with any other DoS vector. Let’s check the most important fields at device context. # tmsh show security dos device-config | grep -A 40 half Statistics Type Count Status Ready Attack Detected 1 *Attacked Dst Detected 0 *Bad Actor Attack Detected 0 Aggregate Attack Detected 1 Attack Count 2 *Stats 1h Samples 0 Stats 408 *Stats Rate 408 Stats 1m 104 Stats 1h 0 Drops 1063 *Drops Rate 1063 Drops 1m 187 Drops 1h 4 *Int Drops 0 *Int Drops Rate 0 *Int Drops 1m 0 *Int Drops 1h 0 Status: This field confirm if this specific DoS vector is ready to detect TCP SYN flood attacks. You have to take into account that you could decide to configure this DoS vector with Auto-threshold, in which case AFM would be in charge of deciding the best threshold. Note that by enabling auto-threshold AFM would need some time to learn the traffic pattern of your environment, until it has enough information to create a correct threshold you will not see this field as Ready but as Learning. If vector is configured manually you always see it as Ready. Attack detected: It informs you if currently there is an ongoing attack and detected by AFM. Aggregate attack detected: Since DoS stats shown above are for device context the detected attack is aggregate. Attack Count: Gives information about how many attacks have been detected since stats were reset last time. It does not decrease. Stats: Number of embryonic connections at this current second. Remember that since this is a snapshot, the counter could go increase or decrease. This is different to other DoS vectors where it only increases. Stats 1m: Average of the Stats in the last minute. This is the average number of embryonic connections that AFM has seen when taking sampling every 1s. Stats 1h: Average of the Stats 1m in the last hour. Drops: It counts the number of wrong ACKs received. In other words this is the current snapshot of: Number of SYN Cookies – Number of validated ACKs received Drops 1m: Average of the Drops in the last minute Drops 1h: Average of the Drops 1m in the last hour I have added an asterisk to some fields to point to values that has not real meaning for SYN Cookie DoS vector, or information is not really useful from my perspective, but they are included to have coherence with other DoS attack stats information. Note: Detection logic for this vector is not based on the Stats 1m, as other DoS vectors, instead it is based on the current number of embryonic connections, that is, value seen in Stats counter. Interpreting stats Once you know what each important stat mean I will give some advises when you interpret SYN Cookie stats. You should know some of them already if you have read all articles in this series: Hardware can offload TMM for validation, so you can see a number of software generated SYN Cookies much bigger than software validated SYN Cookie since software generates cookies that are validated by hardware as well. Since it is possible that a software generated SYN Cookie be accepted by hardware and vice versa, hardware generated SYN Cookies be accepted by software, you could think that value for Total Software is the same than the result of adding the Total Software Accepted plus Total Software rejected. But that is NOT true since it can be possible that a SYN Cookie sent by BIG-IP does not have a response (ACK), quite typical during a SYN flood attack indeed. In this case nothing adds up because generated SYN Cookie was not accepted nor rejected. Remember that SYN cookie’s responses discarded by hardware will be rejected by software, so all rejects are in software. This is why there is not counter for Total Hardware rejected. This means that Total Software Rejected can be increase by Total Hardware. When there is an attack vectors definition that match this attack will be increased. So, for example, during a TCP SYN flood attack you will see that Stats increases for TCP half open and TCP SYN flood (maybe others like low TTL,... as well). Also Stats will increase in all contexts (device and virtual server). The first vector in an specific context whose limit is exceeded it will start to mitigate. This is important because in cases where TCP SYN flood DoS vector has a lower threshold than TCP half open DoS vector, you will notice that traffic is dropped but you did not expect this behavior. Check article 5 for more information. Example In this part you will learn what stats changes you should expect to see when a TCP SYN flood attack is detected and SYN Cookie is activated, so it starts to mitigate the attack. Let’s interpret below stats: During attack Status full-hardware Hardware SYN Cookie Instances 6 Software SYN Cookie Instances 0 Current SYN Cache 1 SYN Cache Overflow 24 Total Software 10.1K Total Software Accepted 0 Total Software Rejected 0 Total Hardware 165.1M Total Hardware Accepted 3 After attack Status not-activated Hardware SYN Cookie Instances 0 Software SYN Cookie Instances 0 Current SYN Cache 15 SYN Cache Overflow 24 Total Software 10.1K Total Software Accepted 0 Total Software Rejected 0 Total Hardware 171.0M Total Hardware Accepted 3 Before the attack, before SYN Cookie is activated, you will see Current SYN Cache stats starts to increase quickly since TCP SYN packets are causing an increment of embryonic connections. Once SYN Cache threshold has been reached in a TMM then SYN Cache Overflow will increase attending to the number of TMMs that detected the attack. In above example you see 24 because this is a 6 CPUs device and 4 TCP SYN Flood attacks were detected by SYN Cookie. Remember this counter will never decrease unless we reset stats. During attack SYN cookie is activated so Current SYN Cache will start to decrease until reaching 0 because SYN Cookie Agent starts to handle TCP 3WHS, in other words, TCP stack stops to receive TCP SYN packets. We can check how many TMMs have SYN cookie activated currently looking at Hardware/Software SYN Cookie Instances counter. Note this match with what I explained for SYN Cache Overflow value. Legitimate connections are counted under Total Hardware/Software Accepted. In this case, we can see that although several millions of TCP SYN packets reached the device only 3 TCP 3WHS were correctly carried out. As you can see this device has Hardware SYN Cookie configured and working, but you can read that software also generates SYN Cookie challenges (Total Software). As commented in article number six of these series, this is expected and can be due to collisions or due to validations of first challenges when SYN Cookie is activated and TMMs handles TCP SYN packets until it enables hardware to do it. This does not affect device performance. In order to change from ‘Status full-hardware’ to ‘Status not-activated’ both HSBs must exit from SYN Cookie mode. Conclusion Now you know how to interpret stats, so you know can deduce information about the past and the present status of your device related to SYN Cookie. In next two article I will end up this series and this troubleshooting part talking about traffic captures and meaning of logs.
