ADC03: Incomplete Observability – A Critical Application Delivery Challenge
Observability is the backbone of modern application delivery, enabling the detection of performance issues, analyzing system usage, and monitoring overall health. However, Incomplete Observability, characterized by insufficient logging, inadequate monitoring tools, and inconsistent data collection, introduces significant business risks. These risks range from limited visibility into performance bottlenecks and prolonged service disruptions to flawed scaling decisions and inefficient operations. To address these challenges effectively, it is crucial to understand the core issues at hand and implement robust strategies and tools, such as F5 BIG-IP and OpenTelemetry, that enhance observability across the infrastructure. Let's explore the impacts of Incomplete Observability and practical solutions, incorporating lessons from a real-world use case. AI Reference Architecture In an AI-powered application ecosystem, observability plays a pivotal role in coordinating and monitoring interactions between end users, frontend applications, and inference services. The following AI Reference Architecture diagram illustrates a typical flow: Diagram Overview: End Users initiate requests through frontend applications. These applications connect with backend Large Language Models (LLMs) based on user-specific needs. Inference Services operate at the core, processing data to deliver accurate and efficient results. Monitoring critical pipelines ensures reliability and scalability while maintaining secure data flows. By aligning observability with such an architecture, AI systems can handle complex pipelines effectively, optimizing performance, security, and governance. Consequences of Incomplete Observability Impact on Performance The absence of complete observability limits an organization’s ability to proactively detect and resolve performance bottlenecks. Without detailed insights into key metrics like latency, response times, and resource utilization, it becomes nearly impossible to identify root causes or improve application responsiveness. For example, undetected spikes in CPU or memory usage can lead to degraded user experiences and even system crashes. Impact on Availability Incomplete observability hampers availability—an essential component of application delivery. Downtime and overlooked critical failures are costly, with 32% of organizations reporting an average outage cost exceeding $500,000 per hour (New Relic). For distributed systems, limited visibility can cause cascading failures, with a minor issue in one system component triggering widespread service interruptions before being detected. Impact on Scalability Dynamic and scalable infrastructure is essential for supporting modern applications with variable workloads. Incomplete observability creates significant obstacles in tracking traffic trends and resource utilization accurately, leading to resource under-provisioning or over-provisioning that wastes budgetary resources or results in outages. Impact on Operational Efficiency Operational inefficiencies arise when IT teams are forced to sift through fragmented, inconsistent data sets to identify issues. Logs spread across incompatible formats or disconnected tools lead to delays in troubleshooting and limited optimization opportunities. This reduces teams' ability to respond to incidents promptly and improve overall system performance. Best Practices for Overcoming Observability Gaps F5's BIG-IP and OpenTelemetry address these challenges by delivering end-to-end observability capabilities requiring real-time insights into application health, performance bottlenecks, and operational metrics. These tools facilitate timely root cause analysis and enable proactive management of distributed systems. Enhanced Observability Framework: Use Case Overview The following diagram illustrates a practical implementation of comprehensive observability using tools like F5 BIG-IP and OpenTelemetry: Use Case Breakdown Consolidate Traffic via F5 BIG-IP LT BIG-IP LTM acts as a centralized point for SSL termination, iRules, and high-speed logging, capturing critical metrics like latency, VIP health, and trace IDs. Traffic is centrally analyzed to provide real-time visibility into application flow dynamics. Capture and Export Logs & Metrics Key metrics, logs, traces, and request IDs are captured and exported for downstream analysis. Logs are standardized across systems, ensuring that valuable data isn't lost in noise. Standardize Observability with OpenTelemetry OpenTelemetry normalizes diverse observability patterns into a unified data model. This enables cross-system compatibility and real-time trend comparisons in distributed environments. Implement Dynamic Alerts & Automated Responses Configure dynamic alerting systems to notify teams when anomalies are detected and integrate automated responses for tasks such as scaling resources or rerouting traffic. Create Unified Dashboards & Analytics Observability platforms like ELK, Prometheus, and Datadog aggregate logs and metrics into a central dashboard, delivering actionable intelligence to IT teams. Establish Feedback Loops for Continuous Improvement Feedback loops using historical performance data enable ongoing improvements in application delivery processes. Insights refine operational decisions and better align infrastructure with real-time demand. Key Benefits Enhanced visibility into application flows, including API interactions, access patterns, and system utilization. Rapid issue detection and mitigation using real-time analytics and automated responses. Resource optimization ensures cost-effective scaling aligned with workload demands. Improved governance and security through dynamic control of inter-application communications. Conclusion Incomplete observability disrupts critical aspects of performance, availability, scalability, and operational efficiency. By leveraging solutions like F5 BIG-IP and OpenTelemetry, alongside enhanced observability frameworks, organizations can address visibility gaps effectively. Dynamic alerting systems, unified dashboards, and standardization tools enable real-time insights, fostering a culture of data-driven decisions and continuous service improvement. Observability is no longer just a supporting feature. It has become the strategic foundation for reliable, high-performing, and secure digital ecosystems. Start improving your observability practices today to achieve long-term success in application delivery. Reference Articles Enhancing BIG-IP with F5 Distributed Cloud: Automated Service Discovery for Scalable Application Delivery and Security Adopting SRE practices with F5: Observability and beyond with ELK Stack Monitor Application Availability with F5 BIG-IP LTM Why Application Observability and Insights Matter Gain insights into the performance of your F5 BIG-IP LTM and DNS solutions ADSP Platform overview The Application Delivery Top 10 AI reference architectureAutomatic Certificate Management with ACMEv2 in F5 BIG-IP
One of the most anticipated features of F5 BIG-IP is integration with ACMEv2. With the General Availability of BIG-IP 21.1.0 on May/26, this feature came into being. In this tutorial, we are going to configure it, using Let's Encrypt as the CA. The domain for which we are generating/renewing certificates is carlosf5lab.lat. The official docs for this feature are located in SSL Certificate Management | BIG-IP Documentation. Pre-requisite 1: DNS Resolver that can reach the internet (at least the CA endpoints). In this case, we are using the native DNS Resolver that comes with BIG-IP. Pre-requisite 2: The internal proxy that will make the connection with the CA. Pre-requisite 3: a self signed SSL certificate that the ACMEv2 protocol uses as the identifier for a device account. You don't have to fill the Subject Alternative Name. For the Common Name, an e-mail contact is advised. Now, we are going to create the ACME Provider object. Give it a name, and select the internal proxy previously created. For the CA Certificate to enable the secure connection with the Directory URL, you can use the default ca-bundle.crt. The Directory URL is the endpoint for the ACMEv2 protocol. In Let's Encrypt case, it is https://acme-v02.api.letsencrypt.org/directory For the Account Key, choose the previously created self-signed certificate. For the trickier part of all, the field "Contacts" is mandatory, and it must be an URL. That’s why you must use the format mailto:email_address. Check the Terms and Conditions, and the Create Account boxes. After a while, the Account Status must read as "Valid". To prove you own the domain whose certificate Let's Encrypt is going to create/renew, it must be pointing to an IP (A Record) where you must have your Virtual Server listening on Port 80 configured to respond to the ACMEv2 Challenge. (In this specific lab, the domain carlosf5lab.lat points to a Public IP mapped to an internal IP). Now you can order your first certificate via ACMEv2 on BIG-IP: After a while, the Key tab should read something like: Which means your certificate was generated: To track the ACME Provider, you can check its statistics: That's it, my friend! If it helped you, give a thumbs up to this post!IIS X-Forward-For ISAPI Filter
A recent customer issue came up where they were load balancing servers but we unable to get the true client address logged in their IIS logs. They had their servers fronted by a BIG-IP and when clients would make requests the address passed to the server was the internal address and not that of the client. This is a common issue with proxies and fortunately there is a standard for forwarding client information. It is the HTTP X-Forwarded-For header which is handled by most proxies. So, I set out to find an existing ISAPI filter to replace the c-ip (client ip) log value in IIS with the contents of the X-Forwarded-For header (if it exists). I was amazed to find that I couldn't find a single instance of any open source (or even commercial) filter that would do this. So, I dug out Visual Studio and whipped up a filter that does just that. It's very basic and contains no user configuration so all you need to do is plug it into your Web Applications list of ISAPI Filters within the IIS Administration and you're set to go. We've released the source under the iControl End User License Agreement (available in any iControl SDK download). You can download it here. If you find a way to optimize this filter, please let me know and I'll update the sources here. After 24-hours of posting, a customer already returned some performance testing on the filter indicating that it only effected the traffic by less than 1 percent. I'm sure there are ways to optimize the memory allocation in the filter to speed this up a bit more, but I'll leave that for the community to work on. Oh, and it should be noted that the X-Forwarded-For header isn't supported the same way across all proxy products so you'll want to make sure you test this out before using it. It is expecting the header to only contain an IP Address as it does a straight substitution on the value in the c-ip section of the log entry. Enjoy! -JoeADC08 – Lack of Security & Regulatory Compliance
As data-driven applications become integral to the digital economy, industries such as Finance, Healthcare, Insurance, Telecommunications, Hi-Tech, Energy, Government, Retail & E-commerce, Automotive, and Manufacturing face increasing pressure to comply with strict data security and regulatory compliance frameworks. Global regulations regarding data sovereignty, privacy, and security are intensifying, requiring organizations to design their systems to adhere to these mandates while maintaining performance, scalability, and operational efficiency. F5 provides critical infrastructure for secure and compliant application delivery through solutions like Web Application Firewall (WAF) and Management Control Protocol (MCP). By leveraging these technologies, applications across industries can effectively mitigate risks while enhancing performance, availability, and scalability amidst continuously evolving compliance challenges. Although this article uses a Finance example to illustrate these concepts, the principles discussed are broadly applicable to all industries. AI Reference Architecture Use Case example: Financial App Security via F5 WAF and MCP The diagram (above) outlines the process flow of a financial application employing F5 WAF and MCP to bolster security and meet compliance requirements. Here's a breakdown of the use case flow: Client/AI Agent Initiates Request A user or an AI system generates a request to the financial application. F5 BIG-IP ADC Layer: The traffic flows through the F5 Application Delivery Controller (ADC), where critical security and compliance measures are applied: SSL/TLS Encryption & Offloading: Protects sensitive data during transmission by encrypting it using industry-standard protocols. SSL/TLS offloading reduces server overhead and ensures seamless performance. Web Application Firewall (WAF): Detects and blocks malicious traffic, including threats like injection attacks, cross-site scripting (XSS), and other OWASP Top 10 vulnerabilities. FIPS Compliance Checkpoint: Enforces adherence to Federal Information Processing Standards (FIPS) for applications handling sensitive financial and government data. Central Logging & Automated Compliance Enforcement All activities are captured through centralized logging and monitored for compliance violations. Automated tools ensure real-time enforcement of regulatory policies. F5 BIG-IP LTM Load Balancing Optimizes traffic distribution across backend servers, ensuring performance and high availability. MCP Server Processing Data is processed and stored within the MCP server infrastructure, maintaining data sovereignty by adhering to local jurisdiction and privacy laws. Observability & Regulatory Reporting Continuous monitoring enhances visibility into application performance and security. Comprehensive reporting ensures regulatory compliance is documented at every layer. How Compliance Impacts Financial Applications Performance: Regulations like data localization laws introduce performance challenges by requiring data to be stored and processed within specific regions. Encryption processes, such as SSL/TLS, add to computational overhead, and inefficient encryption management can create bottlenecks, particularly in latency sensitive AI driven financial applications. Availability: Lack of compliance with security regulations leads to greater exposure to breaches and downtime. For example, noncompliance with GDPR in the European Union can result in forced system outages and expensive remedial actions to meet regional requirements. Scalability: Data sovereignty regulations limit scalability by requiring organizations to duplicate infrastructure in multiple regions. This can lead to higher operational costs and hinder AI financial applications from leveraging centralized data for model training and transactions. Operational Efficiency: Addressing compliance failures often demands significant manual intervention, diverting IT resources away from strategic projects. Moreover, regulatory violations expose organizations to costly fines, legal penalties, and reputational harm, further affecting profitability and trustworthiness. Best Practices for Ensuring Security and Compliance Financial institutions can enhance application delivery by implementing a suite of measures targeted at addressing compliance and security risks: Encryption with FIPS-Compliant Devices: Utilize advanced encryption protocols to protect sensitive data in transit and at rest. Deploy FIPS-compliant devices to meet federal standards for handling regulated data, ensuring robust security and regulatory adherence. Web Application Firewall (WAF): Secure applications against common vulnerabilities and ensure compliance with industry standard frameworks like PCI DSS. This is critical for financial applications handling transaction data. Automated Compliance Checks and Centralized Logging: Automate compliance validation and real-time monitoring to streamline operations. Centralized logging aids in regulatory audits and ensures transparency while maintaining operational efficiency. Geolocation-Based Traffic Routing: Use application delivery infrastructure to enforce data residency requirements via geolocation based routing, ensuring compliance with regional data sovereignty laws. Scalable and Redundant Infrastructure: Design scalable architectures with redundant systems in compliance with specific jurisdictions, reducing downtime and ensuring reliability across regions. Conclusion The intersection of security, regulatory compliance, and application delivery is critical across all industries, as failure to meet these standards can have financial, operational, and reputational consequences. While this article focused on the financial services sector as an example, the principles and strategies discussed, such as leveraging F5 solutions like WAF and MCP to enhance security, ensure compliance, and optimize performance, are equally applicable to other industries including Healthcare, Retail, Telecommunications, and more. By prioritizing encryption, automation, and scalability, organizations across sectors can navigate regulatory challenges and deliver secure, scalable, and efficient services in today’s increasingly regulated and data driven landscape. Reference Articles Industry-leading application delivery and security services The Application Delivery Top 10 ADSP Platform overview AI reference architecture Mitigating OWASP API Security Risk: Mass Assignment using F5 BIG-IP F5 BIG-IP Zero Trust with BIG-IP SSL OrchestratorOracle WebLogic Server
F5 and Oracle have long collaborated on delivering market-leading application delivery solutions for WebLogic Server. F5 has designed an integrated, agile, and adaptable network platform for delivering WebLogic applications across the LAN and WAN, and packaged this information in our deployment guides and iApp templates. The result is an intelligent and powerful solution that secures and speeds your WebLogic deployment today, while providing an optimized architecture for the future. The following simple, logical configuration example shows one of the ways you can configure the BIG-IP system for Oracle WebLogic Servers using BIG-IP AAM technology to speed traffic across the WAN. See https://f5.com/solutions/deployment-guides to find the appropriate deployment guide for quickly and accurately configuring the BIG-IP system for Oracle WebLogic Server. If you have any feedback on these or other F5 guides or iApp templates, leave it in the comment section below or email us at [email protected]. We use your feedback to help shape our new iApps and deployment guides.Regional Edge SaaS Application Deployment Recommended Practices
The guidelines presented in this walkthrough are informed by extensive field experience, incorporating insights from customers, F5 Solutions Engineers, Architects, and Professional Services teams. Having supported numerous deployments across diverse environments, this cumulative knowledge provides a practical and reliable starting point for publishing applications. This guide aims to help navigate the Distributed Cloud platform, offering a balanced approach to application delivery and security. The below figure represents the HTTP-LB configuration options alongside the typical flow of traffic from downstream client to upstream origin or application endpoint. Prerequisites: Article on how Distributed Cloud advertises and picks up traffic (Listener Logic) can be found here: F5 Distributed Cloud - Listener Logic Domains and Certificates: This is where you define what the load balancer listens on and how it presents itself to clients. For an initial deployment with a publicly accessible application, the following settings cover most use cases. We recommend a single HTTP-LB per application for visibility, telemetry, day 2 operations, and blast radius. Any more consolidation may cause friction going forward. Recommended Settings: Domain Name:Your application FQDN (e.g., app.example.com) Load Balancer Type: Certificate: Use F5 XC Auto-Cert when possible. If your domain is delegated to XC DNS, certificate management is fully automated. For non-delegated domains, F5 XC provides a cname challenge record value that you can add to your DNS provider to satisfy the Lets Encrypt ACME challenge, then add the provided Host Name.ves.io CNAME record pointing to XC to complete the certificate setup HTTP to HTTPS Redirect:Enable HSTS Header:Enable Listener Port: 443 Client-Side TLS: High security profile Protocol: HTTP/1.1 and HTTP/2 Origins and Health Checking: Origin Pools: A note on structure and origin access: Use Routes (covered in the next section) as the primary mechanism for attaching Origin Pools to your HTTP LB. The default Origin Pool field on the HTTP LB itself is best reserved as a potential fallback option depending on origin and route design (Engage your F5 SE if you need to discuss further). This approach gives you path-aware routing controls and per-route retry and timeout tuning. Also at the origin premise you should limit access to F5 Distributed Cloud RE’s only. You have a few options to achieve this. Limit via a security group or IP Access List to RE IP ranges, mTLS, insert a response header from the HTTP-LB that the origin server is expecting, or a combination of the 3. Recommended Settings for Publicly Available Endpoints: Origin Server Type: Public IP-based. Provide the IPv4 address directly. Where possible, target origins by IP with Host Header. If using public DNS-based origin endpoint targeting, be aware that XC does not honor standard DNS TTL and overrides the value! Origin Server Port: 443 Connection Pool Reuse: Enable Health Check Port: Endpoint port (same as origin port unless otherwise directed) Load Balancing Algorithm: Load Balancer Override (Load Balancer algorithm set at HTTP-LB) Endpoint Selection: Local Endpoints Preferred (Distributed Cloud Construct to leverage local endpoints over remote endpoints goal is to Egress the same RE as Ingress) TLS to Origin: Enable TLS SNI: Host Header TLS Security Level: High Origin Server Verification: Use Default Root CA Certificate mTLS: Disable unless required Other Origin Pool Options: Exception Handling: Setup for specific application/origin server requirements (configurable options available for application specific error-handling requirements) Origin Server Subsets: Disable unless you have a specific canary or subset routing requirement HTTP Protocol Configuration: Automatic (adjust to a specific version if your origin requires it) Proxy Protocol: Disable LB Source IP Persistence: Disable Health Checks: The default health check thresholds are tuned conservatively. In practice, this means a failed origin stays in rotation longer than it should, and a recovered origin comes back into rotation slowly. Adjust the thresholds to react more aggressively to recovery while still being tolerant of transient failures. Recommended Health Check Settings: Setting Recommended Value Default Why Healthy Threshold 1 3 Bring a recovered endpoint back into rotation after a single successful check Unhealthy Threshold 3 1 Require 3 consecutive failures before removing an endpoint — avoids flapping on transient issues Interval 15 seconds 15 seconds No change needed Jitter Percent 30% 30% Stagger health check timing across endpoints (30% of 15s = up to 4.5s offset) Key insight for jitter setting: When you have multiple endpoints in a pool, health checks without jitter all fire at the same instant that can create amongst other issues a temporary artificial network congestion or failure at endpoints. The 30% jitter setting randomizes the start time of each health check within a window (default: 15s interval with a 30% jitter, so checks are offset by up to 4.5 seconds). Leave this at the default unless you have a specific reason to change it. Advanced health check options: Host Header: Set to the value your application expects (do not leave blank if your origin validates the Host header) Path: Set a meaningful health endpoint like /healthz a common convention, but use whatever your application exposes Expected Status Codes: 200, 3xx Request/Response Header Manipulation: Add or remove headers as required by your application Expected http response: Validate the Raw Bytes expected in the Response of HTTP Health Check Origin Pool Display in UI: Routes: Routes are where your HTTP LB gains precision. Rather than sending all traffic to a single origin pool, Routes let you make forwarding decisions based on path, method, headers, and query parameters. They also expose per-route controls for timeouts, retries, header manipulation, and security policy controls that are not available at the origin pool level. Recommended Route Configuration: Route Type: Simple Route HTTP Method: Any Path Match: Prefix (use Regex or Exact match when you need more specificity) / Headers: Add header match conditions only if required Port Match: Adjust only if needed Origin targeting: Origin Pools:Add the Origin Pool(s) you configured in the previous section Host Rewrite Method: Automatic Host Rewrite (or set a specific value if your origin requires a fixed Host header) Query Parameters: Retain (Remove and Replace are available options) Route Activation: Enabled (Disable option) Advanced route options: Load Balancing Control: Use LB Hash Policy Priority: Default Origin Server Subsets: Leave unset unless subset routing is required Request/Response Manipulation: Header add/remove, cookie add/remove, set-cookie add/remove -configure as needed for your application Security (per-route overrides): WAF:Inherit from HTTP LB (recommended) or specify a different policy per route WAF Exclusion:Inherit from HTTP LB or specify an Exclusion Policy — see the note in the Gotchas section below on inline exclusions CORS and CSRF:Configure as needed Protocol Upgrades: SPDY:Disable (enable only if required) WebSockets:Disable (enable only if your application requires WebSocket support) Retry Policy: Key insight “retry values”: The retry settings below are tuned for typical HTTPS application traffic. The per-retry timeout of 1000ms prevents a slow origin from consuming the full route timeout on every attempt, and the retry interval backoff (25ms initial, 2500ms max) avoids hammering a struggling origin. Validate these values with load testing before going live. Custom Retry Policy Settings: Retry Conditions: 500 Gateway-error Connect-failure Refused-stream Reset Retriable-4xx Number of Retries: 3 Per-Retry Timeout: 1000ms Retry Interval: 25ms Max Retry Interval: 2500ms Miscellaneous Route Options: Route Timeout: 30000ms (30 seconds) Route-Specific Buffering: Common Mirroring: Disable Cluster Retract: Disable (validate with testing before enabling) Security Settings: Security configuration in F5 XC is layered. The settings below represent a solid baseline for an initial deployment. Individual applications will likely require tuning — use this as a starting point, not a final state. Web Application Firewall: The recommended starting posture is blocking mode with High and Medium attack signatures active. This catches the most common attack patterns while keeping false positive rates manageable. Do not start in detection-only mode and leave it there — establish a review cycle and move to blocking on a defined schedule. WAF: Enable Enforcement Mode: “Monitor” then after review migrate to “Blocking” Security Policy:Custom Attack Signatures: Default signature set High, Medium, and Low severity Automatic Attack Signature Tuning:Disable Automatic Signature Staging:Enable (7days) Threat Campaigns:Enable Violations:Default Signature-Based Bot Detection:Default Enhance with AI: Enable Mitigate High and Medium (have a review cadence) WAF Exclusion: Use a dedicated WAF Exclusion Policy object — do not use inline exclusions (see Gotchas below) Additional WAF-adjacent features — configure as needed for your application: Data Guard CSRF Protection GraphQL Inspection Cookie Protection API Protection: Enable as needed. If your application exposes a defined API surface, uploading an OpenAPI spec and enabling API Discovery is a worthwhile early step. Malware Protection: Disable DoS Settings: Mitigation Action: Block RPS Threshold: Typically based on Capacity of origin application endpoints Client-Side Challenge: Enabled if web based application Custom Service Policy for DoS: Apply a specific geo location and IPI during DDoS DDoS Mitigation Rules: Default Slow DDoS: Default Service Policies: Service Policies match on a set of criteria and apply an action. They operate at the connection level and complement WAF, which operates at the request content level. Service Policies: Apply Specified Service Policies (list any access restriction policies you need) IP Reputation: select the reputation categories appropriate for your threat model Threat Mesh: Disable User Identifier Policy: Create a policy using Client IP and TLS Fingerprint based on JA4 as identifiers (other identifier types are available) Malicious User Detection: Enable Malicious User Mitigation: Enable, Default settings Rate Limiting: Configure as needed based on expected traffic profile Trusted Client Rules: Configure as needed Client Blocking Rules: Configure as needed CORS Policy: Configure as needed Other Settings: VIP Advertisement: This is what makes the HTTP LB publicly reachable through the F5 Regional Edge network. If you change this to a Custom setting, validate your advertisement policy carefully as a misconfiguration here means no traffic reaches your LB. Advertise Internet Load Balancing Algorithm: Choose based on your application's session and traffic characteristics. Round Robin is the default and works for most stateless applications. If you have session affinity requirements, evaluate the hash-based options. Trusted Client IP: Disable unless you have a specific use case that requires preserving the original client IP through a proxy chain upstream of XC. Location Header: Enable “Add XC RE Ingress Location” to include the Regional Edge location in response headers. This is useful for troubleshooting and for understanding which RE node handled a request. Header and Cookie Options: Configure request/response header additions, removals, and cookie manipulation as required by your application. Error Response: Configure custom error response pages as needed. The default F5 error pages are functional but not branded. Most production deployments will want custom responses for 4xx and 5xx errors. Buffer Policy: Default: No buffering (requests are streamed to the origin) Max Buffer Size: 10,485,760 bytes (10MB) – if the request body exceeds this, XC returns HTTP 413 (Payload Too Large) Timeout behavior: If the full request body is not received before the timeout, XC returns HTTP 408 (Request Timeout) When to enable: Enable buffering if your origin cannot handle streaming request bodies, or if you are using WAF inspection on large POST requests Compression: Algorithm: GZIP only (not configurable) Compression Level: 5 (not configurable) Behavior: XC compresses responses dispatched from the upstream origin when the client signals support via “Accept-Encoding” Enable if your origins do not already compress responses and your client traffic includes browser-based users Idle Timeout: Default: Client Side - 30 seconds Behavior: A stream with no activity (upstream or downstream) for this duration is terminated with HTTP 504 Increase for applications with long-running server-sent events, streaming responses, or slow upload scenarios Common Gotchas: These are the issues that come up repeatedly in initial deployments. Check these before you go live. Using DNS-based origin targeting instead of IP + Host Header: By default Distributed Cloud rewrites the host header so validate and set appropriately. DNS-based origin targeting works but introduces a dependency on DNS TTL behavior that XC does override in non-standard ways. During a failover event, you may experience stale resolution longer than you expect. Use IP-based origin with Host Header wherever possible for predictable behavior. Not Configuring a Health Check at all or Leaving health check thresholds at defaults: By default, a health check is not added to an origin pool. Also, the default Healthy Threshold of 3 means a recovered origin must pass 3 consecutive checks before re-entering rotation. If your check interval is 15 seconds, that is 45 seconds of unnecessary exclusion for an endpoint that came back healthy. Set Healthy Threshold to 1. The default Unhealthy Threshold of 1 is the opposite problem a single failed check removes the endpoint. Set Unhealthy Threshold to 3 to tolerate transient blips. Attaching Origin Pools directly to the HTTP LB instead of utilizing Routes: The default Origin Pool field on the HTTP LB is a catch-all fallback. If you attach your primary origin there and do not configure Routes, you lose access to per-route retry policies, timeouts, header manipulation, and security overrides. Build your routing structure with explicit Routes from the start. Starting in WAF Monitor mode and never switching to blocking: Monitoring mode is a valid tuning step, but it is easy to leave it there indefinitely. Set a review date when you configure monitor mode. Establish your false positive baseline and move to blocking on a defined timeline. Not setting a Host Header on the health check: Leaving the Idle Timeout at 30 seconds for streaming applications: The 30-second idle timeout will terminate long-lived connections, WebSocket connections, server-sent event streams, slow uploads without warning. If your application uses any of these patterns, increase the Idle Timeout before testing.How I Did It: Simplifying BIG-IP Telemetry for Third-Party Observability Platforms
In this article, I introduce two community-supported projects designed to help customers get more value from BIG-IP telemetry. The BIG-IP Telemetry Streaming Validator and Configurator simplifies the deployment of Telemetry Streaming, while the BIG-IP Telemetry Exporter helps integrate BIG-IP metrics and logs into existing observability platforms such Grafana, Splunk, Elastic, Datadog, and other modern analytics ecosystems utilizing OpenTelemetry.HTTP Load Balancer Routes on F5 Distributed Cloud
Route misconfiguration is one of the most common configuration mistakes we see that can cause incidents on F5 Distributed Cloud (F5 XC). The four route types look deceptively simple in the console, but they have distinct behaviors, ordering rules, and gotchas. This article covers all four types with real field names, decision guidance, and the mistakes that actually happen in production. What Routes Do in F5 XC HTTP Load Balancer An HTTP Load Balancer in F5 XC is a full L7 proxy running at the Regional Edge (RE) and depending on the deployment model, Customer Edge (CE). We will use the Regional Edge as a deployment model for this article. When a request arrives, the RE evaluates the route list in order and applies the first matching route. That route determines what happens to the request: forward it to an origin, redirect the client, return a synthetic response, or apply advanced routing logic. Routes can be configured inside the HTTP Load Balancer configuration which opens a new Route Options window: Multi-Cloud App Connect > Load Balancers > HTTP Load Balancers > [your LB] > Routes > Route Options The four route types map to three underlying route actions: XC Route Type Route Action Traffic Goes To Simple Route route Origin Pool Redirect Route redirect Client (3xx response) Direct Response Route direct_response Client (fixed response body) Custom Route route / redirect / direct_response Depends on configuration Route Matching: How XC Evaluates Routes Route evaluation is sequential, stops on first match, and has no automatic specificity ranking. The order you set is the order XC uses. Evaluation Order The HTTP Load Balancer evaluates routes sequentially, top to bottom. The first route that matches the incoming request wins. No further routes are evaluated. This means: More specific routes must appear before broader ones. A catch-all route (prefix /) at the top will swallow everything. Nothing below it will ever match. Path Match Types Three path match types are available across all route types: Match Type Field Behavior Prefix path_prefix Path must begin with the specified string Exact exact Path must equal the value exactly (query string excluded) Regex regex Entire path (minus query string) must match the regex pattern Prefix matching pitfall: The prefix /api matches /api/v1/users but also /apikeys and /api-internal. If you want to match a path segment boundary, use /api/ (trailing slash) or switch to regex. Additional Matching Criteria Beyond path, routes can match on: HTTP methods: GET, POST, PUT, DELETE, etc. Request headers: presence, exact value, regex Query parameters: Retain, Remove, or Replace Combining criteria (e.g., path prefix + method + header) creates an AND condition: all specified criteria must match. Route Type 1: Simple Routes Simple routes are the workhorse of most HTTP Load Balancer configurations. They match a path (and optionally method/headers) and forward traffic to an Origin Pool. When to Use Standard application traffic forwarding Path-based routing to different backend services API versioning (/v1/ → pool A, /v2/ → pool B) Microservice fanout from a single domain Key Configuration Fields Field Description Path match type Prefix / Exact / Regex HTTP Method Any, GET, POST, PUT, etc... Origin Pool The backend pool receiving the request Host Rewrite Method Disable/ Hostname / Header value: rewrites the Host header sent to origin Query Parameters Retain, Remove, Replace Advanced Options Worth Knowing Path rewriting (under Advanced Options): Disabled: path sent to origin unchanged Prefix Replacement: replaces the matched prefix with a new string (e.g., strip /api/v1 prefix before sending to origin) Regex-based: full regex substitution on the path Retry policy: The default retry policy is 1 retry on 5xx responses. Set this explicitly for your application in every route: Disabled: no retries; required for write operations Default: 1 retry on 5xx Custom: specify retry conditions, count, and interval Per-route WAF override: Each simple route can attach its own WAF App Firewall policy. This completely replaces the load balancer-level WAF for matching requests; it is not additive. Use this to enforce stricter rules on sensitive paths (e.g., /admin/) or to relax inspection on certain paths. Example: API Path Routing Route 1: Prefix /api/v2/ → origin-pool-v2 (exact origin for v2) Route 2: Prefix /api/v1/ → origin-pool-v1 (legacy backend) Route 3: Prefix /api/ → origin-pool-api (catch-all for API paths) Route 4: Prefix / → origin-pool-web (catch-all for everything else) Order matters here. If route 3 or 4 appeared first, routes 1 and 2 would never fire. Route Type 2: Redirect Routes Redirect routes return an HTTP 3xx response directly to the client. No origin pool is involved: the RE handles the response entirely. When to Use HTTP → HTTPS redirect (though XC has a dedicated LB-level toggle for this) Domain canonicalization (www.example.com → example.com) Legacy URL migrations (/old-path/ → /new-path/) Temporary redirects during maintenance or A/B migrations Key Configuration Fields Field Description Protocol HTTP or HTTPS Host Target FQDN; supports non-standard ports Redirect Path / URI Target path; if left unset, original URI is preserved (including query string) Response Code 301 (Permanent), 302 (Temporary), 307, 308 Redirect Behavior: URI Preservation When you leave the redirect path unset, XC preserves the original URI path and query string in the Location header. This is useful for protocol/host-only redirects where you just want to change the scheme or domain without touching the path. Example: Redirect all HTTP traffic to HTTPS on the same host: Protocol: HTTPS Host: same-as-request (leave blank or match domain) Path: (unset — preserve original URI) Code: 301 Redirect Route Limitations Simple redirect routes (defined directly on the LB) do not support custom header manipulation on the redirect response. If you need to inject headers (e.g., Cache-Control: no-store on the redirect response), use a Custom Route object instead. Route Type 3: Direct Response Routes Direct response routes return a fully synthetic HTTP response to the client. The request never reaches an origin pool: the RE generates the response itself. When to Use Health check endpoints that should always return 200 (e.g., /healthz) without touching the app Maintenance mode pages: serve a 503 with a message body while origin is down Blocking specific paths with a meaningful error body (vs. a generic deny) Canary or feature-flag placeholders that return 404 before the feature ships Robots.txt or security.txt served from the edge without an origin Key Configuration Fields Field Description HTTP Status Code Any valid HTTP status code (200, 403, 503, etc.) Response Body Static text or HTML body returned to client Path match Same prefix/exact/regex options as other route types Example: Edge-Served Health Check Path: Exact /healthz Method: GET Action: Direct Response Status: 200 Body: OK This responds to health probes from AWS ALB, Kubernetes ingress controllers, or uptime monitors without any load on the backend. Particularly useful during blue/green deployments when the app might not yet be healthy. Route Type 4: Custom Routes Custom routes reference standalone Route objects created separately in XC and attached to one or more HTTP Load Balancers. Unlike the other three types, they follow a service-mesh model rather than a traditional LB model. When to Use Custom Routes Weighted traffic splitting between origin pools (canary releases, blue/green) Request/response header manipulation not available on simple routes Advanced retry policies with specific conditions and intervals Traffic mirroring (shadow traffic to a secondary backend for testing) Reusable route logic shared across multiple load balancers Architecture: Route Objects vs. Inline Routes Inline routes (simple, redirect, direct response) are defined directly on the HTTP Load Balancer. Custom route objects are: Created as standalone objects under Multi-Cloud App Connect Referenced by the HTTP Load Balancer Reusable: multiple LBs can reference the same route object Weighted Clusters Custom routes enable weighted traffic splitting across multiple upstream clusters, equivalent to BIG-IP pool ratio weights or AWS ALB weighted target groups. Route: Prefix /api/ Cluster A (origin-pool-v2): weight 90 Cluster B (origin-pool-v1): weight 10 This is the correct mechanism for canary deployments and gradual traffic shifts on F5 XC. The weights are percentage-based and must sum to 100. Header Manipulation Custom routes support header operations at the route level, applied before forwarding to origin: Operation Direction Example Add header Request X-Forwarded-For: {client-ip} Remove header Request Strip Authorization before certain paths Add header Response Strict-Transport-Security: max-age=31536000 Remove header Response Strip Server header from responses Header manipulation runs in order: route-level → virtual host-level → route configuration-level. Retry Policies Route retry policies take complete precedence over any virtual host-level retry policy. Configure explicitly: Field Description Retry on Conditions: 5xx, gateway-error, reset, connect-failure, retriable-4xx Number of retries Integer Per-try timeout Timeout applied to each individual retry attempt Retry interval Base interval between retries Traffic Mirroring Mirror policies shadow a copy of each request to a secondary cluster. The mirrored request is fire-and-forget. Example: Testing a new backend version against live traffic without affecting users Security analysis pipelines Route Ordering and Priority Route order is the most common source of routing bugs in XC deployments. There is no automatic specificity ranking; you own the order. The Rules Routes evaluate top to bottom. First match wins. Evaluation stops. Disabling a route (via the Route Activation Status toggle) does not remove it: traffic falls through to the next matching route. Recommended Ordering Pattern Order routes from most specific to least specific: Exact paths first Exact /api/v2/auth/token 2. Specific prefixes next Prefix /api/v2/ 3. Broader prefixes after Prefix /api/ 4. Path-specific exceptions Exact /healthz 5. Catch-all last Prefix / Common Ordering Mistakes Mistake Symptom Fix Catch-all prefix / first All traffic hits one origin; other routes never fire Move catch-all to last position /api prefix before /api/v2/ V2 traffic hits wrong origin Reverse the order Disabled/Unused route above active route Traffic silently hits next route with different behavior Remove disabled routes; don't rely on toggle for permanent changes Redirect route below a prefix match Redirect never fires Move redirect above the prefix that would match it first Common Mistakes and Gotchas Prefix /api matches /apikeys. The prefix match does not anchor to path segment boundaries. /api matches /api/, /api/v1/, and also /apikeys, /api-docs. Use /api/ (trailing slash) or regex if segment boundary matters. Per-route WAF is a full replacement, not additive. Attaching a WAF policy to a route does not stack with the LB-level WAF. It replaces it entirely for that route. If your LB WAF is in blocking mode and you attach a route-level WAF in monitoring mode, that route is now in monitoring mode only. Custom routes enforce TLS: test before production. If your origin uses a self-signed certificate and you switch from a simple route to a custom route without uploading the Root CA, connections will fail. Test in a staging environment first. Header manipulation on redirect requires a custom route object. Simple redirect routes in XC do not support response header injection. If you need Cache-Control or Vary headers on your redirects, you must use a standalone custom route object with redirect action. Regex route performance at scale. Regex routes require full path evaluation on every request. At high request volumes, a large number of regex routes adds measurable CPU overhead compared to prefix or exact routes. Use regex only where prefix or exact matching is insufficient. FAQ Q: What is the difference between a Simple Route and a Custom Route? A: Simple routes are inline on the HTTP Load Balancer and forward traffic directly to an Origin Pool. Custom routes are standalone objects that use an Endpoints → Clusters → Routes model, support weighted traffic splits, header manipulation, and mirroring, but cannot reference Origin Pools directly. Q: Why is my catch-all route matching everything instead of the specific routes below it? A: Route evaluation stops at the first match. If your catch-all prefix (/) is above more specific routes, it wins every time. Move the catch-all to the last position in the list. Q: Can I use a Custom Route to send traffic directly to an F5 XC Origin Pool? A: No. Custom routes do not support Origin Pools directly. They use an Endpoints → Clusters → Routes abstraction. If you need weighted splitting with Origin Pool support, custom routes are not the right fit; simple routes forward to Origin Pools but do not support weighted clusters. Q: My POST requests are creating duplicate records and I traced it to XC retries. What is happening? A: The default retry policy on simple routes is "1 retry on 5xx." A POST that hits a 500 gets retried once, potentially double-writing. Set the retry policy to Disabled on any route handling non-idempotent operations (POST, PUT, PATCH, DELETE). Q: Does attaching a WAF policy to a route add rules on top of my LB-level WAF? A: No. Per-route WAF replaces the LB-level WAF entirely for requests matching that route. If your LB WAF is in blocking mode and the route WAF is in monitoring mode, those requests are evaluated in monitoring mode only. Q: My Custom Route TLS connections to origin are failing but the same origin works fine on a Simple Route. Why? A: Custom routes enforce strict TLS with no skip-verify option. Simple routes do not have this requirement. For custom routes, upload the Root CA certificate for your origin, or use the use_volterra_trusted_ca_url flag via the API for public CAs. Self-signed certs without the Root CA uploaded will fail silently. Q: I disabled a route in the console but traffic behavior changed unexpectedly. What happened? A: Disabling a route via the Route Activation toggle does not remove it from evaluation. Traffic falls through to the next matching route in the list. If that route is a broad catch-all, the behavior shift may look correct until something that depends on specific routing breaks. Remove routes you no longer need rather than toggling them off.Catch, Clean, and Release: File Sanitization with F5 and OPSWAT
In today’s hyper-connected business environment, organizations rely heavily on the constant exchange of digital documents, yet did you know over 90% of malware continues to be delivered through seemingly harmless file attachments and downloads. Traditional security tools, such as antivirus scanners and sandboxes, struggle to keep pace because they rely on recognizing known malware signatures or observing suspicious behaviors. As sophisticated attackers increasingly leverage zero-day exploits and polymorphic malware, IT security teams are left vulnerable to threats that have never been seen before, creating an urgent need for a more proactive defense. Content Disarm and Reconstruction (CDR) addresses this gap by shifting the security paradigm from "detection" to "prevention". Instead of wasting critical time trying to determine if a file is malicious, CDR assumes all incoming files are untrustworthy, strips out potentially hazardous components (such as macros, embedded scripts, or active code), and rebuilds a perfectly safe, fully functional copy for the end-user. This proactive sanitization allows organizations to neutralize file-borne threats instantly, eliminating the business disruption of traditional sandboxing delays and ensuring uninterrupted workflow productivity. The "why" part of this solution is needed is fairly obvious now, but the question of "how" remains. This is where the coupling of F5 BIG-IP and OPSWAT Deep CDR™ from OPSWAT's MetaDefender platform offers a quick and easy on ramp to these modern protections. Disarm threats on the fly: Integrating F5 BIG-IP with OPSWAT Deep CDR via ICAP The solution discussed was staged in a lab using real files. It is very flexible, it could leverage ICAP from the standard LTM module of BIG-IP but can also similarly integrate with the Advanced WAF (AWAF) or SSL Orchestrator modules too. Any file traversing the BIG-IP can be directed to the OPSWAT solution for reconstruction, including a topology such as a forward proxy thus offering a cleansing service for employee files egressing a corporate environment. In all likelihood, the most prevalent use case would be in a reverse proxy scenario, where external fileware is being uploaded to a web service. The intent is to provide ironclad shielding from risks embedded into files that are being ingested into a managed network. Of course, all of the existing tried and true features of the BIG-IP remain, for instance this protected web server could consist of a dozen or more intelligently load balanced nodes. The two particular concerns in today’s threat landscape are: Evolving Malware - Modern malware bypasses standard detection using zero-day exploits and polymorphic code Weaponized Documents - A majority of threats enter networks hidden inside benign-looking PDFs, Office docs, and image files The solution from CDR is encapsulated in the following infographic. The BIG-IP is tailor-made to steer fileware to OPSWAT Deep CDR as SSL/TLS interception is a foundational and high-performance attribute of the F5 reverse proxy. In fact, the intercepted files can in turn be provided to MetaDefender via standard ICAP or a secure TLS-based ICAP. Quick first test – scrub images to thwart steganographic attacks A first example, setup in a lab environment, was to cleanse images uploaded to a sample web server dropbox. Think of perhaps a use case of an insurance portal where scanned images are required to corroborate claims. The art of steganography, richly detailed here, is the practice of hiding data inside another file (such as an image, audio file, video, or document) so that the existence of the hidden data is not obvious. Images are made up of millions of pixels, and each pixel contains numerical color values (Red, Green, Blue). A very common technique is called Least Significant Bit (LSB) embedding, simply utilize the last bit to hide a message, message could be an AI prompt or perhaps code, and the human eye will never pick up on the fact that the image has been ever so slightly altered. One pixel stores a tiny piece of the secret message and millions of pixels cleverly used together store the entire message. Normally a PNG or JPEG containing hidden data does not automatically execute code merely because an application opened it, or a staff member simply viewed it. Another component such as a loader, script, vulnerable application, or a browser exploit must extract and use the hidden content. Our task is to subvert the delivery of the instructions and mute the threat. Using one of many available services on-line, a sample image was used to carry a test message via LSB steganography. A companion tool then takes the image file and displays the embedded, hidden message. Our lab deployment is quite simple, as we will upload the above file to a simulated “Resume Upload” dropbox, a typical use case for human resources departments. We see the graphical interface users are presented with when uploading resumes, it is a simple PHP setup based upon NGINX web server. As noted, the graphical image with the steganographic message has been uploaded and recreated by OPSWAT. The test is now to check the recreated file to see if the embedded message has been successfully removed. We see confirmation that the file is now cleansed. The JavaScript diet: removing unwanted code Another threat vector are hidden scripts that no one in HR ever invited into our resume upload service. Using any number of means, perhaps an AI assistant, one can quickly hand craft a pdf that shields within it JavaScript. As seen below, there is no inherent blocking of this file with the native pdf viewer, this is what pops up when we open the file. Let’s take a look at how OPSWAT MetaDefender, and specifically the Deep CDR module, is configured to handle the above file being targeted for upload. The MetaDefender has a rich web UI, by clicking from the main menu “Workflow Management -> Workflow -> File Process” we can ensure that Deep CDR is enabled and further in the setup screen see a list of file types to be scrubbed (double-click on enlarge). To fully explore the OPSWAT capabilities, one can see the documentation here. Simply using pdf as a point of interest, since it pertains to our lab attempt right now to upload a risky file, we see some of the adjustable features around just this one file type. When our test pdf file arrives at the BIG-IP virtual server, we can grab a TCPDump and follow the stream using Wireshark. This is the wire-level details of interest as the ICAP request travels to MetaDefender, in the red font. 1. We observed this is ICAP Request mode targeting the OPSWAT server on port 1344 2. The client’s source IP address is preserved as an X-header value 3. The target of the file, including the DNS name and the full URL path is presented 4. The submitted pdf file is 1045 bytes in length 5. We observed this is indeed our test file, the only text in it is highlighted. The following shows the remaining component of the protocol trace, including the response back from OPSWAT, in blue font, with the modified file. 6. We see towards the end of the submitted pdf, there exists JavaScript 7. OPSWAT MetaDefender now responds to BIG-IP indicating the modified content being provided over ICAP is allowed and may continue its journey 8. The content length has changed from the original 1045 bytes to 930 bytes, the JavaScript has been removed by Deep CDR MetaDefender will leave behind a processing history that can be useful to explore the actions taken by Deep CDR as well as other signatures, such as those from many AV vendors, that may have triggered. The BIG-IP configuration for ICAP is straightforward and described here. In short, it consists of two profiles: Create a virtual server of type “Internal” to create the relationship between the BIG-IP and MetaDefender ICAP Server module. The virtual server will need to reference an “ICAP Profile”. Simply create a linkage in this field to a “child” of the default shipped profile, using this URI syntax (adjust IP address to your OPSWAT server address): icap://200.200.200.2:1344/OMSScanResp-AV In the setup for the virtual server itself that will handle actual received traffic, normally of type “Standard” for a reverse proxy setup, adjust the Request_Adapt_Profile. As normal, create a child of the default Request_Adapt entry and point the “Internal Virtual Name” pull-down to your ICAP virtual server configured in the last set. The hidden passenger – files embedded within files As the ancient saying goes, “It’s turtles all the way down” but more often than not in CDR it can be files all the way down. The last lab test involved trying to forward to our upload service a file which had a wide range of peculiarities associated with it. The main items we want to have dispensed with easily are active URL links that can with one single click connect a user to malicious hiding sites. The second is to remove any embedded files, the files within files threat, that may have malware lurking within. Here we see a demo file with both risks. Simply hovering over the innocuous looking website reveals an active link to a lurking malicious link. Also, note to the right-hand side, this appeared to be a single file, yet the pdf reader clearly indicates there is a Microsoft .docx file attached to the pdf once opened. Double clicking on the attachment, it would be hard to resist but probably not wise; besides yet another masquerading link we also note an executable is hanging around. This is what Word shows. Something tells me that, perhaps, this might not be the best version of putty.exe to install. From the point of view of the users of the protected web experience, the solution was observed to work remarkably fast. Using Developer Tools with Chrome, one notes that the user uploading the above highly questionable file, would see the upload succeed, as per the HTTP 200 OK status code. Interestingly, the entire file reconstruction and delivery is seen in this case to be in the 2 second range. Reviewers of the secure documents may simply be shielded from the details of what was found within the original files, as the security team will already see full details in MetaDefender. However, the user themselves can also be flagged with comments in the cleansed file, indicating in this case to be very cautious, that the file did originally have malicious links at multiple locations. Both approaches are seen respectively in the following examples. The power of prevention at the perimeter The intent of this lab investigation was to put the principles of zero trust into actual practice. Integrating F5 BIG-IP with OPSWAT Deep CDR via ICAP ensures that all file uploads are aggressively sanitized before they ever reach the protected web server. As mentioned, multiple topologies can be supported, in fact the lab BIG-IP was also configured as a transparent forward proxy, and files were similarly cleansed upon egress. The lack of friction in getting a working solution was a key takeaway, BIG-IP is already able to intercept high rates of encrypted TLS traffic and the added time for full inspection and file reconstruction when needed was low to moderate. In an era of stealthy, evasive exploits, relying on detection is a losing game. The strategy is simple: sanitize before you compromise—because a rebuilt file is a secure file.
