Let's Get Critical, Critical
MegaZoneis back again for a roundup of the security news that caught my eye for the week of November 10th - 16th, 2024. This time, I want to get Critical. Yes, let's get into the Critical - issues, of course. We're going to look at some very recent Critical issues making the rounds, as well as issues which made the charts in 2023 - including an old friend which keeps on giving. And I'll end with a critical issue for all of us in the cybersecurity field, one I feel strongly about. Atomic Batteries to Power! Turbines to Speed!Cyber Security Attack Mitigations with BIG-IP features
ArvinF is back to share mitigation options for Cyber Security Attacks with BIG-IP features! This article aim to bring these attack mitigations options much more visible and available. Cyber Security Attacks There are many types of Cyber Security Attacks. I will limit to the types that BIG-IP mostly encounter. Network Attacks Network attacks are aimed at compromising, disrupting, or gaining unauthorized access to an organization’s internal or external network, usually targeting communication protocols, devices, or services within the network. Web Application Attacks Web application attacks target weaknesses in web-based applications to gain unauthorized access to data, manipulate functionality, or exploit users. Web Application and Network Attacks are common and these can affect the Web application and network traffic processed thru F5 Virtual Servers and in some cases, affect the availability of the BIG-IP device itself. Malware Attacks Malware (malicious software) includes viruses, worms, Trojans, ransomware, and spyware designed to damage or gain control over a computer system. BIG-IP Cyber Security Attacks Mitigations But First, Finding "Help" I have always found it helpful to review the Help Tab of the feature and configuration. While logged in on the BIG-IP Configuration utility for the specific BIG-IP Application Security Manager (BIG-IP ASM/Adv WAF) or BIG-IP Advanced Firewall Manager (BIG-IP AFM) menu, the "Help" tab contains details of the relevant configurations. If you hit Launch, it opens on a new window. You can also click on Expand All to view each options documentation. On to the mitigation options.. Bot Defense Profile We have here the description of the Bot Defense Profile Templates where it provides details of each template - Relaxed, Balanced and Strict - on the Verification and Mitigation it will provide. Take note of the Relaxed template Browser verification as it uses "Challenge-Free Verification". This means clients that do not support Javascript such as mobile applications will not be prevented due to verification and is less intrusive. From K42323285: Overview of the unified Bot Defense profile https://my.f5.com/manage/s/article/K42323285 Challenge-Free Verification—The default value when Profile Template is set to Relaxed. The system performs header-based verification but does not perform JavaScript verification. The Balanced and Strict offers more stringent client verification. For the mitigation from Bot Defense to take effect, it should be in blocking mode. Relaxed Mode Defines a permissive security policy that performs basic non-intrusive verification of Browsers, strong verification of Mobile Apps using Anti-Bot Mobile Security SDK, blocks Malicious Bots and allows all other clients. Malicious Bots are detected mostly by using bot signatures. The mode provides basic protection level with very low risk of false positives. Balanced Mode Defines a moderate security policy that performs advanced verification of Browsers, strong verification of Mobile Apps using Anti-Bot Mobile Security SDK, blocks Malicious Bots, initiates CAPTCHA challenge for Suspicious Browsers, limits the total request rate produced by Unknown bots and allows Trusted and Untrusted Bots. Malicious Bots and Suspicious Browsers are identified by using both anomaly detection algorithms and bot signatures. This mode provides an advanced protection level with reduced latency impact because Browser verification is performed by injecting challenge in HTTP response. Strict Mode Defines a strict security policy that performs advanced verification of Browsers, strong verification of Mobile Apps using Anti-Bot Mobile Security SDK, and blocks all bots except Trusted Bots. This mode provides the most advanced and strict protection level using all capabilities of Bot Defense. Browser clients are not allowed to access unless they pass proactive verification. Mobile clients security access requires the use of Anti-Bot Mobile SDK. Here is a sample log for Bot Defense it is detecting a client that is classified as a "Suspicious Browser". The Bot Defense Profile that detected this bot request was configured with the "Relaxed" profile template. Here are sample Bot Traffic detected and actioned by a Bot Defense profile. Notice the Alarm and Block events. A detected "Suspicious Browser" is not blocked but generates an Alarm. A "malicious bot" is blocked DoS Protection Profile For the DoS Protection profiles, detection and mitigation can be configured thru TPS , Behavioral and Stress-based. The threshold for each configuration can be configured with Manual or Automatic thresholds. Manually configured TPS based detection looks at defined conditions and thresholds by Source IP, Device ID, Geolocation, URL and Site Wide and when exceeded, configured mitigation will take effect. The DoS Protection profile should be in Blocking mode for the mitigation to take effect. When using automatic threshold configuration in BIG-IP ASM/Adv WAF DoS Protection profile, the system sets the values using a wide range to begin with, then calculates the values using 7 days of historical data and sets threshold values to the highest levels during normal activity (to minimize false positives). Reference: K000138529: Understanding Automatic Threshold in BIG-IP ASM/Adv WAF DoS Protection profile The by Device ID detection option uses JavaScript to detect clients and requires a Bot Defense profile with Device ID mode to be configured. Here is the description of conditions from the TPS based detection configuration ================== Consider an IP as an attacking entity if either of the following conditions occur: Relative Threshold: TPS increased by: <traffic percentage> and reached at least <TPS> transactions per second OR Absolute Threshold: TPS reached: <TPS> transactions per second ================== Here is the description of conditions when TPS based detection is configured with Automatic Threshold ============== Consider an IP as an attacking entity if TPS reached an auto-calculated threshold in range <minimum TPS> - <upper limit TPS> transactions per second ============== For environments that have a mix of web and mobile application clients, only the Request Blocking option does not use Javascript to mitigate attacking clients. The Behavioral and Stress-based detection threshold can also be configured as Manual or Automatic and have the same configuration options TPS based detection. The difference is Server Stress is a requirement in this detection mode. Ensure that the DoS protection profile is in blocking mode. From the Online Help ============ Behavioral & Stress-based Detection In this area you can configure the system to prevent DoS attacks based on the server’s health condition. An attack is detected if the system finds the server to be under stress and either of the TPS thresholds are crossed. =========== Similar mitigation options are also available. Another section of Behavioral & Stress-based Detection is the Behavioral Detection and Mitigation where Bad Actor detection and mitigation can be configured From the Online Help ========= Bad actors behavior detection: Enables traffic behavior, server's capacity learning, and anomaly detection. Request signatures detection: Enables signatures detection. Use TLS fingerprints identification: Allows the system to distinguish between bad and good actors behind the same IP (NAT). When disabled (default), any attack behind the NAT treats all users behind the NAT as attackers. ========= Do take note of Request signatures detection and Use TLS fingerprints identification as these options are useful in identifying attacking clients behind a NAT device by way of TLS fingerprint and clients with specific HTTP signatures. Here is the Bad actors behavior detection mitigation descriptions. Review the protection levels and use what is appropriate as per your needs. Mitigation: No mitigation:Learns and monitors traffic behavior, but no action is taken. Conservative protection:If «Bad actors detection» enabled, slows down and rate limits requests from anomalous IP addresses based on its anomaly detection confidence and the server's health. If «Request signatures detection» enabled, blocks requests that match the attack signatures. Standard protection:If «Bad actors detection» enabled, slows down requests from anomalous IP addresses based on its anomaly detection confidence and the server's health. Rate limits requests from anomalous IP addresses and, if necessary, rate limits all requests based on the servers health. Limits the number of concurrent connections from anomalous IP addresses and, if necessary, limits the number of all concurrent connections based on the server's health. If «Request signatures detection» enabled, blocks requests that match the attack signatures. Aggressive protection:If «Bad actors detection» enabled, slows down requests from anomalous IP addresses based on its anomaly detection confidence and the server's health. Rate limits requests from anomalous IP addresses and, if necessary, rate limits all requests based on the servers health. Limits the number of concurrent connections from anomalous IP addresses and, if necessary, limits the number of all concurrent connections based on the server's health. Proactively performs all protection actions (even before an attack). Increases the impact of the protection techniques. If «Request signatures detection» enabled, blocks requests that match the attack signatures. Increases the impact of blocked requests. Regarding Server Stress There are a couple of locations where Server Stress can be observed. A spike in Server Stress will trigger Behavioral and Stress-based DoS detection and mitigation if configured. Protected Objects List This is under Security ›› DoS Protection : Protected Objects : Protected Objects List This menu is available when you have BIG-IP AFM provisioned in this sample, notice the Server Stress is at 100/100 - this means the backend server is Stressed and the latency of Server response is High. The protected object's attack status is "red" signaling an attack is ongoing and being mitigated. On this sample, the detected attack is ongoing and the server stress value has gone down to 55/100. The request rate also shows the connections per second and has gone down. Behavioral DoS Dashboard Under the Statistics menu, click on Dashboard Select Behavioral DoS in the Dashboard options Review Server Stress Here are sample DoS Application Events generated thru a DoS protection profile TPS Based detection and mitigation Behavioral detection and mitigation It provides insight which DoS "Threshold condition" was exceeded and what "Mitigation" was applied to the detected "Attack (Attack ID)" also noting the start and end of the attack. Inspecting these events will help in figuring out a threshold you may apply. DoS Dashboard under Security Reporting When looking for details of the attack, the DoS Dashboard under Security Reporting provides insight on the detected attack and related entities and statistics. In this sample, the detected attack was triggered thru App Behavioral and mitigated with Behavioral mitigation In this sample, notice the "transaction outcomes" shows "Blocked Bad Actor" which means the transactions were blocked by Bad Actor detection and mitigation configuration in the DoS protection profile. IP reputation , IP Intelligence license - must have! During DDoS attacks, it is very likely that the some of the source IP addresses will have bad IP reputation. Having the IP Intelligence license available in the BIG-IP will provide mitigation for matched IP addresses. IP Intelligence can be used in BIG-IP LTM (iRules and LTM Policy), AFM (IP Intelligence Policy) and ASM/Adv WAF (Security Policy) configurations. Refer to the following links on sample iRule configurations. https://clouddocs.f5.com/api/irules/IP__reputation.html https://clouddocs.f5.com/api/irules/IP__intelligence.html Note that the IP intelligence feature requires an add-on license.Contact your F5 or Partner salesperson for details on ordering the license. Cyber Security Attack Scenarios and Recommendations As introduced earlier, here are Cyber Security Attack Scenarios that BIG-IP deployments encounter and corresponding mitigation options and configuration recommendations. These sample scenarios will be helpful in finding initial mitigation options using BIG-IP features should it match your deployment configuration. Scenario: DDoS attack on a F5 Virtual Server fronting a web application and the BIG-IP have ASM/Adv WAF license. There are no DoS protection and Bot Defense profile configured on the Virtual Server. Recommendations: For the DoS Protection profile, configure TPS and Behavioral and Stress based Detection. Set Detection Thresholds and mitigations for both options as per your needs. You can monitor the traffic pattern on the Virtual Server you are protecting. Consider the clients that access the web application. If the clients are web browsers and mobile application users, use non Javascript (JS) based detection and mitigations. This will allow mobile application clients that do not support Javascript challenges to access the protected web application and ensure they are not blocked by Javascript challenges or mitigations. Another detection method is thru Bad Actor detection configuration under Behavioral and Stress Based Detection menu. This feature slows down and rate limits requests from anomalous IP addresses based on its anomaly detection confidence and the server's health. Automatic Threshold can also be configured for TPS based detection. For more information on HTTP enabled DoS Protection configured with Automatic Threshold, refer to K000138529: Understanding Automatic Threshold in BIG-IP ASM/Adv WAF DoS Protection profile For the Bot Defense profile, configure the Relaxed Profile template. This does not use Javascript challenges for detecting end clients and will allow mobile application clients that do not support Javascript to access the protected web application. Configure a logging profile that logs remotely to trusted logging server for DoS protection and Bot Defense events. Scenario: DDoS attack on a F5 Forwarding IP or Performance Layer 4 Virtual Server processing network traffic. Recommendations: Ensure DoS mitigations are available to protect the network traffic. BIG-IP AFM have DoS Attack types for Network Based DoS and can be set to Mitigate to detect and mitigate/drop excess packets for the matched attack type. These DoS Attack types can be configured with Fully Manual or Fully Automatic threshold. You should set detection and mitigation thresholds as per your needs. It is important to review traffic statistics and pattern on the Virtual Server you are protecting. AFM threshold values are in EPS - Events Per Second, synonymous to Packets Per Second and the observed traffic pattern can be the basis of the manually configured thresholds. Another method of defining the threshold is thru "Threshold Sensitivity" where the BIG-IP system CPU usage and traffic pattern is the basis. When AFM DoS attack types are configured with Fully Automatic, it uses the "Threshold Sensitivity" configuration. For BIG-IP Advanced Firewall Manager (AFM) systems protecting networks against Distributed Denial of Service (DDoS) attacks, DoS Auto Threshold sensitivity can be configured for system-wide thru Device Protection and per DoS Protection Profile. A setting of High will be more sensitive to changes in BIG-IP System CPU usage and traffic. A setting of Medium is the default configuration. A setting of Low will be less sensitive to changes in BIG-IP System CPU usage and traffic. From the BIG-IP Configuration utility, navigate to: For System Wide: Security ›› DoS Protection : Device Protection per DoS Protection Profile: Security ›› DoS Protection : Protection Profiles .. select the specific profile Reference: K000141430: Configuring BIG-IP AFM DoS Protection Threshold Sensitivity https://my.f5.com/manage/s/article/K000141430 Scenario: DNS DDoS attack on F5 DNS listener Virtual Server Recommendations: BIG-IP AFM have DNS Attack types to detect and mitigate DNS DDoS attacks. If your BIG-IP does not have BIG-IP AFM licensed, it would be beneficial for the DNS service processed thru the F5 DNS listener VS to have mitigation options available from the DNS Attack types in the AFM DoS device or DNS enabled protection profile. Configure thresholds as per your needs. Here is a sample configuration from a lab device where DNS A Query Attack type in Device Protection is configured in Mitigate state and Fully Manual detection and mitigation thresholds. Bad Actor Detection can also be configured with thresholds. It is also possible to configure it for Fully Automatic threshold and will be dependent on the Threshold Sensitivity configuration. Scenario: DDoS attack on F5 Virtual Server and attacking IP addresses needs to be blocked Recommendations: BIG-IP AFM has Network Firewall Policy and rules where it can be configured to match a source address list and drop its traffic. This can also be done thru iRules, however, AFM firewall rule configuration is a native feature and is built for such operations. The F5 SIRT created a playbook for HTTP brute force mitigation and the LTM mitigation options includes such configurations should you decide to use iRules and BIG-IP LTM features. HTTP Brute Force Mitigation Playbook: BIG-IP LTM Mitigation Options for HTTP Brute Force Attacks - Chapter 3 HTTP Brute Force Mitigation Playbook: BIG-IP LTM Mitigation Options for HTTP Brute Force Attacks - Chapter 3 | DevCentral F5 also have K30534815: Attack mitigation matrix using F5 security products and services which lists existing F5 Support articles for attack mitigation. https://my.f5.com/manage/s/article/K30534815 During DDoS attacks, it is very likely that the some of the source IP addresses will have bad IP reputation, it will be beneficial to have IP Intelligence license available and use it in BIG-IP LTM (iRules and LTM Policy), AFM (IP Intelligence Policy) and ASM/Adv WAF (Security Policy) configurations. Refer to the following links on sample iRule configurations. https://clouddocs.f5.com/api/irules/IP__reputation.html https://clouddocs.f5.com/api/irules/IP__intelligence.html Note that the IP intelligence feature requires an add-on license.Contact your F5 or Partner salesperson for details on ordering the license. Scenario: BIG-IP device is suspected to be compromised. Recommendations: BIG-IP can be affected by malware and it finds it's way to it by exposing the BIG-IP management and self (self-IP) IP addresses and configured with insecure or easy to guess BIG-IP user password. BIG-IP product had previous Critical CVEs where authentication was not needed to exploit the vulnerability. F5 has the article K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system to provide guidance on handling suspected compromised BIG-IP devices. https://my.f5.com/manage/s/article/K11438344 To minimize the attack surface of a BIG-IP device against CVEs and unauthorized access, ensure that only trusted authenticated users and networks have access to the BIG-IP management and self (self-IP) IP addresses and the BIG-IP device is running the latest BIG-IP software version. Do review the "Major Release and Long-Term Stability Release versions supported with active software development" of K5903: BIG-IP software support policy as it notes BIG-IP 15.1.x version reaches "End of Technical Support" this December 31, 2024. Note: For BIG-IP Next (BIG-IP 20.x and later), refer toBIG-IP Next software support policy. Major Release and Long-Term Stability Release versions First customer ship End of Software Development End of Technical Support Latest maintenance release 17.1.x March 14, 2023 March 31, 2027 March 31, 2027 17.1.1 16.1.x July 7, 2021 July 31, 2025 July 31, 2025 16.1.5 15.1.x December 11, 2019 December 31, 2024 December 31, 2024 15.1.10 K5903: BIG-IP software support policy https://my.f5.com/manage/s/article/K5903 F5 Distributed Cloud "F5 Distributed Cloud Services are SaaS-based security, networking, and application management services that enable customers to deploy, secure, and operate their applications in a cloud-native environment wherever needed–data center, multi-cloud, or the network or enterprise edge." https://www.f5.com/products/distributed-cloud-services F5 Distributed Cloud have many mitigation options for DDoS attacks. Volumetric DDoS can be handled by F5 Distributed Cloud DDoS Mitigation Service. F5 Distributed Cloud Bot Defense mitigates complex bot attacks. It provides integration option with your mobile application so it can be properly classified and detected as a trusted client. see Making Mobile SDK Integration Ridiculously Easy with F5 XC Mobile SDK Integrator https://www.f5.com/products/distributed-cloud-services/bot-defense https://www.f5.com/products/distributed-cloud-services/l3-and-l7-ddos-attack-mitigation Conclusion The Cyber Security Attack Scenarios and recommendations using BIG-IP features shared are not exhaustive. There are more complex environments and scenarios that BIG-IP deployments may have opportunity to mitigate network and application attack traffic and it is important that the appropriate BIG-IP licenses are available so the relevant modules can be provisioned and related features and configuration can be enabled. This will help your environment's network and application traffic to be resilient against DDoS attacks when BIG-IP is properly licensed, positioned and configured. I hope the sample configuration, logs and configuration help details have been useful and helpful as you consider the BIG-IP features mitigation options and improve your network, application and BIG-IP device security posture. Until next time! The F5 SIRT creates security-related content posted here in DevCentral, sharing the team’s security mindset and knowledge. Feel free to view the articles that are tagged with the following: F5 SIRT series-F5SIRT-this-week-in-security TWISBIG-IP BGP Routing Protocol Configuration And Use Cases
Is the F5 BIG-IP a router? Yes! No! Wait what? Can the BIG-IP run a routing protocol? Yes. But should it be deployed as a core router? An edge router? Stay tuned. We'll explore these questions and more through a series of common use cases using BGP on the BIG-IP... And oddly I just realized how close in typing BGP and BIG-IP are, so hopefully my editors will keep me honest. (squirrel!) In part one we will explore therouting components on the BIG-IP and some basic configuration details to help you understand what the appliance is capable of. Please pay special attention to some of the gotchas along the way. Can I Haz BGP? Ok. So your BIG-IP comes with ZebOS in order to provide routing functionality, but what happens when you turn it on? What do you need to do to get routing updates in to the BGP process? And well does my licensing cover it? Starting with the last question… tmsh show /sys license | grep "Routing Bundle" The above command will help you determine if you’re going to be able to proceed, or be stymied at the bridge like the Black Knight in the Holy Grail. Fear not! There are many licensing options that already come with the routing bundle. Enabling Routing First and foremost, the routing protocol configuration is tied to the route-domain. What’s a route-domain? I’m so glad you asked! Route-domains are separate Layer 3 route tables within the BIG-IP. There is a concept of parent and child route domains, so while they’re similar to another routing concept you may be familiar with; VRF’s, they’re no t quite the same but in many ways they are. Just think of them this way for now. For this context we will just say they are. Therefore, you can enable routing protocols on the individual route-domains. Each route-domain can have it’s own set of routing protocols. Or run no routing protocols at all. By default the BIG-IP starts with just route-domain 0. And well because most router guys live on the cli, we’ll walk through the configuration examples that way on the BIG-IP. tmsh modify net route-domain 0 routing-protocol add { BGP } So great! Now we’re off and running BGP. So the world know’s we’re here right? Nope. Considering what you want to advertise. The most common advertisements sourced from the BIG-IP are the IP addresses for virtual servers. Now why would I want to do that? I can just put the BIG-IP on a large subnet and it will respond to ARP requests and send gratuitous ARPs (GARP). So that I can reach the virtual servers just fine. <rant> Author's opinion here: I consider this one of the worst BIG-IP implementation methods. Why? Well for starters, what if you want to expand the number of virtual servers on the BIG-IP? Well then you need to re-IP the network interfaces of all the devices (routers, firewalls, servers) in order to expand the subnet mask. Yuck! Don't even talk to me about secondary subnets. Second: ARP floods! Too many times I see issues where the BIG-IP has to send a flood of GARPs; and well the infrastructure, in an attempt to protect its control plane, filters/rate limits the number of incoming requests it will accept. So engineers are left to try and troubleshoot the case of the missing GARPs Third: Sometimes you need to migrate applications to maybe another BIG-IP appliance as it grew to big for the existing infrastructure. Having it tied to this interface just leads to confusion. I'm sure there's some corner cases where this is the best route. But I would say it's probably in the minority. </rant> I can hear you all now… “So what do you propose kind sir?” See? I can hear you... Treat the virtual servers as loopback interfaces. Then they’re not tied to a specific interface. To move them you just need to start advertising the /32 from another spot (Yes. You could statically route it too. I hear you out there wanting to show your routing chops.) But also, the only GARPs are those from the self-ip's This allows you to statically route of course the entire /24 to the BIG-IP’s self IP address, but also you can use one of them fancy routing protocols to announce the routes either individually or through a summarization. Announcing Routes Hear ye hear ye! I want the world to know about my virtual servers.*ahem* So quick little tangent on BIG-IP nomenclature. The virtual server does not get announced in the routing protocol. “Well then what does?” Eery mind reading isn't it? Remember from BIG-IP 101, a virtual server is an IP address and port combination and well, routing protocols don’t do well with carrying the port across our network. So what BIG-IP object is solely an IP address construct? The virtual-address! “Wait what?” Yeah… It’s a menu item I often forget is there too. But here’s where you let the BIG-IP know you want to advertise the virtual-address associated with the virtual server. But… but… but… you can have multiple virtual servers tied to a single IP address (http/https/etc.) and that’s where the choices for when to advertise come in. tmsh modify ltm virtual-address 10.99.99.100 route-advertisement all There are four states a virtual address can be in: Unknown, Enabled, Disabled and Offline. When the virtual address is in Unknown or Enabled state, its route will be added to the kernel routing table. When the virtual address is in Disabled or Offline state, its route will be removed if present and will not be added if not already present. But the best part is, you can use this to only advertise the route when the virtual server and it’s associated pool members are all up and functioning. In simple terms we call this route health injection. Based on the health of the application we will conditionally announce the route in to the routing protocol. At this point, if you’d followed me this far you’re probably asking what controls those conditions. I’ll let the K article expand on the options a bit. https://my.f5.com/manage/s/article/K15923612 “So what does BGP have to do with popcorn?” Popcorn? Ohhhhhhhhhhh….. kernel! I see what you did there! I’m talking about the operating system kernel silly. So when a virtual-address is in an unknown or enabled state and it is healthy, the route gets put in the kernel routing table. But that doesn’t get it in to the BGP process. Here is how the kernel (are we getting hungry?) routes are represented in the routing table with a 'K' This is where the fun begins! You guessed it! Route redistribution? Route redistribution! And well to take a step back I guess we need to get you to the ZebOS interface. To enter the router configuration cli from the bash command line, simply type imish. In a multi-route-domain configuration you would need to supply the route-domain number but in this case since we’re just using the 0 default we’re good. It’s a very similar interface to many vendor’s router and switch configuration so many of you CCIE’s should feel right at home. It even still lets you do a write memoryor wr memwithout having to create an alias. Clearly dating myself here.. I’m not going to get in to the full BGP configuration at this point but the simplest way to get the kernel routes in to the BGP process is simply going under the BGP process and redisitrubting the kernel routes. BUT WAIT! Thar be dragons in that configuration! First landmine and a note about kernel routes. If you manually configure a static route on the BIG-IP via tmsh or the tmui those will show up also as kernel routes Why is that concerning? Well an example is where engineers configure a static default route on the BIG-IP via tmsh. And well, when you redistribute kernel routes and that default route is now being advertised into BGP. Congrats! AND the BIG-IP is NOT your default gateway hilarity ensues. And by hilarity I mean the type of laugh that comes out as you're updating your resume. The lesson here is ALWAYS when doing route redistribution always use a route filter to ensure only your intended routes or IP range make it in to the routing protocol. This goes for your neighbor statements too. In both directions! You should control what routes come in and leave the device. Another way to have some disasterous consequences with BIG-IP routing is through summarization. If you are doing summarization, keep in mind that BGP advertises based on reachability to the networks it wants to advertise. In this case, BGP is receiving it in the form of kernel routes from tmm. But those are /32 addresses and lots of them! And you want to advertise a /23 summary route. But the lone virtual-address that is configured for route advertisement; and the only one your BGP process knows about within that range has a monitor that fails. The summary route will be withdrawn leaving all the /23 stranded. Be sure to configure all your virtual-addresses within that range for advertisement. Next: BGP Behavior In High Availability ConfigurationsF5 NGINX HTTP Request Header Rules: What’s Permitted and What’s Not
When managing web servers like F5 NGINX, it's crucial to understand the rules that govern HTTP headers—particularly which characters are allowed or disallowed in both header names and values. HTTP headers play a vital role in the communication between a client and server, carrying essential metadata such as content type, length, and caching policies. NGINX strictly enforces these rules to ensure compliance with HTTP/1.1, HTTP/2, and HTTP/3 protocols. Misconfigured headers or the use of improper characters can lead to a range of issues, including security vulnerabilities, degraded performance, or even rejected requests. One common security threat,HTTP Request Smuggling, exploits desynchronization between devices involved in handling the request, such as frontends, caching servers, load balancers, and web servers. These attacks rely on inconsistencies in how each device parses the request, enabling malicious actors to inject hidden or split requests. This article outlines the allowed and disallowed characters in HTTP request headers as enforced by NGINX, which I compiled throughcode review, test case review and manual testing for HTTP Smugglingon NGINX version "nginx/1.25.5 (nginx-plus-r32-p1)". If I missed something please let me know bycontacting the F5 SIRT. Allowed Characters Header Names Lowercase letters: (a-z) Uppercase letters: (A-Z) Allowed in HTTP/1.1 Disallowed in HTTP/2 and HTTP/3, which require lowercase. Digits: (0-9) Hyphen: (-) Underscore: (_)Allowed in all versions only ifunderscores_in_headers is enabled Colon: (:) Allowed only as a prefix inHTTP/2 and HTTP/3 pseudo headers. Header Values Printable ASCII characters: Characters from ! to ~, including digits, letters, punctuation, and symbols, are allowed. Space: (0x20) Allowed within the value. Horizontal Tab (HT): (0x09) Generally allowed within values but with exceptions in headers such as Content-Length and Host (details below). Disallowed Characters Header Names Control Characters: ASCII control characters (<= 0x20), including space and horizontal tab, are disallowed in header names. DEL (0x7f): The delete character is not allowed. Colon: (:) Disallowed in header names except forHTTP/2 and HTTP/3 pseudo headers. Uppercase Letters: (A-Z) Disallowed in HTTP/2 and HTTP/3 header names (must be lowercase). Special characters: Characters such as ()<>@,;:\"/[]?={} are implicitly disallowed as they don’t belong to any allowed category for header names. Header Values Null Character: (\0) Disallowed in header values. Line Feed (LF): (0x0A) Not allowed within the value and used only to terminate headers. Carriage Return (CR): (0x0D) Not allowed within the value, used for header termination. Control Characters (<= 0x20) with special conditions for Horizontal Tab (HT): Horizontal Tab (HT, 0x09): Disallowed in the Host header value in HTTP/1. Disallowed in the pseudo-header values in HTTP/2 and HTTP/3. Disallowed in the Content-Length header value in HTTP/1. Summary This list of allowed and disallowed characters provides an overview of how NGINX handles HTTP headers across different protocol versions. While HTTP/1.1 is somewhat lenient with uppercase letters and certain characters, HTTP/2 and HTTP/3 enforce stricter validation rules, particularly for header names. Understanding these restrictions ensures your server configuration remains compliant with protocol specifications and avoids potential issues with malformed headers.Ridiculously Easy Bot Protection: How to Use BIG-IP APM to Streamline Bot Defense Implementation
Ever imagined how your Bot solution implementation would be with a standard entry page at your application side--a page that’s easily referred, with clear parameters, and structured customization options? In this article, we are exploring using F5 BIG-IP Access Policy Manager (BIG-IP APM) along side F5 Distributed Cloud Bot Defense (XC Bot Defense). Bot defense solutions' challenges Implementing bot defense solutions presents several challenges, each with unique considerations: Evolving Bot Tactics: Bot tactics constantly evolve, demanding adaptive detection methods to avoid both false positives (blocking legitimate users) and false negatives (allowing malicious bots through). Effective solutions must be highly flexible and responsive to these changes. Multi-Environment Integration: Bot defenses need to be deployed across diverse environments, including web, mobile, and APIs, adding layers of complexity to integration. Ensuring seamless protection across these platforms is critical. Balancing Security and Performance: Security measures must be balanced with performance to avoid degrading the user experience. A well-calibrated bot defense should secure the application without causing noticeable slowdowns or other disruptions for legitimate users. Data Privacy Compliance: Bot solutions often require extensive data collection, so adherence to data privacy laws is essential. Ensuring that bot defense practices align with regulatory standards helps avoid legal complications and maintains user trust. Resource Demands: Integrating bot defense with existing security stacks can be resource-intensive, both in terms of cost and skilled personnel. Proper configuration, monitoring, and maintenance require dedicated resources to ensure long-term effectiveness and efficiency. What F5 BIG-IP APM brings to the table? For teams working on bot defense solutions, several operational challenges can arise: Targeted Implementation Complexity: Identifying the correct application page for applying bot defense is often a complex process. Teams must ensure the solution targets the page containing the specific parameters they want to protect, which can be time-consuming and resource-intensive. Adaptation to Application Changes: Changes like upgrades or redesigns of the application page often require adjustments to bot defenses. These modifications can translate into significant resource commitments, as teams work to ensure the bot solution remains aligned with the new page structure. BIG-IP APM simplifies this process by making it easier to identify and target the correct page, reducing the time and resources needed for implementation. This allows technical and business resources to focus on more strategic priorities, such as fine-tuning bot defenses, optimizing protection, and enhancing application performance. Architecture and traffic flow In this section, let's explore how F5 XC Bot Defense and BIG-IP APM works together, let's list the prerequisites, F5 XC account with access to Bot Defense. APM licensed and provisioned. F5 BIG-IP min. v16.x for native connector integration. BIG-IP Self IP rechability to Internet to communicate with F5 XC, mainly to reach this domin (ibd-web.fastcache.net). Now, time to go quickly through our beloved TMM packet order. Due to the nature of BIG-IP APM Access events take precedence to the Bot enforcement, hence we will rely on simple iRule to apply Bot Defense on BIG-IP APM logon page. BIG-IP Bot Defense is responsible for inserting the JS and passing traffic from client to APM VS back and forth. BIG-IP APM responsible for logon page, MFA, API security or SSO integrations to manage client Access to the backend application. Solution Implementation Let's start now with our solution implementation, F5 Distributed Cloud Bot defense connector with BIG-IP was discussed in details in this Article F5 Distributed Cloud Bot Defense on BIG-IP 17.1 You will follow the steps mentioned in the article, with few changes mentioned below, API Hostname Web: ibd-web.fastcache.net For Per-session policies we use /my.policy as the target URL, while for Per-request and MFA implementation, you need to add /vdesk/*. Protection Pool - Web: Create pool with FQDN ibd-web.fastcache.net Virtual server, Create LTM virtual server to listen to incoming traffic, perform SSL offloading, HTTP profile and attach Bot Defense connector profile. Forwarding iRule, attach forwarding irule to the Bot virtual server. when CLIENT_ACCEPTED { ## Forwarding to the APM Virtual Server virtual Auth_VS } BIG-IP APM Policies,In this step we are creating two options of our deployment, Per-Session policy, where BIG-IP presents Logon page to the user. Per-Request policy, which services in case initial logon is handled at remote IdP and APM handles Per-request, MFA authentication or API security. Now, it's time to run the traffic and observe the results, From client browser, we can see the customer1.js inserted, From F5 XC Dashboard, Conclusion The primary goal of incorporating BIG-IP APM into the Bot Defense solution is to strike a balance between accelerating application development across web and mobile platforms while enforcing consistent organizational bot policies. By decoupling application login and authentication from the application itself, this approach enables a more streamlined, optimized, and secure bot defense implementation. It allows development teams to concentrate on application performance and feature enhancements, knowing that security measures are robustly managed and seamlessly integrated into the infrastructure. Related Content F5 Distributed Cloud Bot Defense on BIG-IP 17.1 Bot Detection and Security: Stop Automated Attacks 2024 Bad Bots ReviewAutomating ACMEv2 Certificate Management on BIG-IP
While we often associate and confuse Let's Encrypt with ACMEv2, the former is ultimately a consumer of the latter. The "Automated Certificate Management Environment" (ACME) protocol describes a system for automating the renewal of PKI certificates. The ACME protocol can be used with public services like Let's Encrypt, but also with internal certificate management services. In this article we explore the more generic support of ACME (version 2) on the F5 BIG-IP.
APM Configuration to Support Duo MFA using iRule
Overview BIG-IP APM has supported Duo as an MFA provider for a long time with RADIUS-based integration. Recently, Duo has added support for Universal Prompt that uses Open ID Connect (OIDC) protocol to provide two-factor authentication. To integrate APM as an OIDC client and resource server, and Duo as an Identity Provider (IdP), Duo requires the user’s logon name and custom parameters to be sent for Authentication and Token request. This guide describes the configuration required on APM to enable Duo MFA integration using an iRule. iRules addresses the custom parameter challenges by generating the needed custom values and saving them in session variables, which the OAuth Client agent then uses to perform MFA with Duo. This integration procedure is supported on BIG-IP versions 13.1, 14.1x, 15.1x, and 16.x. To integrate Duo MFA with APM, complete the following tasks: 1. Choose deployment type: Per-request or Per-session 2. Configure credentials and policies for MFA on the DUO web portal 3. Create OAuth objects on the BIG-IP system 4. Configure the iRule 5. Create the appropriate access policy/policies on the BIG-IP system 6. Apply policy/policies and iRule to the APM virtual server Choose deployment type APM supports two different types of policies for performing authentication functions. Per-session policies: Per-session policies provide authentication and authorization functions that occur only at the beginning of a user’s session. These policies are compatible with most APM use cases such as VPN, Webtop portal, Remote Desktop, federation IdP, etc. Per-request policies: Per-request policies provide dynamic authentication and authorization functionality that may occur at any time during a user’s session, such as step-up authentication or auditing functions only for certain resources. These policies are only compatible with Identity Aware Proxy and Web Access Management use cases and cannot be used with VPN or webtop portals. This guide contains information about setting up both policy types. Prerequisites Ensure the BIG-IP system has DNS and internet connectivity to contact Duo directly for validating the user's OAuth tokens. Configure credentials and policies for MFA on Duo web portal Before you can protect your F5 BIG-IP APM Web application with Duo, you will first need to sign up for a Duo account. 1. Log in to the Duo Admin Panel and navigate to Applications. 2. Click Protect an application. Figure 1: Duo Admin Panel – Protect an Application 3. Locate the entry for F5 BIG-IP APM Web in the applications list and click Protect to get the Client ID, Client secret, and API hostname. You will need this information to configure objects on APM. Figure 2: Duo Admin Panel – F5 BIG-IP APM Web 4. As DUO is used as a secondary authentication factor, the user’s logon name is sent along with the authentication request. Depending on your security policy, you may want to pre-provision users in Duo, or you may allow them to self-provision to set their preferred authentication type when they first log on. To add users to the Duo system, navigate to the Dashboard page and click the Add New...-> Add User button. A Duo username should match the user's primary authentication username. Refer to the https://duo.com/docs/enrolling-users link for the different methods of user enrollment. Refer to Duo Universal Prompt for additional information on Duo’s two-factor authentication. Create OAuth objects on the BIG-IP system Create a JSON web key When APM is configured to act as an OAuth client or resource server, it uses JSON web keys (JWKs) to validate the JSON web tokens it receives from Duo. To create a JSON web key: 1. On the Main tab, select Access > Federation > JSON Web Token > Key Configuration. The Key Configuration screen opens. 2. To add a new key configuration, click Create. 3. In the ID and Shared Secret fields, enter the Client ID and Client Secret values respectively obtained from Duo when protecting the application. 4. In the Type list, select the cryptographic algorithm used to sign the JSON web key. Figure 3: Key Configuration screen 5. Click Save. Create a JSON web token As an OAuth client or resource server, APM validates the JSON web tokens (JWT) it receives from Duo. To create a JSON web token: 1. On the Main tab, select Access > Federation > JSON Web Token > Token Configuration. The Token Configuration screen opens. 2. To add a new token configuration, click Create. 3. In the Issuer field, enter the API hostname value obtained from Duo when protecting the application. 4. In the Signing Algorithms area, select from the Available list and populate the Allowed and Blocked lists. 5. In the Keys (JWK) area, select the previously configured JSON web key in the allowed list of keys. Figure 4: Token Configuration screen 6. Click Save. Configure Duo as an OAuth provider APM uses the OAuth provider settings to get URIs on the external OAuth authorization server for JWT web tokens. To configure an OAuth provider: 1. On the Main tab, select Access > Federation > OAuth Client / Resource Server > Provider. The Provider screen opens. 2. To add a provider, click Create. 3. In the Name field, type a name for the provider. 4. From the Type list, select Custom. 5. For Token Configuration (JWT), select a configuration from the list. 6. In the Authentication URI field, type the URI on the provider where APM should redirect the user for authentication. The hostname is the same as the API hostname in the Duo application. 7. In the Token URI field, type the URI on the provider where APM can get a token. The hostname is the same as the API hostname in the Duo application. Figure 5: OAuth Provider screen 8. Click Finished. Configure Duo server for APM The OAuth Server settings specify the OAuth provider and role that Access Policy Manager (APM) plays with that provider. It also sets the Client ID, Client Secret, and Client’s SSL certificates that APM uses to communicate with the provider. To configure a Duo server: 1. On the Main tab, select Access > Federation > OAuth Client / Resource Server > OAuth Server. The OAuth Server screen opens. 2. To add a server, click Create. 3. In the Name field, type a name for the Duo server. 4. From the Mode list, select how you want the APM to be configured. 5. From the Type list, select Custom. 6. From the OAuth Provider list, select the Duo provider. 7. From the DNS Resolver list, select a DNS resolver (or click the plus (+) icon, create a DNS resolver, and then select it). 8. In the Token Validation Interval field, type a number. In a per-request policy subroutine configured to validate the token, the subroutine repeats at this interval or the expiry time of the access token, whichever is shorter. 9. In the Client Settings area, paste the Client ID and Client secret you obtained from Duo when protecting the application. 10. From the Client's ServerSSL Profile Name, select a server SSL profile. Figure 6: OAuth Server screen 11. Click Finished. Configure an auth-redirect-request and a token-request Requests specify the HTTP method, parameters, and headers to use for the specific type of request. An auth-redirect-request tells Duo where to redirect the end-user, and a token-request accesses the authorization server for obtaining an access token. To configure an auth-redirect-request: 1. On the Main tab, select Access > Federation > OAuth Client / Resource Server > Request. The Request screen opens. 2. To add a request, click Create. 3. In the Name field, type a name for the request. 4. For the HTTP Method, select GET. 5. For the Type, select auth-redirect-request. 6. As shown in Figure 7, specify the list of GET parameters to be sent: request parameter with value depending on the type of policy For per-request policy: %{subsession.custom.jwt_duo} For per-session policy: %{session.custom.jwt_duo} client_id parameter with type client-id response_type parameter with type response-type Figure 7: Request screen with auth-redirect-request (Use “subsession.custom…” for Per-request or “session.custom…” for Per-session) 7. Click Finished. To configure a token-request: 1. On the Main tab, select Access > Federation > OAuth Client / Resource Server > Request. The Request screen opens. 2. To add a request, click Create. 3. In the Name field, type a name for the request. 4. For the HTTP Method, select POST. 5. For the Type, select token-request. 6. As shown in Figure 8, specify the list of POST parameters to be sent: client_assertion parameter with value depending on the type of policy For per-request policy: %{subsession.custom.jwt_duo_token} For per-session policy: %{session.custom.jwt_duo_token} client_assertion_type parameter with value urn:ietf:params:oauth:client-assertion-type:jwt-bearer grant_type parameter with type grant-type redirect_uri parameter with type redirect-uri Figure 8: Request screen with token-request (Use “subsession.custom…” for Per-request or “session.custom…” for Per-session) 7. Click Finished. Configure the iRule iRules gives you the ability to customize and manage your network traffic. Configure an iRule that creates the required sub-session variables and usernames for Duo integration. Note: This iRule has sections for both per-request and per-session policies and can be used for either type of deployment. To configure an iRule: 1. On the Main tab, click Local Traffic > iRules. 2. To create an iRules, click Create. 3. In the Name field, type a name for the iRule. 4. Copy the sample code given below and paste it in the Definition field. Replace the following variables with values specific to the Duo application: <Duo Client ID> in the getClientId function with Duo Application ID. <Duo API Hostname> in the createJwtToken function with API Hostname. For example, https://api-duohostname.com/oauth/v1/token. <JSON Web Key> in the getJwkName function with the configured JSON web key. Note: The iRule ID here is set as JWT_CREATE. You can rename the ID as desired. You specify this ID in the iRule Event agent in Visual Policy Editor. Note: The variables used in the below example are global, which may affect your performance. Refer to the K95240202: Understanding iRule variable scope article for further information on global variables, and determine if you use a local variable for your implementation. proc randAZazStr {len} { return [subst [string repeat {[format %c [expr {int(rand() * 26) + (rand() > .5 ? 97 : 65)}]]} $len]] } proc getClientId { return <Duo Client ID> } proc getExpiryTime { set exp [clock seconds] set exp [expr $exp + 900] return $exp } proc getJwtHeader { return "{\"alg\":\"HS512\",\"typ\":\"JWT\"}" } proc getJwkName { return <JSON Web Key> #e.g. return "/Common/duo_jwk" } proc createJwt {duo_uname} { set header [call getJwtHeader] set exp [call getExpiryTime] set client_id [call getClientId] set redirect_uri "https://" set redirect [ACCESS::session data get "session.server.network.name"] append redirect_uri $redirect append redirect_uri "/oauth/client/redirect" set payload "{\"response_type\": \"code\",\"scope\":\"openid\",\"exp\":${exp},\"client_id\":\"${client_id}\",\"redirect_uri\":\"${redirect_uri}\",\"duo_uname\":\"${duo_uname}\"}" set jwt_duo [ ACCESS::oauth sign -header $header -payload $payload -alg HS512 -key [call getJwkName] ] return $jwt_duo } proc createJwtToken { set header [call getJwtHeader] set exp [call getExpiryTime] set client_id [call getClientId] set aud "<Duo API Hostname>/oauth/v1/token" #Example: set aud https://api-duohostname.com/oauth/v1/token set jti [call randAZazStr 32] set payload "{\"sub\": \"${client_id}\",\"iss\":\"${client_id}\",\"aud\":\"${aud}\",\"exp\":${exp},\"jti\":\"${jti}\"}" set jwt_duo [ ACCESS::oauth sign -header $header -payload $payload -alg HS512 -key [call getJwkName] ] return $jwt_duo } when ACCESS_POLICY_AGENT_EVENT { set irname [ACCESS::policy agent_id] if { $irname eq "JWT_CREATE" } { set ::duo_uname [ACCESS::session data get "session.logon.last.username"] ACCESS::session data set session.custom.jwt_duo [call createJwt $::duo_uname] ACCESS::session data set session.custom.jwt_duo_token [call createJwtToken] } } when ACCESS_PER_REQUEST_AGENT_EVENT { set irname [ACCESS::perflow get perflow.irule_agent_id] if { $irname eq "JWT_CREATE" } { set ::duo_uname [ACCESS::session data get "session.logon.last.username"] ACCESS::perflow set perflow.custom [call createJwt $::duo_uname] ACCESS::perflow set perflow.scratchpad [call createJwtToken] } } Figure 9: iRule screen 5. Click Finished. Create the appropriate access policy/policies on the BIG-IP system Per-request policy Skip this section for a per-session type deployment The per-request policy is used to perform secondary authentication with Duo. Configure the access policies through the access menu, using the Visual Policy Editor. The per-request access policy must have a subroutine with an iRule Event, Variable Assign, and an OAuth Client agent that requests authorization and tokens from an OAuth server. You may use other per-request policy items such as URL branching or Client Type to call Duo only for certain target URIs. Figure 10 shows a subroutine named duosubroutine in the per-request policy that handles Duo MFA authentication. Figure 10: Per-request policy in Visual Policy Editor Configuring the iRule Event agent The iRule Event agent specifies the iRule ID to be executed for Duo integration. In the ID field, type the iRule ID as configured in the iRule. Figure 11: iRule Event agent in Visual Policy Editor Configuring the Variable Assign agent The Variable Assign agent specifies the variables for token and redirect requests and assigns a value for Duo MFA in a subroutine. This is required only for per-request type deployment. Add sub-session variables as custom variables and assign their custom Tcl expressions as shown in Figure 12. subsession.custom.jwt_duo_token = return [mcget {perflow.scratchpad}] subsession.custom.jwt_duo = return [mcget {perflow.custom}] Figure 12: Variable Assign agent in Visual Policy Editor Configuring the OAuth Client agent An OAuth Client agent requests authorization and tokens from the Duo server. Specify OAuth parameters as shown in Figure 13. In the Server list, select the Duo server to which the OAuth client directs requests. In the Authentication Redirect Request list, select the auth-redirect-request configured earlier. In the Token Request list, select the token-request configured earlier. Some deployments may not need the additional information provided by OpenID Connect. You could, in that case, disable it. Figure 13: OAuth Client agent in Visual Policy Editor Per-session policy Configure the Per Session policy as appropriate for your chosen deployment type. Per-request: The per-session policy must contain at least one logon page to set the username variable in the user’s session. Preferably it should also perform some type of primary authentication. This validated username is used later in the per-request policy. Per-session: The per-session policy is used for all authentication. A per-request policy is not used. Figures 14a and 14b show a per-session policy that runs when a client initiates a session. Depending on the actions you include in the access policy, it can authenticate the user and perform actions that populate session variables with data for use throughout the session. Figure 14a: Per-session policy in Visual Policy Editor performs both primary authentication and Duo authentication (for per-session use case) Figure 14b: Per-session policy in Visual Policy Editor performs primary authentication only (for per-request use case) Apply policy/policies and iRule to the APM virtual server Finally, apply the per-request policy, per-session policy, and iRule to the APM virtual server. You assign iRules as a resource to the virtual server that users connect. Configure the virtual server’s default pool to the protected local web resource. Apply policy/policies to the virtual server Per-request policy To attach policies to the virtual server: 1. On the Main tab, click Local Traffic > Virtual Servers. 2. Select the Virtual Server. 3. In the Access Policy section, select the policy you created. 4. Click Finished. Figure 15: Access Policy section in Virtual Server (per-request policy) Per-session policy Figure 16 shows the Access Policy section in Virtual Server when the per-session policy is deployed. Figure 16: Access Policy section in Virtual Server (per-session policy) Apply iRule to the virtual server To attach the iRule to the virtual server: 1. On the Main tab, click Local Traffic > Virtual Servers. 2. Select the Virtual Server. 3. Select the Resources tab. 4. Click Manage in the iRules section. 5. Select an iRule from the Available list and add it to the Enabled list. 6. Click Finished.
