Automating ACMEv2 Certificate Management on BIG-IP
While we often associate and confuse Let's Encrypt with ACMEv2, the former is ultimately a consumer of the latter. The "Automated Certificate Management Environment" (ACME) protocol describes a system for automating the renewal of PKI certificates. The ACME protocol can be used with public services like Let's Encrypt, but also with internal certificate management services. In this article we explore the more generic support of ACME (version 2) on the F5 BIG-IP.
7. SYN Cookie: Troubleshooting Stats
Introduction In this article I will explain what SYN Cookie stats you can consult and their meaning. There are more complex stats than the explained in this article but they are intended for helping F5 engineers when cases become complex. SYN Cookie stats First at all, in order to troubleshoot SYN Cookie you need to know how you can check SYN Cookie stats easily and understand what you are reading. The easiest way to see these stats in a device running LTM module based SYN Cookie is by running below command and focusing in SYN Cookies section of the output: # tmsh show ltm virtual <virtual> SYN Cookies Status full-hardware Hardware SYN Cookie Instances 6 Software SYN Cookie Instances 0 Current SYN Cache 2.0K SYN Cache Overflow 24 Total Software 4.3M Total Software Accepted 0 Total Software Rejected 0 Total Hardware 21.7M Total Hardware Accepted 3 Let’s go through each field to know what they means: Status: [full-software/full-hardware|not-activated]. This value describes what type of SYN Cookie mode has been activated, software or hardware. Once an attack has finished it is normal that LTM takes some time to deactivate SYN Cookie mode after attack traffic stops. Calculation is complex and related to specific platform and several factors, also we do not want SYN Cookie entering in an activating/deactivating loop in case TCP SYN packets per second reaching device are near to the configured SYN cache threshold. So delay of 30-60 seconds before SYN Cookie being deactivated is in normal range. Hardware SYN Cookie Instances: How many TMMs are under Hardware SYN Cookie mode. Software SYN Cookie Instances: How many TMMs are under Software SYN Cookie mode. It indicates if software is currently generating SYN cookies. Current SYN Cache: Indicates how many embryonic connections are handled by BIG-IP (refreshed every 2 seconds). SYN cache is always counting embryonic connections, regardless if SYN Cookie is activated or not. So even if SYN Cookies is completely turned off, we still have embryonic flows that have not been promoted to full flows yet (SYNs which has not completed 3WHS), this means that cache counter will still be used regularly, so is normal having a value different than 0. Disabling SYN Cookie will not avoid this counter increase but thresholds will not be taken into account. Since SYN cache is no longer a cache, as explained in prior articles, and the stat is merely a counter of embryonic flows, it does not consume memory resources. As the embryonic flows are promoted or time out, this value will decrease. SYN Cache Overflow (per TMM): It is incremented whenever the SYN cache threshold is exceeded and SYN cookies need to be generated. It increments in one per each TMM instance and this value is a counter that only increases, so we can consult the value to know how many times SYN Cookie has been activated since last stats reset, remember, per TMM. Total Software: Number of challenges generated by Software. This is increased regardless if client sends a response, does not send any response or the response is not correct. Total Software Accepted: Number of TCP handshakes that were correctly handled with clients. Total Software Rejected: Number of wrong responses to challenges. Remember that ALL rejected SYN cookies are in software, there is no hardware rejected. Total Hardware: Number of challenges generated by Hardware. Total Hardware Accepted: Number of TCP handshakes that were correctly handled with clients. In case you have configured AFM based SYN Cookie then you can use two easy sources of information, the already explained LTM command above, but also you can check stats for TCP half open DoS vector, at device or virtual server context, as you would do with any other DoS vector. Let’s check the most important fields at device context. # tmsh show security dos device-config | grep -A 40 half Statistics Type Count Status Ready Attack Detected 1 *Attacked Dst Detected 0 *Bad Actor Attack Detected 0 Aggregate Attack Detected 1 Attack Count 2 *Stats 1h Samples 0 Stats 408 *Stats Rate 408 Stats 1m 104 Stats 1h 0 Drops 1063 *Drops Rate 1063 Drops 1m 187 Drops 1h 4 *Int Drops 0 *Int Drops Rate 0 *Int Drops 1m 0 *Int Drops 1h 0 Status: This field confirm if this specific DoS vector is ready to detect TCP SYN flood attacks. You have to take into account that you could decide to configure this DoS vector with Auto-threshold, in which case AFM would be in charge of deciding the best threshold. Note that by enabling auto-threshold AFM would need some time to learn the traffic pattern of your environment, until it has enough information to create a correct threshold you will not see this field as Ready but as Learning. If vector is configured manually you always see it as Ready. Attack detected: It informs you if currently there is an ongoing attack and detected by AFM. Aggregate attack detected: Since DoS stats shown above are for device context the detected attack is aggregate. Attack Count: Gives information about how many attacks have been detected since stats were reset last time. It does not decrease. Stats: Number of embryonic connections at this current second. Remember that since this is a snapshot, the counter could go increase or decrease. This is different to other DoS vectors where it only increases. Stats 1m: Average of the Stats in the last minute. This is the average number of embryonic connections that AFM has seen when taking sampling every 1s. Stats 1h: Average of the Stats 1m in the last hour. Drops: It counts the number of wrong ACKs received. In other words this is the current snapshot of: Number of SYN Cookies – Number of validated ACKs received Drops 1m: Average of the Drops in the last minute Drops 1h: Average of the Drops 1m in the last hour I have added an asterisk to some fields to point to values that has not real meaning for SYN Cookie DoS vector, or information is not really useful from my perspective, but they are included to have coherence with other DoS attack stats information. Note: Detection logic for this vector is not based on the Stats 1m, as other DoS vectors, instead it is based on the current number of embryonic connections, that is, value seen in Stats counter. Interpreting stats Once you know what each important stat mean I will give some advises when you interpret SYN Cookie stats. You should know some of them already if you have read all articles in this series: Hardware can offload TMM for validation, so you can see a number of software generated SYN Cookies much bigger than software validated SYN Cookie since software generates cookies that are validated by hardware as well. Since it is possible that a software generated SYN Cookie be accepted by hardware and vice versa, hardware generated SYN Cookies be accepted by software, you could think that value for Total Software is the same than the result of adding the Total Software Accepted plus Total Software rejected. But that is NOT true since it can be possible that a SYN Cookie sent by BIG-IP does not have a response (ACK), quite typical during a SYN flood attack indeed. In this case nothing adds up because generated SYN Cookie was not accepted nor rejected. Remember that SYN cookie’s responses discarded by hardware will be rejected by software, so all rejects are in software. This is why there is not counter for Total Hardware rejected. This means that Total Software Rejected can be increase by Total Hardware. When there is an attack vectors definition that match this attack will be increased. So, for example, during a TCP SYN flood attack you will see that Stats increases for TCP half open and TCP SYN flood (maybe others like low TTL,... as well). Also Stats will increase in all contexts (device and virtual server). The first vector in an specific context whose limit is exceeded it will start to mitigate. This is important because in cases where TCP SYN flood DoS vector has a lower threshold than TCP half open DoS vector, you will notice that traffic is dropped but you did not expect this behavior. Check article 5 for more information. Example In this part you will learn what stats changes you should expect to see when a TCP SYN flood attack is detected and SYN Cookie is activated, so it starts to mitigate the attack. Let’s interpret below stats: During attack Status full-hardware Hardware SYN Cookie Instances 6 Software SYN Cookie Instances 0 Current SYN Cache 1 SYN Cache Overflow 24 Total Software 10.1K Total Software Accepted 0 Total Software Rejected 0 Total Hardware 165.1M Total Hardware Accepted 3 After attack Status not-activated Hardware SYN Cookie Instances 0 Software SYN Cookie Instances 0 Current SYN Cache 15 SYN Cache Overflow 24 Total Software 10.1K Total Software Accepted 0 Total Software Rejected 0 Total Hardware 171.0M Total Hardware Accepted 3 Before the attack, before SYN Cookie is activated, you will see Current SYN Cache stats starts to increase quickly since TCP SYN packets are causing an increment of embryonic connections. Once SYN Cache threshold has been reached in a TMM then SYN Cache Overflow will increase attending to the number of TMMs that detected the attack. In above example you see 24 because this is a 6 CPUs device and 4 TCP SYN Flood attacks were detected by SYN Cookie. Remember this counter will never decrease unless we reset stats. During attack SYN cookie is activated so Current SYN Cache will start to decrease until reaching 0 because SYN Cookie Agent starts to handle TCP 3WHS, in other words, TCP stack stops to receive TCP SYN packets. We can check how many TMMs have SYN cookie activated currently looking at Hardware/Software SYN Cookie Instances counter. Note this match with what I explained for SYN Cache Overflow value. Legitimate connections are counted under Total Hardware/Software Accepted. In this case, we can see that although several millions of TCP SYN packets reached the device only 3 TCP 3WHS were correctly carried out. As you can see this device has Hardware SYN Cookie configured and working, but you can read that software also generates SYN Cookie challenges (Total Software). As commented in article number six of these series, this is expected and can be due to collisions or due to validations of first challenges when SYN Cookie is activated and TMMs handles TCP SYN packets until it enables hardware to do it. This does not affect device performance. In order to change from ‘Status full-hardware’ to ‘Status not-activated’ both HSBs must exit from SYN Cookie mode. Conclusion Now you know how to interpret stats, so you know can deduce information about the past and the present status of your device related to SYN Cookie. In next two article I will end up this series and this troubleshooting part talking about traffic captures and meaning of logs.A Very Chinese New Year
Happy New Year everyone! It's a new year, with new news, and the same old(er) MegaZone. This time we're looking at the news that I found worthy from the week of January 5-11, 2025. (Have you gotten used to typing 2025 yet?) I found it to be a fairly slow news week, and not much really grabbed my attention enough that I felt it was worth commenting on. That's not too unusual for the start of a new year, as there is often a bit of a post-holiday lull. Not that there was no news at all, it is never truly quiet in cybersecurity, just that most of it was run-of-the-mill stuff, IMHO. Oh, and as for the title of this 'issue', I know the Lunar New Year (aka Chinese New Year) isn't until January 29th, but I couldn't pass up the play on words given the topic below. And with that, let's dive in.Mitigating OWASP API Security Top 10 risks using F5 NGINX App Protect
This 2019 API Security article covers the summary of OWASP API Security Top 10 – 2019 categories and newly published 2023 API security article covered introductory part of newest edition of OWASP API Security Top 10 risks – 2023. We will deep-dive into some of those common risks and how we can protect our applications against these vulnerabilities using F5 NGINX App Protect. Excessive Data Exposure Problem Statement: As shown below in one of the demo application API’s, Personal Identifiable Information (PII) data, like Credit Card Numbers (CCN) and U.S. Social Security Numbers (SSN), are visible in responses that are highly sensitive. So, we must hide these details to prevent personal data exploits. Solution: To prevent this vulnerability, we will use the DataGuard feature in NGINX App Protect, which validates all response data for sensitive details and will either mask the data or block those requests, as per the configured settings. First, we will configure DataGuard to mask the PII data as shown below and will apply this configuration. Next, if we resend the same request, we can see that the CCN/SSN numbers are masked, thereby preventing data breaches. If needed, we can update configurations to block this vulnerability after which all incoming requests for this endpoint will be blocked. If you open the security log and filter with this support ID, we can see that the request is either blocked or PII data is masked, as per the DataGuard configuration applied in the above section. Injection Problem Statement: Customer login pages without secure coding practices may have flaws. Intruders could use those flaws to exploit credential validation using different types of injections, like SQLi, command injections, etc. In our demo application, we have found an exploit which allows us to bypass credential validation using SQL injection (by using username as “' OR true --” and any password), thereby getting administrative access, as below: Solution: NGINX App Protect has a database of signatures that match this type of SQLi attacks. By configuring the WAF policy in blocking mode, NGINX App Protect can identify and block this attack, as shown below. If you check in the security log with this support ID, we can see that request is blocked because of SQL injection risk, as below. Insufficient Logging & Monitoring Problem Statement: Appropriate logging and monitoring solutions play a pivotal role in identifying attacks and also in finding the root cause for any security issues. Without these solutions, applications are fully exposed to attackers and SecOps is completely blind to identifying details of users and resources being accessed. Solution: NGINX provides different options to track logging details of applications for end-to-end visibility of every request both from a security and performance perspective. Users can change configurations as per their requirements and can also configure different logging mechanisms with different levels. Check the links below for more details on logging: https://www.nginx.com/blog/logging-upstream-nginx-traffic-cdn77/ https://www.nginx.com/blog/modsecurity-logging-and-debugging/ https://www.nginx.com/blog/using-nginx-logging-for-application-performance-monitoring/ https://docs.nginx.com/nginx/admin-guide/monitoring/logging/ https://docs.nginx.com/nginx-app-protect-waf/logging-overview/logs-overview/ Unrestricted Access to Sensitive Business Flows Problem Statement: By using the power of automation tools, attackers can now break through tough levels of protection. The inefficiency of APIs to detect automated bot tools not only causes business loss, but it can also adversely impact the services for genuine users of an application. Solution: NGINX App Protect has the best-in-class bot detection technology and can detect and label automation tools in different categories, like trusted, untrusted, and unknown. Depending on the appropriate configurations applied in the policy, requests generated from these tools are either blocked or alerted. Below is an example that shows how requests generated from the Postman automation tool are getting blocked. By filtering the security log with this support-id, we can see that the request is blocked because of an untrusted bot. Lack of Resources & Rate Limiting Problem Statement: APIs do not have any restrictions on the size or number of resources that can be requested by the end user. Above mentioned scenarios sometimes lead to poor API server performance, Denial of Service (DoS), and brute force attacks. Solution: NGINX App Protect provides different ways to rate limit the requests as per user requirements. A simple rate limiting use case configuration is able to block requests after reaching the limit, which is demonstrated below. Conclusion: In short, this article covered some common API vulnerabilities and shows how NGINX App Protect can be used as a mitigation solution to prevent these OWASP API security risks. Related resources for more information or to get started: F5 NGINX App Protect OWASP API Security Top 10 2019 OWASP API Security Top 10 2023
API Security: How to implement API Access Control with F5
Per Gartner, API Security comprises two aspects: API Threat Protection and API Access Control. API Threat Protection is addressed with measures such as API L7 firewall, bot protection, DDoS mitigation, etc. API Access Control is most comprehensively addressed via OAuth 2.0. This article will see our attention focused on API Access Control, and how F5 APM can be leverage for this universal challenge when protecting API.Mitigating OWASP Web Application Security Top 10 – 2021 risks using F5 Distributed Cloud Platform
Overview: In the early 90’s, applications were in dormant phase and JavaScript & XML were dominating this technology. But in 1999, the first web application was introduced after the release of the Java language in 1995. Later with the adoption of new languages like Ajax, HTML, Node, Angular, SQL, Go, Python, etc. and availability of web application frameworks have boosted application development, deployment, and release to production. With the evolving software technologies, modern web applications are becoming more and more innovative, providing users with a grand new experience and ridiculously ease of interface. With these leading-edge technologies, novel exploit surfaces are also exposed which made them a primary target for intruders/hackers. Application safeguarding against all these common exploits is a necessary step in protecting backend application data. Open Worldwide Application Security Project (OWASP) is one of those security practices which protects application with above issues. This article is the first part of the series and covers OWASP evolution, its importance and overview of top 10 categories. Before diving into OWASP Web Application Security Top 10, let’s time travel to era of 1990’s and try to identify challenges the application customers, developers and users were facing. Below are some of them: Rapid and diversified cyber-attacks has become a major concern and monitoring/categorizing them was difficult Product owners are concerned about application security & availability and are in desperate need of a checklist/report to understand their application security posture Developers are looking for recommendations to securely develop code before running into security flaws in production No consolidated repo to manage, document and provide research insights for every security vulnerability After running into the above concerns, people across the globe have come together in 2001 and formed an international open-source community OWASP. It’s a non-profit foundation which has people from different backgrounds like developers, evangelist, security experts, etc. The main agenda for this community is to solve application related issues by providing: Regularly updating “OWASP TOP 10” report which provides insights of latest top 10 security issues in web applications Report also provides security recommendations to protect them from these issues Consolidated monitoring and tracking of application vulnerabilities Conducting events, trainings and conferences around the world to discuss, solve and provide preventive recommendations for latest security issues OWASP also provides security tools, research papers, libraries, cheat sheets, books, presentations and videos covering application security testing, secure development, and secure code review OWASP WEB SECURITY TOP 10 2021: With the rapid increase of cyber-attacks and because of dynamic report updates, OWASP gained immense popularity and is considered as one of the top security aspects which application companies are following to protect their modern applications against known security issues. Periodically they release their Top 10 vulnerabilities report and below are the latest Top 10 - 2021 categories with their summary: A01:2021-Broken Access Control Access controls enforce policy such that users cannot act outside of their intended permissions. Also called authorization, it allows or denies access to your application's features and resources. Misuse of access control enables unauthorized access to sensitive information, privilege escalation and illegal file executions. Check this article on protection against broken access vulnerabilities A02:2021-Cryptographic Failures In 2017 OWASP top 10 report, this attack was known as Sensitive Data Exposure, which focuses on failures related to cryptography leading to exposure of sensitive data. Check this article on cryptographic failures A03:2021-Injection An application is vulnerable to injection if user data and schema is not validated by the application. Some of the common injections are XSS, SQL, NoSQL, OS command, Object Relational Mapping (ORM), etc., causing data breaches and loss of revenue. Check this article on safeguarding against injection exploits A04:2021-Insecure Design During the development cycle, some phases might be reduced in scope which leads to some of the vulnerabilities. Insecure Design represents the weaknesses i.e., lack of security controls which are not tracked in other categories throughout the development cycle. Check this article on design flaws and mitigation A05:2021-Security Misconfiguration This occurs when security best practices are overlooked allowing attackers to get into the system utilizing the loopholes. XML External Entities (XXE), which was previously a Top 10 category, is now a part of security misconfiguration. Check this article on protection against misconfiguration vulnerabilities A06:2021-Vulnerable and Outdated Components Applications used in enterprises are prone to threats such as code injection, buffer overflow, command injection and cross-site scripting from unsupported, out of date open-source components and known exploited vulnerabilities. Utilizing components with security issues makes the application itself vulnerable. Intruders will take use of this defects and exploit the deprecated packages thereby gaining access to backend applications. Check this article on finding outdated components A07:2021-Identification and Authentication Failures Confirmation of the user's identity, authentication, authorization and session management is critical to protect applications against authentication-related attacks. Apps without valid authorization, use of default credentials and unable to detect bot traffic are some of the scenarios in this category. Check this article on identifying and protection against bots A08:2021-Software and Data Integrity Failures Software and data integrity failures occurs when updates are pushed to the deployment pipeline without verifying its integrity. Insecure Deserialization, which was a separate category in OWASP 2017, has now become a part of this larger category set. Check this article on software failures protection A09:2021-Security Logging and Monitoring Failures As a best recommendation, we shall always log all incoming request details and monitor application for fraudulent transactions, invalid logins, etc. to identify if there are any attacks or breaches. Applications without logging capabilities provide opportunities to the attackers to exploit the application and may lead to many security concerns. Without logging and monitoring we won’t be able to validate the application traffic and can’t identify the source of the breach. Check this article for identifying logging issues A10:2021-Server-Side Request Forgery Server-Side Request Forgery (SSRF) attack is a technique which allows intruders to manipulate the server-side application vulnerability and make a malicious request to the internal-only resources. Attacker exploits this flaw by modifying/crafting a URL which forces the server to retrieve and disclose sensitive information. Check this article which focusses on SSRF mitigation NOTE: This is an overview article of this OWASP series, check the below links to prevent these vulnerabilities using F5 Distributed Cloud Platform. OWASP Web Application Security Series: Broken access mitigation Cryptographic failures Injection mitigation Insecure design mitigation Security misconfiguration prevention Vulnerable and outdated components Identification failures prevention Software failures mitigation Security logging issues prevention SSRF MitigationHow I did it - “Delivering Kasm Workspaces three ways”
Securing modern, containerized platforms like Kasm Workspaces requires a robust and multi-faceted approach to ensure performance, reliability, and data protection. In this edition of "How I did it" we'll see how F5 technologies can enhance the security and scalability of Kasm Workspaces deployments.F5 Distributed Cloud JA4 detection for enhanced performance and detection
JA4+ is a suite of network fingerprinting methods. These methods are both human and machine readable to facilitate more effective threat-hunting and analysis. The use cases for these fingerprints include scanning for threat actors, malware detection, session hijacking prevention, compliance automation, location tracking, DDoS detection, grouping of threat actors, reverse shell detection, and many more. Introduction In a previous article, Identity-Aware decisions with JA4+ we discussed using JA4 fingerprints with BIG-IP. In this article, we are exploring the use of JA4 in F5 Distributed Cloud. A very useful use case for using JA4 in F5 Distributed Cloud is explained at F5 App Connect and NetApp S3 Storage – Secured Scalable AI RAG. Let's go through the steps of getting the JA4 fingerprints applied to a traffic sample. Implementation In this example we are using NGINX instance deployed via F5 Distributed Cloud Distributed Apps. Deploy Virtual K8s through Distributed Apps. Create service policy with the matching JA4 fingerprints to block. JA4 Database can be found over here JA4 Database Service policy creation From Distributed Cloud UI > Distributed Apps > Manage > Service Policies > Service Policies Add Service Policy Add name: ja4-service-policy Under rules, select Custom rules and then click configure Click Add item Update the below, Add name, Actions. Show advanced fields in the client section. TLS Fingerprint Matcher: JA4 TLS Fingerprint Click Configure JA4 TLS Fingerprint Click Add item and match the needed JA4 fingerprint. In our case, we are blocking curl, wget fingerprints. Click Apply, to save, then Save, and Exit. Now, we attach the service policy to our HTTP Load balancer. Manage > HTTP Loadbalancer > Click Manage configurations Click Edit Configurations At Common Security Controls section, Select Apply Service Policies and click Edit Configurations. Select the configured policy, then Apply. Testing From Firefox browser From Ubuntu using curl Observing logs from F5 Distributed Cloud From HTTP Loadbalancers > select the created loadbalancer and click Security Monitoring Click Security Events to check the requests You can see the events with the requests and client information From Action column, you can select Explain with AI to gain further information and recommendations. We have the service policy configured and attached. It can be attached as well to different component for client identification as well. Related Content F5 App Connect and NetApp S3 Storage – Secured Scalable AI RAG | DevCentral Fingerprint TLS Clients with JA4 on F5 BIG-IP using iRules JA4 Part 2: Detecting and Mitigating Based on Dynamic JA4 Reputation | DevCentral Identity-Aware decisions with JA4+ | DevCentral Setting Up A Basic Customer Edge To Run vk8s in F5 Distributed Cloud App Stack | DevCentral
