F5 May 2025 QSN, Big dollar cough up, buggy-spy chat apps
On May 5th, F5 disclosed 12 issues, 11 Highs, and 1 Medium Severity CVEs for the F5 May 2025 Quarterly Security Notifications. Most of the issues disclosed were classic DoS on BIG-IP products and the BIG-IP NEXT products and are fixed in the latest BIG-IP 17.5, 16.1.6, and most in15.1.10.7 versions and the latest BIG-IP NEXT versions.The Future Soon
It's the first May, first of May, outdoor... Oh, hi, didn't see you there. Welcome back, once again, to This Week In Security the weekly (mostly weekly) newsletter where we take the random security news of the week and run it down. I'm your host this week, , and these are the items that happened to catch my eye as I once again drank from the firehose of doom, or security news, same thing.Policy Puppetry, Jumping the Line, and Camels
Notable news for the week of April 20th - April 27th, 2025. This week, your editor is Jordan_Zebor from F5 Security Incident Response Team. This week, I’m diving into some big updates around Generative AI security. Every time a new tech wave hits—whether it was social media, crypto, IoT, or now AI—you can bet attackers and security teams are right there, too. The latest threats, like Policy Puppetry and Line Jumping show just how fast things are moving. On the flip side, defenses like CaMeL are helping us stay one step ahead. If we want AI systems that people can trust, security engineers have to stay sharp and keep building smarter defenses. Policy Puppetry: A Universal Prompt Injection Technique It seems like weekly, new ways to perform prompt injection are being discovered. Recent research by HiddenLayer has uncovered "Policy Puppetry," a novel prompt injection technique that exploits the way LLMs interpret structured data formats such as XML and JSON. This works because the full context—trusted system instructions and user input alike—is flattened and presented to the LLM without inherent separation (something I will touch on later). By presenting malicious prompts that mimic policy files, attackers can override pre-existing instructions and even extract sensitive information, compromising the integrity of systems powered by LLMs like GPT-4, Claude, and Gemini. For security engineers, this discovery underscores a systemic vulnerability tied to LLMs' training data and context interpretation. Existing defenses, such as system prompts, are insufficient on their own to prevent this level of exploitation. The emergence of Policy Puppetry adds to the ongoing discussion about prompt injection as the most significant Generative AI threat vector, highlighting the urgent need for comprehensive safeguards in AI system design and deployment. MCP Servers and the "Line Jumping" Vulnerability Trail of Bits uncovered a critical vulnerability in the Model Context Protocol (MCP), a framework used by AI systems to connect with external servers and retrieve tool descriptions. Dubbed "line jumping," this exploit allows malicious MCP servers to embed harmful prompts directly into tool descriptions, which are processed by the AI before any tool is explicitly invoked. By bypassing the protocol’s safeguards, attackers can manipulate system behavior and execute unintended actions, creating a cascading effect that compromises downstream systems and workflows. This vulnerability undermines MCP's promises of Tool Safety and Connection Isolation. The protocol is designed to ensure that tools can only cause harm when explicitly invoked with user consent and to limit the impact of any compromised server through isolated connections. However, malicious servers bypass these protections by instructing the model to act as a message relay or proxy, effectively bridging communication between supposedly isolated components. This creates an architectural flaw akin to a security system that activates prevention mechanisms only after intruders have already breached it. Google's CaMeL: A Defense Against Prompt Injection In response to prompt injection threats, Google DeepMind has introduced CaMeL (Capability-based Model Execution Layer), a groundbreaking mechanism designed to enforce control and data flow integrity in AI systems. By associating “capabilities”—metadata that dictate operational limits—with every value processed by the model, CaMeL ensures untrusted inputs cannot exceed their designated influence. Instead of sprinkling more AI magic fairy dust on the problem, this approach leans on solid, well-established security principles concepts into the AI domain. By implementing a protective layer around the LLM, developers can reduce risks such as unauthorized operations and unexpected data exfiltration, even if the underlying model remains vulnerable to prompt injection. While CaMeL has not yet been tested broadly, its potential represents a significant advancement toward secure-by-design AI systems, establishing a new architectural standard for mitigating prompt injection vulnerabilities. That’s it for this week — hope you enjoyed!VulnCon 2025, EU CRA, CVE funding, Smishing Kit
Notable security news for the week of April 13th through April 20th. Your editor this week is Chris from the F5 Security Incident Response Team. A bit of a different format this week as I was in Raleigh for VulnCon 2025 the previous week. I will discuss highlights from that as well as notable events from the past week. VulnCon 2025 The 2025 Vulnerability Management Ecosystem Collaboration, Ideation, and Action Conference (aka “VulnCon”), which was sponsored by FIRST and the CVE Program, was held from April 7th through April 10th this year. The aim of this conference is to promote collaboration between various vulnerability management and cybersecurity professionals to better help the whole cybersecurity ecosystem. Key topics that were highlighted this year were the EU's Cyber Resilience Act (CRA), Vulnerability Exploitability eXchange (VEX), Cybersecurity Assurance Framework (CSAF), and publishing more complete CVE records or Vulnrichment. I will discuss the CRA in the next paragraph. VEX facilitates the exchange of vulnerability information, fostering collaboration to swiftly address emerging threats. Concurrently, CSAF ensures a standardized approach to cybersecurity practices. One of the pushes that was discussed was to get security scanners to start ingesting VEX to help decrease the amount of false positives and focus more on vulnerabilities that are exploitable. As for Vulnrichment, it is alarming that a large number of CVEs that are disclosed every year do not include Common Weakness Enumerations (CWE) or Common Vulnerability Scoring System (CVSS) scores. I agree that adding this information at a minimum would be very beneficial to both the consumers as well as the vendor. The vendor is in the best position to assign these in a more accurate manner since they are most familiar with the products. https://www.first.org/conference/vulncon2025/ https://openssf.org/blog/2023/09/07/vdr-vex-openvex-and-csaf/ EU Cyber Resilience Act (CRA) The Cyber Resilience Act introduces mandatory cybersecurity requirements for hardware and software products, throughout their whole lifecycle. The main goals of this act are to ensure that products with digital elements placed on the EU market have fewer vulnerabilities that Manufacturers remain responsible for cybersecurity throughout a product’s life cycle, improve transparency on security of hardware and software products and bring benefits to business users and consumers from better protection. Products will bear the CE marking as is common with many other products sold, which means they have been assessed to meet high safety, health, and environmental protection requirements. The three main roles that are laid out are: Manufacturers: If you develop or manufacture products with "digital elements" for sale in the EU. Open-Source Software (OSS) Stewards: Entity other than a manufacturer that provides support on a sustained basis for the development of specific products with digital elements, qualifying as free and open-source software and intended for commercial activities, and that ensures the viability of those products. Examples of this would be the Linux Foundation, Apache Foundation, etc.... OSS Developers: Upstream maintainer or developer of open-source software that is used by the manufacturer. The key point to note in this distinction of these three roles is that if there is a vulnerability exploited in open-source software, the manufacturer is the one held liable. The OSS Steward and the OSS Developers are not being held liable. This makes it a good idea to develop working relationships with the OSS upstream developers for when emergencies do arise. Now to focus on the manufacturer requirements. I will not touch all of them as the CRA is a large document but will point out some of the key topics. Secure-By-Default and Secure-By-Design principles will now be a requirement and not just a pledge. Products with digital elements shall be delivered without any known exploitable vulnerabilities. Manufacturers will need to provide evidence that the product was checked before release. A risk assessment will need to be provided with the product and the contents of that assessment are laid out in Annex I of the document. Manufactures must be able to provide SBOMS in either SPDX or CycloneDX format at the request of authorities. Manufacturers must , address and remediate vulnerabilities without delay, which includes providing security updates. Manufacturers must provide support for a minimum of 5 years, including security updates and that each security update remains available after it has been issued for a minimum of 10 years or for the remainder of the support period, whichever is longer. There are also reporting requirements. Highlight a couple of them; they pertain to actively exploited vulnerabilities and severe incidents having an impact on the security of the product. An early warning notification of an actively exploited vulnerability or severe incident, without undue delay and in any event within 24 hours of the manufacturer becoming aware of it. Then "an incident notification, without undue delay and in any event within 72 hours of the manufacturer becoming aware of the incident, which shall provide general information, where available, about the nature of the incident, an initial assessment of the incident, as well as any corrective or mitigating measures taken, and corrective or mitigating measures that users can take, and which shall also indicate, where applicable, how sensitive the manufacturer considers the notified information to be". That was taken from the document and you can see how thorough they are being when detailing what to report. The final report will need to be submitted by 14 days for active exploits and one month for severe incidents. Then to explain where to report: "The notification shall be submitted using the electronic notification end-point of the CSIRT designated as coordinator of the Member State where the manufacturers have their main establishment in the Union and shall be simultaneously accessible to ENISA". This means that the manufacturer will need to choose one of the European country's CSIRT teams to be the point of contact. As for the consequences of non-compliance, the EU is not playing around with that either: Non-compliance with the essential cybersecurity requirements set out in Annex I and the obligations set out in Articles 13 and 14 shall be subject to administrative fines of up to EUR 15,000,000 or, if the offender is an undertaking, up to 2.5% of its total worldwide annual turnover for the preceding financial year, whichever is higher. Non-compliance with the obligations set out in Articles 18 to 23, Article 28, Article 30(1) to (4), Article 31(1) to (4), Article 32(1), (2) and (3), Article 33(5), and Articles 39, 41, 47, 49 and 53 shall be subject to administrative fines of up to EUR 10,000,000 or, if the offender is an undertaking, up to 2% of its total worldwide annual turnover for the preceding financial year, whichever is higher. The supply of incorrect, incomplete or misleading information to notified bodies and market surveillance authorities in reply to a request shall be subject to administrative fines of up to EUR 5,000,000 or, if the offender is an undertaking, up to 1% of its total worldwide annual turnover for the preceding financial year, whichever is higher. To explain those bullet points more simply, everything I mentioned above about manufacturer requirements and reporting all fall under Annex I or Articles 13 and 14 so would be subject to the most severe penalties per incident. As for the timelines of this act, the CRA was officially adopted on October 10, 2024, and entered into force on December 10, 2024. However, the CRA's main obligations will apply starting from December 11, 2027. Some earlier obligations will apply, such as the reporting of vulnerabilities and severe incidents, starting from September 11, 2026. Additionally, the rules on conformity assessment bodies will be applicable from June 11, 2026. https://digital-strategy.ec.europa.eu/en/policies/cyber-resilience-act https://github.com/SecurityCRob/presentations/blob/main/CRA%20PSIRT%20TL_DR.pdf CVE Program Funding The Common Vulnerabilities and Exposures (CVE) program, managed by MITRE, was facing funding expiration on April 16, 2025. The program is essential for identifying and tracking security vulnerabilities in software and hardware. Without funding, the CVE program would stop adding new vulnerabilities, which could lead to significant impacts on national vulnerability databases, cybersecurity tools, incident response operations, and critical infrastructure. MITRE's Vice President Yosry Barsoum expressed hope that the government is making efforts to continue supporting the program. Luckily, the Department of Homeland Security's (DHS) Cybersecurity & Infrastructure Security Agency (CISA) was able to secure funding at the last moment, to fund the program for 11 more months. This is despite ongoing budget and staffing cuts to CISA by the current administration. The cybersecurity community has expressed concern over the potential loss of the CVE program, emphasizing its importance in standardizing vulnerability information and aiding in the timely patching of security flaws. On April 16, MITRE announced the creation of a non-profit entity called "The CVE Foundation" to continue the program's work under a new funding mechanism. https://krebsonsecurity.com/2025/04/funding-expires-for-key-cyber-vulnerability-database/ The CVE Foundation The CVE Foundation was launched to secure the future of the CVE Program. The CVE Foundation was formally established on April 16, 2025, to ensure the long-term viability, stability, and independence of the Common Vulnerabilities and Exposures (CVE) Program. The CVE Program has been a U.S. government-funded initiative for 25 years, raising concerns about sustainability and neutrality due to its reliance on a single government sponsor. MITRE notified the CVE Board on April 15, 2025, that the U.S. government would not renew its contract for managing the program. In response, a coalition of CVE Board members developed a strategy to transition CVE to a dedicated, non-profit foundation to continue delivering high-quality vulnerability identification. CVE identifiers and data are crucial for cybersecurity professionals worldwide, aiding in security tools, advisories, threat intelligence, and response. Going forward, the CVE Foundation aims to eliminate a single point of failure in the vulnerability management ecosystem and ensure the CVE Program remains globally trusted and community-driven. https://www.thecvefoundation.org/home Pay Your Tolls!! About 3 and a half years ago, I was driving into Denver on Interstate 70 coming from the East. I had no need to drive through Denver as I was heading north through Wyoming anyway. Well, a few miles before the city there was an offramp for E-470, a toll highway that would bypass Denver and connect to I-25 a few miles north of the city. I had never used a toll highway before since I live in Eastern Washington where they are unheard of. I was picturing a scene out of a movie where you pull up to a booth and pay an attendant. I was surprised as I merged onto the highway and everyone was driving at 60+ MPH. I saw a sign that stated the system used E-ZPass and would scan the license plate. Fast forward a few months and I receive a bill in the mail from E-ZPass, a pretty nice way to bypass driving through the middle of a large city, I thought. Now, unfortunately, a smishing campaign is targeting that same system to trick victims into giving them their payment information. Since mid-October 2024, multiple financially motivated threat actors have been using a smishing kit developed by "Wang Duo Yu" to target toll road users in eight U.S. states. The campaign impersonates U.S. electronic toll collection systems like E-ZPass, sending SMS messages and Apple iMessages about unpaid tolls, urging recipients to click on fake links. Victims are prompted to solve a fake CAPTCHA challenge and enter personal and financial information on fraudulent pages, which is then siphoned off to the threat actors. Wang Duo Yu, a computer science student in China, is alleged to be the creator of the phishing kits used by the Smishing Triad, a Chinese organized cybercrime group. The Smishing Triad has conducted large-scale smishing attacks targeting postal services in 121 countries, using failed package delivery lures to harvest personal and financial information. Services like Oak Tel facilitate smishing on a global scale, allowing cybercriminals to send bulk SMS and manage campaigns efficiently. I have personally received 2 or 3 of these over the past few months. Luckily, I know the one time I drove on a toll road, so it was obvious to me that this was fake. I worry about people that are using these systems more regularly that may fall victim. https://thehackernews.com/2025/04/chinese-smishing-kit-behind-widespread.htmlOracle hack, North Korean Hackers, Critical Flaw in Apache
This week in cybersecurity, it seems both hackers and defenders have been working overtime, While hackers have been creatively causing chaos, cybersecurity professionals have been equally busy patching up the digital world. It's a never-ending game of whack-a-mole, but with higher stakes and fewer mallets.IngressNightmare, Next.js critical, More Agents, pwned
Introduction Hello! ArvinF is your editor covering 23 to 29 March 2025 for this edition of F5 SIRT This Week in Security. Credit to the original sources. IngressNightmare Wiz has discovered serious vulnerabilities in the admission controller component of Ingress-Nginx Controller that could allow the total takeover of Kubernetes clusters – and thinks more than 6,000 deployments of the software are at risk on the internet. This vulnerability is fixed in Ingress NGINX Controller versions 1.12.1 and 1.11.5, so do update to the latest version. Ensure the admission webhook endpoint is not exposed externally. Other mitigations if an upgrade is not yet possible are: Enforce strict network policies so only the Kubernetes API Server can access the admission controller. Temporarily disable the admission controller component of Ingress-NGINX F5 published K000150538: Kubernetes ingress-nginx vulnerabilities CVE-2025-1097, CVE-2025-1098, CVE-2025-1974, and CVE-2025-24514 as F5 NGINX has a similarly named but different product NGINX Ingress Controller. The product has been assessed and it is not vulnerable to IngressNightmare related CVEs. https://my.f5.com/manage/s/article/K000150538 F5 also released in the March 27 2025 attack signature update (ASU) an attack signature to address IngressNightmare, namely, 200103569. K000150594: Attack Signatures for IngressNightmare: CVE-2025-1097, CVE-2025-1098, CVE-2025-24514, and CVE-2025-1974 https://my.f5.com/manage/s/article/K000150594 High 200103569 New Kubernetes NGINX Ingress Admission Controller Command Execution Command Execution CVE-2025-1097, CVE-2025-1098, CVE-2025-1974, CVE-2025-24514 Kubernetes Ingress NGINX controller is vulnerable to remote command execution via a malicious AdmissionReview request https://www.wiz.io/blog/ingress-nginx-kubernetes-vulnerabilities https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/ Next.js critical CVE-2025-29927 A critical severity vulnerability has been discovered in the Next.js open-source web development framework, potentially allowing attackers to bypass authorization checks. The flaw, tracked as CVE-2025-29927, enables attackers to send requests that reach destination paths without going through critical security checks. Next.js is a popular React framework with more than 9 million weekly downloads on npm. It is used for building full-stack web apps and includes middleware components for authentication and authorization. Next.js uses a header called 'x-middleware-subrequest' that dictates if middleware functions should be applied or not. The header is retrieved by the 'runMiddleware' function responsible for processing incoming requests. If it detects the 'x-middleware-subrequest' header, with a specific value, the entire middleware execution chain is bypassed and the request is forwarded to its destination. In the original research paper, the header and value "x-middleware-subrequest: /pages/_middleware" and "x-middleware-subrequest: middleware:middleware:middleware:middleware:middleware" and its variations were the examples and are potential "Indication of Attack" and will be useful with monitoring and logging systems and threat hunting. F5 released in the March 23 2025 ASU, an attack signature to address CVE-2025-29927, namely, 200013111. High 200013111 New Next.js Middleware Authorization Bypass Authentication/Authorization Attacks CVE-2025-29927 Next.js is vulnerable to an authorization check bypass on middleware via a specially crafted request https://zhero-web-sec.github.io/research-and-things/nextjs-and-the-corrupt-middleware https://www.bleepingcomputer.com/news/security/critical-flaw-in-nextjs-lets-hackers-bypass-authorization/ https://www.ncsc.gov.uk/news/vulnerability-affecting-nextjs-web-development-framework https://nextjs.org/blog/cve-2025-29927 https://jfrog.com/blog/cve-2025-29927-next-js-authorization-bypass/ Chrome emergency patches zero-day Google pushed out an emergency patch for Chrome on Windows this week to stop attackers from exploiting a sandbox-breaking zero-day vulnerability, seemingly used by snoops to target certain folks in Russia. Now Mozilla’s doing damage control, too, after spotting a similar flaw – albeit unexploited, as far as we’re aware – lurking in the code of its Firefox browser. "The vulnerability CVE-2025-2783 really left us scratching our heads, as, without doing anything obviously malicious or forbidden, it allowed the attackers to bypass Google Chrome’s sandbox protection as if it didn’t even exist," wrote Kaspersky researchers Igor Kuznetsov and Boris Larin. https://securelist.com/operation-forumtroll/115989/ https://www.theregister.com/2025/03/28/google_kaspersky_mozilla/ More Security Copilot Agents Microsoft revealed an expanded flight plan for Security Copilot, which is now assisted by 11 task-specific AI agents that interact with products like Defender, Purview, Entra, and Intune. Of the 11 Security Copilot agents introduced, five came from Microsoft Security partners. The Microsoft-made agents include: Phishing Triage Agent in Microsoft Defender, for sorting phishing reports. Alert Triage Agents in Microsoft Purview, for triaging data loss prevention and insider risk alerts. Conditional Access Optimization Agent in Microsoft Entra, for monitoring and preventing identity and policy issues. Vulnerability Remediation Agent in Microsoft Intune, for prioritizing vulnerability remediation. Threat Intelligence Briefing Agent in Security Copilot, for curating threat intelligence. Microsoft Security partners have also contributed to the agent pool: Privacy Breach Response Agent (OneTrust), for distilling data breaches into reporting guidance. Network Supervisor Agent (Aviatrix), for doing root-cause analysis on network issues. SecOps Tooling Agent (BlueVoyant), for assessing security operations center controls. Alert Triage Agent (Tanium), for helping security analysts prioritize alerts. Task Optimizer Agent (Fletch), for forecasting and prioritizing threat alerts. The eleventh agent resides in Microsoft Purview Data Security Investigations (DSI), an AI-based service designed to help data security teams deal with data exposure risks. Essentially, these agents use the natural language capabilities of generative AI to automate the summarization of high-volume data like phishing warnings or threat alerts so that human decision makers can focus on signals deemed to be the most pressing. F5 has reference literature on Agentic AI. The MS Security Pilot are "AI Agents" - focused on executing specific tasks based on predefined rules. "Agentic AI" has "autonomy and adaptive decision making" and is a combination of GenAI and AI Agents. Agentic AI combines extremely specific directive code that executes jobs with AI inference to generate or predict rich and contextual answers. Agentic AI is not magic, but it is more powerful than agents or GenAI operating alone. These two building blocks can be assembled in various amounts and combinations, automating a flow of work to produce tremendously valuable results. Here is a simple diagram depicting an automated agentic AI workflow. It uses multiple types of specialized agents and AI models to complete a set of actions. The solution executes until an acceptable outcome is achieved, and then it is fed back to the user. https://www.theregister.com/2025/03/24/microsoft_security_copilot_agents/ https://www.f5.com/company/blog/ai-agents-vs-agentic-ai-understanding-the-difference https://www.f5.com/company/blog/security-context-matters-for-agentic-ai https://community.f5.com/kb/technicalarticles/agentic-rag---securing-genai-with-f5-distributed-cloud-services/339571 https://www.youtube.com/watch?v=Pwb8k3LPKgI HaveIbeenPwned mail list leaked Infosec veteran Troy Hunt of HaveIBeenPwned fame is notifying thousands of people after phishers scooped up his Mailchimp mailing list. The list comprises around 16,000 records, and every active subscriber will be receiving a notification and apology email soon. Around half of these records (7,535), however, pertain to individuals who had unsubscribed from the list. Based on the original blog post, tiredness was a factor in the momentary lapse in judgment. The brief moment where the credentials were captured, the attacker was able to export the HaveIbeenPwned mailing list, suspecting an automated attack. In general, we should protect our digital fingerprints as extensively as we can. Having a 2nd factor of authentication may not be sufficient anymore with the advancement in phishing attacks. Use phishing-resistant MFAs where possible and stick to basics - if you are not sure, don’t click. https://www.troyhunt.com/a-sneaky-phish-just-grabbed-my-mailchimp-mailing-list/ https://www.theregister.com/2025/03/25/troy_hunt_mailchimp_phish/ https://www.pcmag.com/news/creator-of-haveibeenpwned-data-breach-site-falls-for-phishing-email https://www.cisa.gov/news-events/news/phishing-resistant-mfa-key-peace-mind NCSC taps influencers to make 2FA go viral In related news covering a much wider audience, the UK’s National Cyber Security Centre (NCSC) has hopped on the bandwagon to preach two-factor authentication (2FA) to the masses. It's the latest effort to improve the nation's cyber resilience as part of Stop! Think Fraud campaign launched in February 2024 under Rishi Sunak’s government, drafting in comedic sketch artists and Instagram personal finance gurus to promote wider uptake of security technologies. "To boost public awareness about the crucial benefits of enabling two-step verification on their most important accounts, we’ve partnered with popular social media influencers to amplify this vital message and encourage a wider audience to adopt secure online habits," https://www.theregister.com/2025/03/26/ncsc_influencers_2fa/ https://www.ncsc.gov.uk/collection/top-tips-for-staying-secure-online/activate-2-step-verification-on-your-email BlackLock ransomware gang pwned Cybersecurity vendor Resecurity is admitting to breaking into a notorious ransomware crew's infrastructure and gathering data it relayed to national agencies to help victims. Dubbed “BlackLock” (aka "El Dorado" or "Eldorado"), the ransomware-as-a-service (RaaS) outfit has existed since March 2024. In Q4 of last year, it increased its number of data leak posts by a staggering 1,425% quarter-on-quarter. According to independent reporting, a relatively new group has rapidly accelerated attacks and could become the most dominant RaaS group in 2025. Resecurity identified a vulnerability present at the Data Leak Site (DLS) of BlackLock in the TOR network - successful exploitation of which allowed our analysts to collect substantial intelligence about their activity outside of the public domain. The Resecurity blog is a good read as they walk thru their exploit, findings and communications with the BlackLock group. Another group mentioned was DragonForce - very similar in techniques and also pwned the BlackLock Data Leak Site. https://www.resecurity.com/blog/article/blacklock-ransomware-a-late-holiday-gift-with-intrusion-into-the-threat-actors-infrastructure https://www.theregister.com/2025/03/27/security_shop_pwns_ransomware_gang/ https://www.infosecurity-magazine.com/news/blacklock-2025s-most-prolific/ And that's the week that was I hope you find the security news I picked educational. We have a mix of vulnerabilities, a zero day, Security CoPilot AI agents and F5 Agentic AI, phishing, UK NCSC 2FA public awareness campaign and for the cherry on top, a ransomware gang getting pwned. For vulnerabilities and zero days, update your system and software as soon as possible to mitigate these. Implement WAF, Bot Defense or DDoS mitigations where possible and in anticipation of future vulnerabilities and application attacks. Ensure only trusted users and networks have access to your systems. If a system does not need to be exposed for public access, ensure that it is not. For phishing, stand up the human firewall - be vigilant on received emails and links. If unsure, don’t click. Use MFA/phishing resistant MFA. An MFA is better than "no MFA”. AI agents and Agentic AI - we might have been using it and we may not have known. If you own one, ensure to protect it and implement security controls - Think F5 Distributed Cloud. Till next time - Stay Safe and Secure! As always, if this is your first TWIS, you can always read past editions. We also encourage you to check out all of the content from the F5 SIRT.
BIG-IP TMOS Forensic Pointers
Notes on Forensics and Support Computer forensics is typically conducted by specialized firms that have all of the licensing and documented training needed to conduct said forensics. Typically in the United States, licensing requirements start with the firm being licensed for private investigations, and this also often requires individual forensic analysts to be licensed as private investigators. In some states, they may have additional licensing for computer forensics, and some states attach criminal liability to operating as a forensic analyst for hire without a license. Forensic analysts also need to be able to establish in court their own expertise. This is where documented training and experience comes in as a forensic analyst is only useful testifying in court as an expert witness. Because of these requirements, F5 Support is able to assist in forensic investigations as an advisor to either customers internal forensic analysts or hired outside forensic analysts. We can provide information about how F5 products work, how data is moved, stored, retrieved and sent on F5 products and other questions the forensic analysts have while conducting their investigation. UCS Files UCS Files are the configuration archive file for BIG-IP TMOS. They are a gzipped tar archive of configuration files and database dumps. We have previously gone over the contents of the file in an article. Since its a tar archive, it can be unpacked and examined and compared to previous versions using file tree comparison tools. K4423: Overview of UCS archives K13132: Backing up and restoring BIG-IP configuration files with a UCS archive K86995543: Identifying large files in UCS It is recommended that customers establish a mechanism to periodically capture UCS files and store them in case of an incident, many customers also create before-and after UCS files as part of their change management process. Passwords and passphrases on the system at rest and in a UCS file are encrypted using the Secure Vault feature. This prevents them from being decrypted without the master key being stored on the hardware. All units in a cluster of BIG-IP systems share the same master key. K73034260: Overview of the BIG-IP system Secure Vault feature K000149030: Copying the master key from the source to the target system when restoring a UCS archive QKViews The "Quick View" or QKView file was introduced in BIG-IP TMOS Version 9.0 as a snapshot of the running BIG-IP for support to analyze. At the time, it was a single zipped file with the output of a myriad of commands in it. In Version 10.0 it was expanded into what it is today, a gzipped tar archive full of configuration files, log files, database dumps and statistical (RRD) data from the system. The QKView process typically truncates log files and some other files to 5MB during collection, but that size can be set to the maximum allowed by executing the qkview command on the command line with "qkview -s 0", see link below for additional command line options. QKViews are typically viewed in the iHealth tool, which provides a graphical interface for examining logs, command output, and graphs and also runs a number of diagnostic heuristics against the QKView to identify common issues. K12878: Generating diagnostic data using the qkview utility K23928121: Overview of qkview command-line options K000134522: Reviewing journal data files from F5 Qkviews K60787205: Understand and manage sensitive data in a QKView The current article overviewing the contents of a QKView is K60787205 referenced above. In addition, we would like to point out the function of some files in the QKView: HWINFO: Basic hardware information, like the bas MAC address, cores, memory size and interfaces. PLATFORM: Hardware/Software platform type. Virtual editions are Z100 and vCMP are Z101. VERSION(.LTM): Software version information. *_module.xml: Files for each QKView module. Most of these correspond to either licensed modules like APM, AFM AWAF (named ASM in the QKView), specific daemons like postgres or QKView collection modules like proc and mcp. The proc_module.xml files contain information about running processes. qkview_run.data: Information about the QKView generation process for this QKView as well as error output, if applicable. skipped_files: Files that were skipped, typically because they were empty. The QKView should contain key files in home directories /home/<user> and /root. There are a couple of other directories that may be notable: /var/tmp/asm_tables_dump: Contains a partial dump of the MySQL tables AWAF/ASM uses for configuration and logging. /var/tmp/qkview-rrd: Contains the RRD data used to generate graphs in the BIG-IP TMOS user interface and also in iHealth when a QKView is loaded there. Daemons The BIG-IP TMOS system uses a large number of daemons to process traffic, update configurations and log events. Here are the key daemons on a BIG-IP system: tmm: Processes traffic and provides all LTM features except some health checks. bd: Processes web requests and responses for the Advanced WAF feature. avrd: Processes traffic for statistical and DoS detection features. bigd: Provides monitoring capability for monitoring pools and other traffic opbjects. big3d: Provides a communication channel for the GTM/GSLB feature. gtmd: Manages the GTM/GSLB feature. iprepd: Downloads IP Reputation updates when that feature is licensed. mcpd: Manages the BIG-IP configuration and updates various daemons when configuration changes are made. For a more complete list see: K48615077: BIG-IP daemons (15.x - 17.x) Disk Configuration BIG-IP systems that contain a RAID array utilize the software RAID feature in Linux. All BIG-IP TMOS systems use the Linux LVM system for storage management and utilize a "boot slot" abstraction, for example: Volumes that are shared across all boot slots: /dev/vg-db-vda/dat.maint.1 /dev/vg-db-vda/dat.share /dev/vg-db-vda/dat.appdata /dev/vg-db-vda/dat.log /dev/vg-db-vda/dat.swapvol Volumes that are specific to boot slot 1: /dev/vg-db-vda/set.1.root /dev/vg-db-vda/set.1._usr /dev/vg-db-vda/set.1._config /dev/vg-db-vda/set.1._var K14403: Maintaining disk space on the BIG-IP system The volumes on the BIG-IP TMOS system can be mounted in Linux if they are located in a virtual machine disk image, or if the RMA Disk Retention Option is purchased, by removing the disks from the BIG-IP system. K92469153: Mounting a BIG-IP VE disk volume for access on a linux VM It is also possible to use the Maintenance Operating System boot option to get a rudimentary Linux environment and copy disks using tools like dd and netcat, but this is not recommended because it may cause modifications to the disks while being performed. K14245: Overview of the recovery tasks performed from the MOS (11.x - 17.x) Memory Configuration The BIG-IP TMOS system uses normal Linux memory allocation for most processes with a couple of exceptions: tmm: TMM will allocate all the memory it will ever use on startup and internally manage its memory. bd: BD allocates an amount of memory on startup and internally manages that, up to a limit it will allocate more memory as needed, but it will not return that memory when freed in its internal allocator. Process memory can be copied live from the /proc filesystem, but the standard disclaimers about performing forensics on a running system applies. TMM Memory Configuration An overview of TMM memory allocation is linked below and the current state of memory allocation can be viewed using the tmsh commands "show sys memory", the output of which is also available when viewing a QKView on iHealth. We are currently working on determining what additional information related to TMM memory allocation can be publicly documented. K16419: Overview of BIG-IP memory usage Coring TMM A core file can be generated for TMM and that file examined, in F5 Support. Core files will only be able to be examined by F5 Engineering Services or F5 Product Development, and we typically do not examine core files from compromised systems. The process memory can be examined by a forensic analyst using GDB or other tools, but the proprietary symbol files cannot be provided to customers. K15569543: Producing a TMM diagnostic core file Bash and TMSH History Two shells are available to users with access to BIG-IP TMOS, the native shell is the TMSH shell, which provides commands to add, view, change and delete configuration, view statistics and log files and other administrative tasks. Since TMSH supports the complete access control scheme of BIG-IP TMOS, it can be made available to users of all access levels. The second shell available is the Bash shell, referred to as the Advanced Shell in the documentation, this shell is only available to users with administrator-level access and runs as a superuser when accessed. K000135228: Enabling bash or tmsh access on the BIG-IP system Bash history is located in the typical .bash_history file in the users directory. Commands are timestamped using the unix epoch. TMSH history is located in a file called .tmsh-history-<username> in the users directory. Commands are timestamped with the system time. User directories are /root for root and /user/<username> for other users. Log Files Logs are located in /var/log, this includes many of the typical log files for a Linux system. Some additional logging is done by systemd in /var/log/journal/* information on how to view these log files is available in K000134522. An overview of the different log files is available in K16197, but we would like to review some pointers here: /var/log/secure and /var/log/audit will contain information about logins to the system and some information about changes. Sometimes changes in the GUI are logged in /var/log/tomcat/catalina.out especially when an error happens. Changes made with iControl REST may be logged to /var/log/restjavad*. Changes sometimes generate messages in /var/log/ltm*. Apache httpd is typically not configured to log accesses, log files are in /var/log/httpd/, errors are still logged. Some Linux daemons log to /var/log/daemon.log including BIND/named. K16197: Reviewing BIG-IP log files K000134522: Reviewing journal data files from F5 Qkviews The BIG-IP TMOS system typically rotates log files once a day when cron.daily is executed. By default, log files are retained for 8 days. These settings are configurable depending on how much space is allocated for log files and how much logging is being generated. If the log volume becomes over-full, the log rotation process may be run early to attempt to purge log files to prevent it from becoming completely full. If the log volume becomes completely full, some daemons, in particular httpd may fail to run correctly, so care must be taken with logging to the local filesystem. K13367: Managing log files on the BIG-IP system (11.x - 17.x) K29900456: Troubleshooting issues with log rotation and archiving F5 encourages customers to configure remote logging for the system so that log files are harder to compromise by an attacker and so that if there is a catastrophic failure of the system, some data can be examined to determine what happened. K13080: Configuring the BIG-IP system to log to a remote syslog server (11.x - 17.x) Malware on BIG-IP TMOS We have put together an article linked below describing the process the F5 SIRT uses to check for malware in QKViews. This may provide additional insights into a compromised system. There is also a support article linked below that provides guidance on how to deal with a compromised BIG-IP. How the F5 SIRT Looks for Malware K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system In Conclusion This article was put together to assist forensic analysts in examining the BIG-IP TMOS system. We will revise this article as needed when we have more information that can be incorporated into it or articles linked from it. The F5 SIRT team welcomes your feedback here on DevCentral. You can also provide feedback to individual articles in the knowledge base using the feedback tool at the bottom of each article or in the course of support cases with the F5 Support team. You can Contact the F5 SIRT by opening up a service request and/or follow the guidance on the Report a Vulnerability page.
