Question & Answer: Regarding CVE-2020-5902
Briefly... In our recent live-streams (DevCentral Videos on YouTube) we committed to doing our best to answer all of your questions; needless to say, there are a lot of questions. Our experts are doing the best they can to provide full and complete answers. In the interest of getting information to you in a timely manner we are releasing this Q&A Article with as many answers as we have at the moment. If you don't see your question/answer right away you can check back, subscribe to our General Article RSS Feed, or ask a question in the comments section here. Updates to this article are expected and will be published as they become available. Update Date Update notes July 14, 2020 Original publication. Edit to add hyperlinks to KB articles. July 17, 2020 Added "New From Ses 4&5" Section to General Support Questions. About Determining Vulnerability | General Support | New from Ses 4&5 | General Scope | Questions and Concerns About the Fix | About the Knowledge Base Article | Patch vs. Upgrade vs. Update | About Vulnerability Age | Regarding Breach Potential | Breach Detection | Breach Remediation | Cloud Mitigation | About Modifying the Disclosure Process | About F5s Strategic Response to the Vulnerability | Product Direction Questions About Determining Vulnerability How do I know if I’m vulnerable? All versions of BIG-IP software released before the fixed versions (11.6.5.2, 12.1.5.2, 13.1.3.4, 14.1.2.6, 15.1.0.4, and 16.0.0) are considered vulnerable. Versions 11.x, 12.x, 13.x, 14.x, and 15.x are vulnerable, including Engineering Hotfixes. Patches are available for all of these sustaining versions. As far as exposure to the issue goes, that really depends on what networks and users the TMUI is exposed to. Those systems that are exposed to the Internet are most at risk. If you’ve followed our long-standing guidance to restrict network access to TMUI, then the risk is significantly reduced but not completely eliminated. You must also take into consideration which systems or users can access the TMUI. There are plenty of PoC scripts out there, NMAP scripts, Metasploit module, etc. that will help detect the issue by exploiting it, but just know that any BIG-IP version that is not listed as “fixed” in the KB article is vulnerable. The F5 iHealth system has a heuristic that can alert customers on many issues, notably a version that is vulnerable to this issue and if the management port and/or self-IP use public IPs without a port lockdown. Are we good (safe) if the TMUI is restricted to private IP addresses and user access is limited? No. You may also be at risk from pivot/lateral attacks and inside. -------The following three questions have the same answer------- We don’t have any interface open to the public, but we have SSH access on all F5 interface, so [does that] still fall under this new vulnerability (CVE 2020-5902)? For Self-IP, if there is no public exposure (i.e., one-to-one NAT) and we use an "Allow Custom" option and specified a port, say 179 for BGP, do you consider the risk mitigated at that point? Is DNS/GTMs traffic is safe? Yes. This vulnerability only affects the port TMUI (httpd) is listening on. For most devices that is port 443, though it may vary - see https://support.f5.com/csp/article/K52145254 for details. Other ports such as SSH 22 are not affected. How can we determine if our TMUI is exposed to the Internet? If it is not exposed, is the risk less critical? iHealth has heuristics that look for a number of different pieces of evidence of interfaces exposed to the Internet, but it may not be able to identify all cases. How can I verify if my management interface is exposed to the Internet? iHealth has heuristics which look for a number of different pieces of evidence of interfaces exposed to the net, but it may not be able to identify all cases. If we just stop httpd and java then we are safe? This is not a supported action on BIG-IP and may make the system unstable or unusable. My TMUI has never been exposed to the Internet. Do I have to patch? No. However we strongly recommend you do because of the risk of pivot/lateral attacks and insider threats. What about Self IPs exposed in internet listening only on 4353 for iQuery traffic? This vulnerability only affects the port TMUI (httpd) is listening on. For most devices that is port 443, although it may vary. See https://support.f5.com/csp/article/K52145254 for details. Other ports such as 4353 are not affected. Would the landing page of an APM policy be vulnerable since it uses HTTPD? No; this vulnerability only affects the control plane and not the data plane. Only the TMUI httpd port is vulnerable. Do we have to upgrade our VIPRION chassis? Yes If my Self-IPs all have the state "Allow None" and my management is only available from a secure location, then is the issue mitigated? No, No. You may also be at risk from pivot/lateral attacks and inside threats. To restrict TMUI, is it sufficient to follow K13309 or would following K46122561 be more secure? F5 recommends upgrading to a fixed software version as soon as possible. Until that is possible, please follow the mitigation steps from https://support.f5.com/csp/article/K52145254. We are one of the bigger users of BIG-IP (about 200 boxes), so to update all of them would be very time consuming. Are the mitigating actions (restrict access & TMUI httpd) still valid? The currently provided mitigations work against all known attack vectors at this time, but this may change if new vectors are discovered. F5 encourages installing a fixed software version as soon as possible. We have already upgraded the [BIG-IP] boxes. Should we still restrict the self-IPs to allow none? Yes. Restricting access via Allow None or Allow Custom is best practice. We use BIG-IP APM, and it’s running on engineering hotfix. How should we proceed? If the IDs you have an Engineering Hotfix (EHF) for have been fixed in the release, you may simply upgrade. If not, please open a support ticket to request a new EHF on top of a fixed release. Have you considered hosting a vulnerable system like Cloudflare did with heartbleed? We have considered this but have chosen not to because sharing an instance with this vulnerability requires sharing a Linux distro. General Support Questions Do we need to take action, even if management interface is not exposed to Internet? a. What about for self-IP networks that are not exposed? b. Can you elaborate on the Port Lockdown recommendation? Yes. Please refer to https://support.f5.com/csp/article/K52145254: TMUI RCE vulnerability CVE-2020-5902. What about for Self-IP networks that are not exposed? F5 advises using Port Lockdown and Allow None or Allow Custom on all Self-IPs. You may also be at risk from pivot/lateral attacks and insider threats. The hardware/software compatibility matrix (K9476) only shows that I can't upgrade to a resolved version. What recommendations would you have for me? There is a clause in K9476 that states “Compatible software versions: BIG-IP software versions supported by the platform (oldest through newest). The software versions include their respective hotfixes and point releases, unless otherwise stated.” If httpd is configured to allow only certain addresses/network, would those settings be enforced upon even if we use Self-IP to access TMUI? F5 recommends following the mitigation steps advised in the Security Advisory. We have seen customers attempt to use 'httpd allow' directives have devices compromised and are not recommending this is a mitigation at this time. The Security Advisory says the best thing I can do is Upgrade. Do you have quick guidance on the most efficient way to get my boxes upgraded? What is our guidance? Both TMUI and iControl use httpd on port 443, but the vulnerability itself is not found in the code used by iControl. The Security Advisory says the best thing I can do is Upgrade. Do you have quick guidance on the most efficient way to get my boxes upgraded? What is our guidance? K84554955: Overview of BIG-IP system software upgrades. We have also published this video on patching an HA Device group: https://www.youtube.com/watch?v=wcaBq-_zjbs -------The following two questions have the same response------- About APM (SSL-VPN). Does it affect the login screen of SSL-VPN used by users? Is the VPN portal of BIG-IP APM affected? No; this vulnerability only affects the control plane and not the data plane. Only the TMUI httpd port is vulnerable. Can you point to the article that discusses how port 443 can be customized? There are a few articles that mention it; here is one https://support.f5.com/csp/article/K51358480. Do you recommend upgrading hosts or guests first on VIPRION systems? Upgrade hosts first, then guests. Make sure your standby guests are on the same hosts. Does a point release upgrade require a license renewal? Point releases, which end in x.x.x.x, are automatically encompassed by the major versions they are associated with. There is no need to update a license when upgrading to a point release. For more information about versioning, refer to K8986: F5 software lifecycle policy. Does that mean that a jump from 13.1 to, say, 14.2 could cause issues? The larger the jump, the greater the amount of technical debt to make up which increases the risk of there being an issue in the upgrade. A major version upgrade will have a higher risk, so F5 suggest testing this in a staging environment before attempting it in production. -------The following two questions have the same answer------- [Can] F5 ASM (WAF) not be configured to assign a blocking policy set against the TMUI? I have not seen a KB article on how to put TMUI behind BIG-IP ASM. Is that commonly documented? While this may be technically possible, it would be extremely unusual to have an ASM, or any WAF, in front of a management interface. If ASM were used to protect the management interface of a BIG-IP, the question then becomes, where is the management interface of the ASM, and is it protected? This is a complex solution that is not recommended. For the vulnerability on self-IPs, do you consider specifying the port (non-default) via "Allow Custom", a mitigation? (That is, does allowing specific ports reduce the risk?) For this vulnerability, the TMUI httpd port is the only one of concern, so as long as that is not allowed in Allow Custom the vulnerability is mitigated for that interface (the default being 443). In general, using Allow Custom and only allowing the ports that are needed for the given deployment will provide the highest level of security possible from Port Lockdown. How does Allow None Port Lockdown affect HA (high availability) configurations? Allow None will break HA configurations by blocking required traffic. Allow Custom can be used with port 1026 allowed. See the Security Advisory, which has links to additional documentation on Port Lockdown. Is it possible to use SSH tunneling for https traffic and only work httpd with localhost address on the F5 platform? While not common, it would be possible to SSH tunnel into the BIG-IP and then connect to TMIUI on localhost. However, you would still need to apply mitigations and restrict access to TMUI to avoid direct attacks. If the management IP is a private IP address and only reachable via a bastion host, will this impact the BIG-IP system? While this would lower the risk level, presuming self-IPs are also locked down, there is still the risk from pivot/lateral attacks and insider threats. F5 would still encourage installing a fixed software version and applying mitigations until that is possible. The question from my management is whether the vulnerability is limited to the management UI or if this is exploited via the iControl API. The official answer for platforms out of support is that you should upgrade to a supported hardware platform. No official fixes will be released for any platform that is no longer supported. That said, since you're already running an unsupported platform, you might consider unsupported software. F5 does not take steps to break upgrades on unsupported platforms and, generally, later releases from the same branch as a supported version will continue to work. For example, a system that was last supported on 12.1.3 would very likely run 12.1.5.2. It is not officially supported, but then, the entire platform is no longer officially supported. In the meantime, the mitigations provided in the Security Advisory can be applied immediately. -------The following two questions have the same response------- Is the web UI the only vector for this attack or are there risks because with this vulnerability in having other "default ports" open on self-IPs? People are saying this only affects the Web Admin Interface, isn't the issue with shell access (SSH) as well? This vulnerability only affects the port TMUI (httpd) is listening on. For most devices, that is port 443, though it may vary. See https://support.f5.com/csp/article/K52145254 for details. Other ports, such as SSH 22, are not affected. -------The following three questions have the same answer------- Mitigation is for TMUI httpd. What is the impact of that? Regarding mitigations, what is the impact, for example, if we use a self-IP for some automation via iControl? Will the mitigation keep that from working? So. the HTTP remediation does not block all TMUI requests, only requests with ";" or "hsqldb" in them. Is it expected that normal GUI or iControl calls do not use these strings in a request? There should be no impact on legitimate traffic from the mitigations. Except, of course, if Port Lockdown is used to block all access to httpd, which will stop all traffic including iControl REST. The KB article that describes Perform Clean Install implies its only for version 14. Does it cover version 15, too? Yes. The KB has been updated to 16.0.0. https://support.f5.com/csp/article/K13117 Is this an Apache issue, or is it specific to F5's customization of Apache? It is neither. Apache is simply in the path of the attack chain and provides a point where a mitigation may be applied. Is there an article for system hygiene checks? The Security Advisory contains information on Indicators of Compromise, and F5 would also recommend using iHealth. Does this vulnerability affects the F5 WAF [BIG-IP ASM] features? No; this vulnerability only affects the control plane and not the data plane. Only the TMUI httpd port is vulnerable. What about upgrading, for example, to 2000s? The 2000 series is still getting new software, so this should be a normal update. What is the procedure to apply hotfix on BIG-IP VE (virtual edition)? It is the standard update process. https://support.f5.com/csp/article/K84554955. Why does this patch update the F5 Edge Client? The fix for CVE-2020-5902 was included in software updates, which also contain other updates and fixes. Any change to the F5 Edge Client is unrelated to this CVE. With the notes about VIPRION B2250, does this includes vCMP guests that are hosted in VIPRION B2250 or just VIPRION TMUI itself? Only the Host installation is affected and not vCMP Guest instances. Added Questions from Live Q&A Sessions 4 & 5 Can you go through the manual reactivation process? Please see https://support.f5.com/csp/article/K7727. Does license reactivation affect traffic? It may; please see https://support.f5.com/csp/article/K7727. Have you seen many reports of the latest fixed branch of 14.x causing issues with VE editions and an old bug in which TMM interfaces cease passing traffic? Had this on a few upgrades now. We are not aware of any specific issues with version 14.x VE. If you are experiencing issues, please open a support ticket. If a vCMP guest doesn't let you log in after an upgrade, how can you interrupt boot from console to boot from the previous partition? From the vCMP hypervisor, use the vconsole utility to connect to the console of the guest. For more information, please see K15372. -------These two questions have the same answer------- If we are going to update to a major version and we will need to upgrade first to middle versions so applying images will be done to new non-boot partition image after image? Or on same created partition? For applying images will be applied to same new created partition and after apply the last image then we will make it as boot partition, correct? Follow any guidance associated with the version you are upgrading to. If you need to stop at a middle version, you will need to boot into that version before continuing on and upgrading again. This tech doc has release notes per branch that will help: https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-bigip-15-1-0.html#rns_upgrade. Is there an article for listing behavior changes between major software releases? The best source of this information would be the Release Notes published for each branch and updates: https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-bigip-15-1-0.html#rns_upgrade Is it best to upgrade manually or via BIG-IQ? Any cons in one method or the other? The “best” way to upgrade will depend on the user's needs, environment, and comfort level with the different methods. For large BIG-IP deployments, automation, such as through BIG-IQ or Ansible, is often less resource-intensive than manually upgrading each unit. For a smaller environment. it may be easier to manually update each unit rather than do the work required to set up the automation. When changing the boot location should you choose the option load configuration? This depends on whether you want to replace the configuration on the target boot location with something else. One reason you may want to do this is if configuration changes were made to the BIG-IP after new software was installed but before booting into the new software version. See K14704 for more information. Is there guidance on the possible ramifications of the compromise should be treated, e.g. certs, keys, secrets, credentials? Yes! Please see https://support.f5.com/csp/article/K11438344 Any step document for the VIPRION chassis? Performing a software update on a VIPRION chassis is generally the same as other F5 platforms. See K84554955 for these steps. For a VIPRION that is a vCMP host, see the “Upgrade Considerations” section of K15930 How necessary is the MD5 check? isn't the download TCP? What are we verifying with the MD5 check? The MD5 check is really a safety check before installation. Many users will download the files from F5 to a server, and then it may be transferred again to a USB stick, or to a jump box, etc., before finally being transferred to the BIG-IP. At any link in the chain, the file could be accidentally or deliberately corrupted, so it always pays to confirm the validity before attempting to install. If Self-IPs are set to Allow None, will the ICMP work for monitor traffic? Self-IPs set to Allow None will still respond to ICMP echo requests, for example, from a 3rd party monitoring system. Regarding the BIG-IP's own health monitors, setting a self-IP to Allow None will not disrupt any BIG-IP health checks that are sent using that self-IP. If the Self-IP of the HA is set to none, will it affect HA communication? Yes, it will. Please review the mitigation steps in K52145254 , including the documents it links to, which detail the ports you'll need to open in Allow Custom. Is it necessary to set HA Self-IP to Allow None? Yes. Restricting access via Allow None or Allow Custom is best practice. -------The following two questions have the same answer------- Post upgrade, the QKview & UCS is lost. Is there a way to retrieve those without going back to that old partition? QKVIEW and UCS were taken just before the upgrade, but a local copy of the UCS wasn't downloaded... How do we retrieve the pre-code upgrade UCS? Yes, it is possible. See https://support.f5.com/csp/article/K51222154 Regarding MD5 check, the mentioned command (md5sum --check <filename.md5>) in one of the articles - K8337 was not working. JFYI... The command “md5sum --check <filename.md5>” works, provided that <filename.md5> is in the same directory as <filename.iso>, and that the real filename of <filename.iso> matches the filename inside of the .md5 file. Should we remove the LocationMatch directives after loading fixed code? There is no need to remove the LocationMatch mitigations after installing the fixed software. We have few F5 boxes running Engineering Hotfixes. How would these patches impact such scenarios? If the IDs you have an Engineering Hotfix for have been fixed in the release, you may simply upgrade. If not, please open a support ticket to request a new EHF on top of a fixed release. We see lot of IPv6-wide IPs after upgrading to 12.1 from 11. How do we get rid of them ? It sounds like you’re seeing the behavior described in K99302411 or K72173643. What if you have an old platform, for example BIG-IP 6900, which is not compatible with the fixed software versions. What you must do to permanently mitigate this CVE? For platforms out of support you should upgrade to a supported hardware platform. No official fixes will be released for any platform that is no longer supported. In the meantime, we recommend you implement the mitigations provided in K52145254 immediately. Which 13.1.x.x patch is exactly for TMUI to protect against the outside world? 13.1.3.4 is the first version with the fixes on the 13.1.x branch. Why is the http only attribute introduced in an enabled form starting with v12? We believe this is in reference to https://support.f5.com/csp/article/K30524234 This was a product enhancement, a new feature, introduced in 12.0. General Scope Questions Which customers are impacted? All BIG-IP customers need to address this vulnerability by installing a fixed version of software. Because we know updating software can be time-consuming, we have vetted temporary configuration mitigations that are detailed in the KB article. What is the "101" of CVE-2020-5902? Please refer to https://support.f5.com/csp/article/K52145254 Can it be accessed through a port other than 443 or 22? This vulnerability only affects the port TMUI (httpd) is listening on. For most devices that is port 443, although it may vary, See https://support.f5.com/csp/article/K52145254 for details. Other ports, such as SSH 22, are not affected. Who is the most affected? Is it folks that have VEs deployed to the public clouds that are affected? All BIG-IP devices not running a fixed software version have the vulnerability. Devices with TMUI exposed to the Internet at at the highest level of risk. Questions and Concerns About the Fix Any concerns or risk with the B2250 hotfix? As noted in the Security Advisory, there is an issue with 13.1.3.4 on the B2250. Only the Host installation is affected and not vCMP Guest instances. -------The following three questions have the same response------- Are there any known issues with the mitigation patches? Do we know if the temporary mitigation is finished being tweaked? (We are in a hospital environment with 2 engineers applying temporary mitigations to 50 boxes only for the temp fix to get changed daily.) We applied mitigation actions three times on all our F5 (30 appliances). I hope it is the last update on mitigation. The currently provided mitigations work against all known attack vectors at this time, but this may change if new vectors are discovered. F5 encourages installing a fixed software version as soon as possible. Can we be confident if a port lockdown configuration has been applied on all exposed self-IPs that there is no other known vulnerability active than can overpass ACL protection? Port Lockdown blocks access at a low level. At this time there are no known bypasses or vulnerabilities in Port Lockdown. Do the current mitigations cause any service interruption other than to TMUI? There should be no impact on legitimate traffic from the mitigations. Except, of course, if Port Lockdown is used to block all access to httpd, which will stop all traffic, including iControl REST. -------The following two questions have the same response------- How do we know the upgrade actually fixes the vulnerability? Can it be tested? How do we prove it is fixed? F5 will not provide actual exploits to use in testing, but there are many available online. If the latest code has the fix, why can't that fix be isolated and delivered as a patch if it takes users an average of 100 days to upgrade? The 100-day stat is the average amount of timeit takes an organization (industry standard)to patch a critical vulnerability, not upgrade F5 products. Given the risk of pivoting attacks originating from other non-F5 critical remote code execution or command injection vulnerabilities that have been released in the past ~100 days (at a minimum) that may not be patched in your organization, we are recommending everyone apply the patch to mitigate the risk of internal pivoting attacks. F5's current product and support architecture is designed around block updates and not in-situ patching. Patching introduces a number of issues with regard to product stability and supportability, as it creates a matrix of possible product configurations depending on the base version and which patches have, and have not, been applied. A unified software update creates a known set of features and changes, which simplifies the testing required and helps ensure a more stable, reliable product. It also makes it easier to support customers and any given version has a known set of features and fixes. For those customers unable to immediately update their software, F5 has provided temporary configuration-based mitigations to address the vulnerability until an update can be performed. Version 16 was released just 2 day ago. How confident is F5 about its stability in terms of fixes 16.0.0 has the same code fixes that were included in the other fixed versions. We implemented the last two mitigation recommendations and now there is a third. What is F5's confidence that this mitigation will cover all the vulnerabilities while we work on patching our devices? The currently provided mitigations work against all known attack vectors at this time, but this may change if new vectors are discovered. F5 encourages installing a fixed software version as soon as possible. What is F5's confidence level of the patch/fixed code in terms of stability? At this time F5 has a high confidence level in the fix and we have seen no indicates of exploits on fixed software. The community at large has had access to the fixed code and knowledge of the CVE since June 30th and, while there have been bypasses to the temporary mitigation originally provided, there has been no indication of weakness in the code fix itself. What's F5's biggest concern at this point with respect to the vulnerability? F5's biggest concern is with our customers safety. Customers are at risk from this vulnerability, and we want to encourage them to upgrade to a fixed software version as quickly as possible. When will the workaround be permanent? The currently provided mitigations work against all known attack vectors at this time, but this may change if new vectors are discovered. F5 encourages installing a fixed software version as soon as possible. Questions About the Knowledge Base Article What is the specific knowledge base article? https://support.f5.com/csp/article/K52145254 BIG-IP Self-IP is what we use to access the BIG-IP GUI. If I do the Self-IP Allow None, would this take my management access away? Yes, if you use the Self-IP to access TMUI you must allow port 443 (or whichever port httpd is listening on). F5 strongly advises that this not be accessible from the Internet, but only from a access controlled network. Of course, you should upgrade to a fixed version as soon as possible and apply the mitigations in the meantime. In reference to K46122561 and considering HA F5 pairs, do I need to allow port 1026 as well? For guidance on which ports to allow, please see: K17333: Overview of port lockdown behavior (12.x - 16.x) K13092: Overview of securing access to the BIG-IP system K31003634: The Configuration utility of the Single-NIC BIG-IP Virtual Edition now defaults to TCP port 8443 K51358480: The single-NIC BIG-IP VE may erroneously revert to the default management httpd port after a configuration reload What will be the impact if I wish to change the Port Lockdown from Allow Default to Allow None? Allow None will block all ports on the Self-IP. For further guidance, please see: K17333: Overview of port lockdown behavior (12.x - 16.x) K13092: Overview of securing access to the BIG-IP system K31003634: The Configuration utility of the Single-NIC BIG-IP Virtual Edition now defaults to TCP port 8443 K51358480: The single-NIC BIG-IP VE may erroneously revert to the default management httpd port after a configuration reload Patch vs. Upgrade vs. Update Questions -------The following four questions have the same response------- You keep saying patches; do you mean “upgrade”? Can you clarify? Are we patching or upgrading? Everything I see says “upgrading.” Is the vulnerability fixed by patching? Or just by upgrading? Why are you using the word “patching”? I think it’s directly upgrading systems, right? As far as I know F5 does not have a separate hotfix, it is included in the image itself. Please correct me if I am wrong. When we say “patch” we do mean installing a fixed version of software. If we already upgraded the units to 14.1.2.6, do we need to apply the workaround as well? If you've already upgraded to a fixed version, then the vulnerability is resolved. You do not need to apply the httpd mitigation. Restricting access to Self-IPs and the management interface is always recommended as a best practice. Please explain how LocationMatch works. The LocationMatch mitigation is looking for patterns in the incoming requests that are found in exploits and blocking them at the httpd layer. For details on how LocationMatch works, please see the Apache documentation: https://httpd.apache.org/docs/2.4/mod/core.html#locationmatch. How is the patch different from the workaround? Does the patch also include the parameters from the workaround, then can we simply mitigate this vulnerability by applying workaround? As a high-level overview, there are three links in the attack chain: Apache (httpd), Tomcat, and F5's Java code. The code fix addresses all three layers and completely eliminates all known aspects of the vulnerability. The mitigation only addresses the first link in the chain, at httpd, by blocking directory traversal attempts and queries which exploit the vulnerabilities in the next two links. The initial httpd mitigation blocked the known exploit vectors. However, it has been necessary to update the mitigation as new vectors are discovered, and it may be necessary to further update if new vectors emerge. F5 encourages installing a fixed version of software as soon as possible to eliminate any risk with the mitigation being bypassed. What about versions that are not listed in the KB, such as version 11.1 to version 11.5? Should I patch the current version, or do I need to upgrade to version 11.6.x? Software that has reached End of Technical Support (EoTS) is not evaluated for vulnerabilities. It is believed this vulnerability has been present for a number of years, and all versions of BIG-IP software released prior to the fixed versions should be considered vulnerable. Questions About Vulnerability Age -------The following questions all have the same response------- How long has the BIG-IP been vulnerable with this? How long has this vulnerability existed in the code base? Has this vulnerability always been there? How long does this vulnerability date back? Was this an open issue on much older codes like v9? It is believed this vulnerability has been present for a number of years, and all versions of BIG-IP software released prior to the fixed versions should be considered vulnerable. (Note that software that has reached End of Technical Support (EoTS) is not evaluated for vulnerabilities.) Questions Regarding Breach Potential Does this [vulnerability] mean that my certificates might be compromised; in other words, are my private key now known to external parties? Note that all information on a compromised system should itself be considered compromised. This includes but is not limited to account credentials, keys, certificates, logs, and any other sensitive data. When restoring a compromised system, it is recommended to change or reissue these objects so as not to reuse compromised data. For exposed or potentially compromised devices, can we consider that restoring a UCS backup from 22 June 2020 is safe enough, or should we consider recovering from scratch? When deciding whether or not to restore from a configuration backup, each user must determine their own comfort level. If there is any uncertainty or doubt as to the integrity of the backup, F5 would not recommend using it. If the configuration backup was stored on a host suspected of being compromised, the backup should itself be considered as potentially compromised. From the exploit, were the filestore files such as private keys able to be taken from the box? All information on a compromised system should itself be considered compromised. This includes but is not limited to account credentials, keys, certificates, logs, and any other sensitive data. When restoring a compromised system, it is recommended to change or reissue these objects so as not to reuse compromised data. Is it possible for an attacker to leverage this vulnerability to export data from a tcpdump? Yes. A full exploit gives the attacker root access and they can do anything root can do, including running tcpdump. Originally, the report was that over 8,000 devices were exposed to the Internet. Do you know how many of those were Cloud deployed or on premises? F5 has seen these numbers reported in the public, from Shodan reports, but we do not have additional information on this datapoint to report at this time. Is there any vector for the attack from an APM perspective, where the BIG-IP would be hosting content such as login pages? No; this vulnerability only affects the control plane and not the data plane. Only the TMUI httpd port is vulnerable. Breach Detection Questions Where do I go to find out if I have been exploited? Regarding TMUI RCE vulnerability, see CVE-2020-5902. For indicators of compromise, see https://support.f5.com/csp/article/K52145254. For [general] considerations and guidance when you suspect a security compromise on any BIG-IP system, see https://support.f5.com/csp/article/K11438344. What indicators of compromise (IOCs) should I look for? See the Security Advisory Indications of Compromise https://support.f5.com/csp/article/K52145254. We know that advanced attackers will hide their tracks to maintain system access so, unfortunately, in some cases there may not be any noticeable indicators of compromise. Our KB article recommends some things to look for but I would say any client making a request with “..;” should be suspect along with any client requesting “hsqldb” in the path. Those are almost a certain indication of malicious behavior. There are also some reliable heuristics that can detect a compromised BIG-IP running malware, but can’t catch 100% of the cases. -------The following five questions have the same response------- Are there markers we can look for on the systems to see if the system has been exploited? Can I see some kind of log to know if someone exploited the vulnerability on my F5? Can we search for specific IOCs in BIG-IP logs? If so, which ones? Is there an article on how to tell if our BIG-IP box has been breached? Are there any attacker TTPs that you guys discovered? See the Security Advisory Indications of Compromise https://support.f5.com/csp/article/K52145254. Has F5 discovered any additional TTPs that aren't noted in the https://support.f5.com/csp/article/K52145254 article? We endeavor to add all credible IOCs/TTPs to the K52145254 article. Does iHealth find such kind of vulnerability and provide recommendations? iHealth will alert on this CVE for any unfixed version of software. There are additional heuristics in iHealth which may flag other risk factors. I cannot access my BIG-IP system via the Configuration Utility and attempts to restore it fail. Does this mean I'm compromised? How can I tell? This issue resulted in article K03275122: Unable to access BIG-IP system via Configuration Utility and attempts to restore it fail. Blank grey page or 404 error is seen instead of login page. Can F5 share more details on the threats that are targeting the F5 TMUI CVE-2020-5902/5903? Numerous exploit scripts are now available “in the wild.” If [the system is] Internet connected and hasn't at least had the mitigation applied, is it safe to assume by this stage that the system is most likely compromised? If your BIG-IP system has TMUI exposed to the Internet and it does not have a patched version of software installed, there is a high probability that it has been compromised and you should follow your internal incident response procedures. Breach Remediation Questions -------The following two questions have the same response------- What considerations do I need to keep in mind when restoring from pre-exploit backups? If I assume my BIG-IP is compromised and I have pre-exploit backups, what steps do I need to consider as I upgrade and restore, outside of changing credentials and revoking certificates? When deciding whether or not to restore from a configuration backup, each user must determine their own comfort level with the backup they’re reverting back to. If there is any uncertainty or doubt as to the integrity of the backup, F5 would not recommend using it. If the configuration backup was stored on a host suspected of being compromised, the backup should itself be considered as potentially compromised. As general guidance, CVE-2020-5902 became public knowledge on Tuesday, June 30th, 2020. While the first exploits were seen a few days later, F5 would recommend using a backup that was taken before the CVE was published, out of an abundance of caution. For customers with heightened concern, note that one of the underlying issues in the CVE had an ID opened on April 2, 2020. Because it was reported to F5 by an external researcher, the issue was known outside of F5 earlier than April 2, 2020. This vulnerability has existed in the code for a number of years, and there is no way to know if it had been independently discovered by any other parties. Users must make their own determination as to their comfort level when restoring from backups. Note that all information on a compromised system should itself be considered compromised. This includes but is not limited to account credentials, keys, certificates, logs, and any other sensitive data. When restoring a compromised system, it is recommended to change or reissue these objects so as not to reuse compromised data. -------The following two questions have the same response------- I want to wipe out all my HD1.xx partitions as well as /appdata and start all over, but I am having problems seeing exactly how to do this from a new ISO file. Your thoughts? if I manually ERASE /appdata and my unused HD1.xx partitions, will installing a new version using image2disk and an ISO rebuild everything? Please refer to https://support.f5.com/csp/article/K13117. If the BIG-IP box is already compromised, will a version patch update help to mitigate or block this CVE? If the system is already compromised, then updating the software or applying mitigations will not help. The compromise will need to be addressed by following your internal procedures. The knowledge base support article https://support.f5.com/csp/article/K52145254 has some tips and links to additional guidance. What is your recommended remediation against insider threat (that is, against people already inside the network)? 1) Install the fixed software as soon as possible to eliminate the vulnerability. 2) Apply the httpd mitigation to prevent unauthenticated insiders from exploiting the vulnerability. 3) Restrict access to Self-IPs and the management interface to reduce the number of users who may be able to attack the system. ------The following two questions have the same response------- What should I do if an F5 [system] was the victim of an attack? I have an F5 with an unknown user that was created 2 days ago and tmsh functions are disabled (list, create, modify). What would be the best action to follow on a compromised system? Our advice is contained in https://support.f5.com/csp/article/K11438344. Who can I call at F5 if I think I’ve been breached? You can open a support ticket here: https://support.f5.com/csp/my-support/service-request, but also see https://support.f5.com/csp/article/K11438344. Cloud Mitigation Questions -------The following three questions have the same response------- How do we mitigate on Azure? Are the versions with the fix already available for public cloud platforms? Where are the instruction to patch / upgrade for azure hosted? Why are the Cloud providers not up to date? I would expect them to be able to turn this around much faster for a zero-day exploit. It’s almost a week now. Releases for all cloud platforms are now either out or are working through their respective approval process with individual cloud providers. Questions About Modifying the Disclosure Process I think any 10-point score should be covered up and not documented in release notes. Would not highlight to attackers then where to go probing. CVSS is an industry standard and F5 believes in transparency. We believe hiding this score would be a disservice to our customers and provide only a false sense of security as the black hat community does not act based on the score—as evidenced by recent exploits of other vendors' CVEs which were released without a score. Furthermore, once published, NVD provides public scores for all CVEs without consulting the vendor. Which organization was it that made the vulnerability public? It would be nice if feedback could be given them to not release right before the start of a major holiday. That makes it hard to react. F5 published the vulnerability. It was originally reported to F5 by Mikhail Klyuchnikov of Positive Technologies, and F5 worked with them on the disclosure. We had a commitment to publish within 90 days, which had the end of June as its deadline. While we had hoped to publish earlier, due to issues in getting all of the releases out, June 30th was the earliest possible date of publication. With the fixed software released and available to reverse engineer, it would be too risky to delay announcement to our customers. The timing was unfortunate, but we felt it was the appropriate action to take. Would F5 consider modifying their disclosure process to first notify F5 clients of patches and fixes prior to public disclosure? We have heard this feedback and there will be internal discussions on the issue. Questions About F5’s Strategic Response to the Vulnerability -------The following four questions have the same response------- How do we proactively stop the next vulnerability? Some would consider this type of vulnerability pretty trivial. What is being done at F5 to ensure that new and existing vulnerabilities of this flavor are being caught before release? Is F5 planning to invest on a better penetration test bounty program? Will F5 be considering anti-malware on these "virtual" appliances? Something like CrowdStrike Falcon, SentinelOne, or Cylance. We have comprehensive security practice across the company. From secure coding practices, to rigorous testing, internal and external audits to the way we handle vulnerability management and disclosure. With the acquisition of Shape Security earlier this year, we are accelerating the integration of their world-class security approaches, drawing on experience from Google, the Department of Defense and a number of other organizations that have led the way in the creation of industry best practices. That said, we continue to review our processes internally to determine where we can do better for our customers, both in how we build our products, and our approach to vulnerabilities. Product Direction Questions Any thoughts on moving TMUI off Apache/Tomcat to NGINX? There are no plans for replacing TMUI in BIG-IP at this time. About Determining Vulnerability | General Support | New from Ses 4&5 | General Scope | Questions and Concerns About the Fix | About the Knowledge Base Article | Patch vs. Upgrade vs. Update | About Vulnerability Age | Regarding Breach Potential | Breach Detection | Breach Remediation | Cloud Mitigation | About Modifying the Disclosure Process | About F5s Strategic Response to the Vulnerability | Product DirectionTraffic Management User Interface Vulnerability: The Fix and Temporary Mitigation Options
*** Updated July 8, 2020 *** A new Remote Code Execution vulnerability announcement that affects several versions of BIG-IP was just released on June 30, 2020. Remote Code Execution (RCE) vulnerabilities are arguably the most severe vulnerabilities that exist...because an attacker can take over your system and start running commands (modifying files, disabling services, executing code, etc). This specific RCE allows unauthenticated attackers, or authenticated users, with network access to the Traffic Management User Interface (TMUI), through the BIG-IP management port and/or Self IPs, to remotely execute system commands. This vulnerability does not affect the data plane, but the control plane is affected and could lead to complete system compromise. The Fix Because of the severity of this vulnerability, we recommend immediately installing the latest patched version in order to protect your BIG-IP system from compromise. The following is a table of known vulnerable BIG-IP versions and the associated fixed versions: Temporary Mitigation Options One of the primary reasons this vulnerability is exploitable is that admins open the management port of their BIG-IP to the Internet. This is not a good idea...you should never expose your management port (TMUI) to the open Internet. If you do, you are allowing attackers an open invitation to do all kinds of bad things to your BIG-IP system. Certainly, your BIG-IP system will process traffic from the open Internet via Virtual Servers, etc configured on your BIG-IP system, but the administrative control of your system (TMUI) should be locked down to secure, internal access only. The other reason this vulnerability is exploitable is that Self-IPs are not locked down properly on BIG-IP systems. By default, Self-IPs are locked down (Port Lockdown set to "Allow None") but some admins change this setting to open certain ports for some Self-IPs. If a Self-IP port is open to the default TMUI port of 443 (or, in some cases, 8443), then that Self-IP will have access to the TMUI and an attacker could gain access to your system via a compromised Self-IP that is then allowed to access the TMUI. The good news is that all of this is configurable to lock it down properly. First, as I said before, never expose your management port (TMUI) to the open Internet. Next, lock down your Self-IP ports to "Allow None"...or, if you really must open ports for your Self-IP then be sure to not open the port for your TMUI. Also, you can configure which port the TMUI listens on. So, there are a multitude of options for you to configure your BIG-IP system properly in order to avoid exploitation of a significant vulnerability like this. Other Important Things to Consider I would recommend using the F5 iHealth service to diagnose vulnerabilities on your BIG-IP system. iHealth is awesome...you simply upload a QKView file from your BIG-IP and then iHealth runs that file against a multitude of configuration recommendations, security checks, etc and then it gives you a list of recommendations of what to change on your BIG-IP system and how to change it all. You can't go wrong! The Internet is a complex place, and we all want to help each other thrive as we navigate these inevitable hurdles of security issues. Use the comment section below to ask any questions or provide feedback for others who might be looking for great advice from this amazing community. We are all in this together...thanks for being the best community in the world! For more information, check out the video below where I discuss more about this vulnerability and how to mitigate it.
