How the F5 SIRT Looks for Malware
Introduction Welcome to another article by F5 SIRT Security Engineer Kyle Fox. Today we explore the ways that F5 SIRT looks for malware on suspected compromised systems. We will explore some iHealth heuristics and one of the common ways malware is hidden on systems. First, we need to get some administrative matters out of the way. The F5 SIRT and F5 Support are not able to provide computer forensics services. There are a number of reasons for this including licensing, certifications and legal issues. We can assist in finding indicators of compromise, identifying suspect logs or data in QKViews, and answering questions from forensic analysts about F5 products and how they work. If you suspect that you have a compromised F5 system, you can Contact the F5 SIRT by opening up a service request and/or follow the guidance in K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system. iHealth Heuristics H511618: Malware may have infected the BIG-IP system This heuristic compares process names with a list of known-good process names on BIG-IP systems. It does not always alert when malware has infected the system and also sometimes it alerts on expected processes such as networked FIPS HSM drivers and others. If it is seen, more investigation may be needed. There are three other heuristics that also help identify how a system was compromised, or if your looking to secure a system, identify how to secure it: H444724: The management interface is allowing access from public IP addresses This heuristic is checking the IP addresses of connections to the management interface and looking for public IP addresses. This will identify when the BIG-IP is getting such access, even if such connections are being IP or port forwarded to a private IP address on the BIG-IP. H458565: Public IP addresses configured as a BIG-IP Self IP This heuristic will identify when public IP addresses are configured as a Self-IP and not properly secured using port lockdown. See K17333: Overview of port lockdown behavior (12.x - 17.x) for details on Port Lockdown. H727910: The configuration contains user accounts with insecure passwords This heuristic looks for default and common passwords in the user accounts. Typically this will hit on admin or root being set to something like "admin", "default", or "root". But there are a limited number of other common passwords it will look for. Remember to change the password on admin and root to a stronger password even if you disable those accounts. Typical Malware What the F5 SIRT has observed is the typical malware we see does flooding, cryptocurrency mining, or both. This malware is typically named with random characters and is hidden by deleting the executable after it is run. Attackers may stop at deploying such malware or may continue to try and infiltrate the victims networks from there. Typically they avoid implanting malware when using the F5 system to pivot into a network. One way we can look for this malware without using the iHealth heuristics is by looking for executables that are deleted, in the lsof command. They look like this: From iHealth -> Commands -> Standard -> UNIX -> Networking -> lsof -n (text only) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME [...] xDwJc 23367 root cwd DIR 253,12 4096 106664 /var/service/restjavad xDwJc 23367 root rtd DIR 253,9 1024 2 / xDwJc 23367 root txt REG 253,9 282 30733 /tmp/xDwJc (deleted) xDwJc 23367 root 0r CHR 1,3 0t0 1028 /dev/null xDwJc 23367 root 1r CHR 1,3 0t0 1028 /dev/null xDwJc 23367 root 2r CHR 1,3 0t0 1028 /dev/null xDwJc 23367 root 3u sock 0,7 0t0 2873878973 protocol: TCP xDwJc 23367 root 4u a_inode 0,10 0 6105 [eventfd] So, lets review what we see here. The main thing we are looking for is the line with the FD column named txt. This is the executable file the process is running from, and we see in the NAME column it says (deleted). In Linux, a file exists after it’s been deleted as long as it’s still open by one or more processes, so deleting the file does not cause the process to end. Malware writers have used this to hide malware from antimalware software that scans the filesystem for malicious executables. Let’s review all the columns in this listing: COMMAND - The command being executed. PID - The process ID, we will use this later. USER - The user the command is running as. FD - The file descriptor type. Here we see: cwd - Current working directory. rtd - Root directory. txt - Program executable. <number><letter> - file descriptors. The letter can be r for read, w for write and u for both. TYPE - Type of filesystem node associated with the file: DIR - Directory. REG - Regular File. CHR - Character special file, in this case a device called null. sock - A socket. a_inode - An anonymous inode (file system node) in this case a connection to the event forwarder daemon. DEVICE - The device numbers for the file. NODE - Node number of the file. NAME - The file name. From what we can see, the working directory is /var/service/restjavad. This strongly implies that the malware got on the system using an exploit for iControl REST, but it may also have gotten on the system through a longer route through some other part of TMUI. If you have a forensic analyst or otherwise want to analyze the malware, you can copy the executable even though it’s deleted by copying from /proc/<PID>/exe, in this case: cp /proc/23367/exe /var/tmp/malwarefile Typical Ways Intruders Break In In most cases of compromised systems, the attackers got in because the system had an Internet-facing Self-IP that was not secured. We recommend that you do not use Internet-facing Self-IPs for HA or administration, and to set Port Lockdown on Internet-facing Self-IPs to Allow None. See K17333: Overview of port lockdown behavior (12.x - 17.x) for details on Port Lockdown. Attackers typically use TMUI or iControl rest vulnerabilities to gain access to BIG-IP systems. The two major ones we see are CVE-2020-5902 and CVE-2022-1388, but many other vectors exist, so organizations should strive to keep their BIG-IPs software up to date. Attackers also try default passwords, and we have noted some systems that have default passwords in production. Make sure to change the admin and root passwords to strong ones on your BIG-IPs, even if you disable those accounts. Attackers who are trying passwords often will look for open SSH ports to try them against, so the same guidance above of not exposing TMUI/REST to the internet applies to SSH. Outro I hope this was helpful. If your dealing with a compromised system, you may find my article on Inspecting a UCS file from a compromised BIG-IP to be helpful in recovering from a compromise. Finally, again, if you suspect that you have a compromised F5 system, you can Contact the F5 SIRT by opening up a service request and/or follow the guidance in K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system.
iHealth Upgrade Advisor: Making upgrades a little easier
Whether it is upgrading the firmware on a switch, the OS on a server, an important business application or the software on a BIG-IP, performing upgrades is something that makes almost all IT Admins and Network Engineers nervous. We’ve learned from (sometimes painful) experience that things don’t always go as planned. Good preparation greatly increases the likelihood that an upgrade will be successful, which is why F5 has created the iHealth Upgrade Advisor. Its goal is to provide an additional service from F5 that will complement your existing upgrade preparations, increasing the predictability of the upgrade while reducing your upgrade time. The iHealth Upgrade Advisor service provides a way for users to gain insight into potential issues with a BIG-IP upgrade before they attempt the upgrade. It provides guidance that is specific to a BIG-IP based on its configuration, the version of software it is currently running and the version you are planning to upgrade to. When an issue can be avoided by making a configuration change prior to upgrading, the Upgrade Advisor will tell you exactly what to change. For some issues, it will list the corrective actions to take after the upgrade. Demo Video This short video demonstrating the Upgrade Advisor shows you how to use it and some examples of the guidance it provides. Accessing the Upgrade Advisor The next time you are preparing to upgrade a BIG-IP, login to ihealth.f5.com, upload a .qkview file from that BIG-IP and then view the qkview after iHealth has analyzed it. The Upgrade Advisor can be accessed by clicking on its tab in the left-hand menu. Simply select the version of BIG-IP you are planning to upgrade to in the advisor and review the results. Here is a screenshot of the Upgrade Advisor: Give it a Try Try out the F5 upgrade Advisor today and let us know what you think using the feedback option (circled in red on the right side of the screenshot above).
Enterprise Manager VS Big IQ
I used to be under the assumption that BigIQ was replacing Enterprise Manager, but I am not sure that is the case. What is the the real differences between the products because they seem to have overlapping feature sets and I am not sure why one customer would pick one appliance over the other. I see that there are different options (or maybe modules) for BigIQ, but I have no idea what its purpose is in relation the EM.
iHealth API returns 404 for BigIP calls
I can access anything else in the iHealthAPI except for the BigIP information. Every calls returns a 404... Including the same calls they are using in here - I am trying to grab the hostname out of the system information. Also, the hostname does show up in the QKView on iHealth. urls tried: 'https://ihealth-api.f5.com/qkview-analyzer/api/qkviews/{}/bigip' 'https://ihealth-api.f5.com/qkview-analyzer/api/qkviews/{}/bigip/system' 'https://ihealth-api.f5.com/qkview-analyzer/api/qkviews/{}/bigip/serial_numbers/chassis/' 'https://ihealth-api.f5.com/qkview-analyzer/api/qkviews/{}/bigip/serial_numbers/' With the {} replaced with the qkview ID. Can someone else check to see if it works? Thank you!
irule http to https redirect with a little nuance
Hi Everyone, I was hoping for a hand, I'm working on a irule that says: if you reach http://sysdev redirect to https://sysdev.x.com/login.aspx otherwise, if someone has a bookmark to http://sysdev/form1.aspx redirect to https://sysdev.x.com/form1.aspx So this is what I have: when HTTP_REQUEST { if { [HTTP::uri] == "/" } { HTTP::redirect https://sysdev.x.com/login.aspx } else { HTTP::redirect https://sysdev.x.com[HTTP::uri] } } Would this work? I cant find much documentation on 'else' only on 'elseif' - any help would be appreciated.
Upload QKviews to Internet via PROXY
Hello, I am trying to use the new function in V13 on F5 to upload the qkviews to ihealth. As we use a proxy for internet access I set it up with the: tmsh modify /sys db proxy.host value 10.x.xx.xxx modify /sys db proxy.port value 80 modify /sys db proxy.protocol value http We do not use authentication from this host. So user & password were not set. We tried access with a curl on the CLI to www.google.com and it worked fine. We get the following errors on the GUI qkview: Your device may not have access to the Internet. Verify that you can access api.f5.com and ihealth-api.f5.com and try your upload again. I do not see any traffic on a capture of this. Am i mssing something ? Thx in advance, LucF5 BIG_IP 3900 (C106) - Software Upgrade Advisor not working
Hello, I need to upgrade a 3900(C106) from version 11.5.4 HF2 to 12.1.3.7 I have uploaded the QKView to the iHealth tool to perform a Software Upgrade advisor but it prompts the following message: "You are already running the latest version". The menu to select to version to be upgraded to does not appear I have checked this model can be upgraded to this version in the compatibility matrix, in fact, in the Diagnostics section of the Status at the iHealth the following Upgrade options appear. Upgrade Options: 11.5.4.HF4(hotfix) 11.5.9(stability release) 12.1.3.7(latest release) Any idea why this is happening? Thankscannot access ihealth report
Hi, I want to ask regarding report in f5 ihealth site. Last month i can access my report in ihealth site. Right now i cannot access it. It say Request Cannot Be Completed The requested URL was rejected. Incident ID: 6183918755495845809 i have tried to clear cookie, clear cache and reupload the qkview. Is there any solution or workaround so i can access my report?iHealth report using case number
Currently we try to to upload iHealth report curl -i --user-agent "F5-EM/3.0" --insecure --cookie path_to_file/LoginCookie.txt --cookie-jar path_to_file/UploadCookie.txt -o - -F qkview=@path_to_file/IHEALTH_REPORT.tar.gz -F 'visible_in_gui=True' -F 'f5_support_case=case_no' https://ihealth-api.f5.com/qkview-analyzer/api/qkviews We have validated all the files are in the corresponding path. But it is not working. Kindly help us. We are currently using V11, V12 and V13 versions, let us know the generic way for all the devices.
