How the F5 SIRT Looks for Malware
Introduction Welcome to another article by F5 SIRT Security Engineer Kyle Fox. Today we explore the ways that F5 SIRT looks for malware on suspected compromised systems. We will explore some iHealth heuristics and one of the common ways malware is hidden on systems. First, we need to get some administrative matters out of the way. The F5 SIRT and F5 Support are not able to provide computer forensics services. There are a number of reasons for this including licensing, certifications and legal issues. We can assist in finding indicators of compromise, identifying suspect logs or data in QKViews, and answering questions from forensic analysts about F5 products and how they work. If you suspect that you have a compromised F5 system, you can Contact the F5 SIRT by opening up a service request and/or follow the guidance in K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system. iHealth Heuristics H511618: Malware may have infected the BIG-IP system This heuristic compares process names with a list of known-good process names on BIG-IP systems. It does not always alert when malware has infected the system and also sometimes it alerts on expected processes such as networked FIPS HSM drivers and others. If it is seen, more investigation may be needed. There are three other heuristics that also help identify how a system was compromised, or if your looking to secure a system, identify how to secure it: H444724: The management interface is allowing access from public IP addresses This heuristic is checking the IP addresses of connections to the management interface and looking for public IP addresses. This will identify when the BIG-IP is getting such access, even if such connections are being IP or port forwarded to a private IP address on the BIG-IP. H458565: Public IP addresses configured as a BIG-IP Self IP This heuristic will identify when public IP addresses are configured as a Self-IP and not properly secured using port lockdown. See K17333: Overview of port lockdown behavior (12.x - 17.x) for details on Port Lockdown. H727910: The configuration contains user accounts with insecure passwords This heuristic looks for default and common passwords in the user accounts. Typically this will hit on admin or root being set to something like "admin", "default", or "root". But there are a limited number of other common passwords it will look for. Remember to change the password on admin and root to a stronger password even if you disable those accounts. Typical Malware What the F5 SIRT has observed is the typical malware we see does flooding, cryptocurrency mining, or both. This malware is typically named with random characters and is hidden by deleting the executable after it is run. Attackers may stop at deploying such malware or may continue to try and infiltrate the victims networks from there. Typically they avoid implanting malware when using the F5 system to pivot into a network. One way we can look for this malware without using the iHealth heuristics is by looking for executables that are deleted, in the lsof command. They look like this: From iHealth -> Commands -> Standard -> UNIX -> Networking -> lsof -n (text only) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME [...] xDwJc 23367 root cwd DIR 253,12 4096 106664 /var/service/restjavad xDwJc 23367 root rtd DIR 253,9 1024 2 / xDwJc 23367 root txt REG 253,9 282 30733 /tmp/xDwJc (deleted) xDwJc 23367 root 0r CHR 1,3 0t0 1028 /dev/null xDwJc 23367 root 1r CHR 1,3 0t0 1028 /dev/null xDwJc 23367 root 2r CHR 1,3 0t0 1028 /dev/null xDwJc 23367 root 3u sock 0,7 0t0 2873878973 protocol: TCP xDwJc 23367 root 4u a_inode 0,10 0 6105 [eventfd] So, lets review what we see here. The main thing we are looking for is the line with the FD column named txt. This is the executable file the process is running from, and we see in the NAME column it says (deleted). In Linux, a file exists after it’s been deleted as long as it’s still open by one or more processes, so deleting the file does not cause the process to end. Malware writers have used this to hide malware from antimalware software that scans the filesystem for malicious executables. Let’s review all the columns in this listing: COMMAND - The command being executed. PID - The process ID, we will use this later. USER - The user the command is running as. FD - The file descriptor type. Here we see: cwd - Current working directory. rtd - Root directory. txt - Program executable. <number><letter> - file descriptors. The letter can be r for read, w for write and u for both. TYPE - Type of filesystem node associated with the file: DIR - Directory. REG - Regular File. CHR - Character special file, in this case a device called null. sock - A socket. a_inode - An anonymous inode (file system node) in this case a connection to the event forwarder daemon. DEVICE - The device numbers for the file. NODE - Node number of the file. NAME - The file name. From what we can see, the working directory is /var/service/restjavad. This strongly implies that the malware got on the system using an exploit for iControl REST, but it may also have gotten on the system through a longer route through some other part of TMUI. If you have a forensic analyst or otherwise want to analyze the malware, you can copy the executable even though it’s deleted by copying from /proc/<PID>/exe, in this case: cp /proc/23367/exe /var/tmp/malwarefile Typical Ways Intruders Break In In most cases of compromised systems, the attackers got in because the system had an Internet-facing Self-IP that was not secured. We recommend that you do not use Internet-facing Self-IPs for HA or administration, and to set Port Lockdown on Internet-facing Self-IPs to Allow None. See K17333: Overview of port lockdown behavior (12.x - 17.x) for details on Port Lockdown. Attackers typically use TMUI or iControl rest vulnerabilities to gain access to BIG-IP systems. The two major ones we see are CVE-2020-5902 and CVE-2022-1388, but many other vectors exist, so organizations should strive to keep their BIG-IPs software up to date. Attackers also try default passwords, and we have noted some systems that have default passwords in production. Make sure to change the admin and root passwords to strong ones on your BIG-IPs, even if you disable those accounts. Attackers who are trying passwords often will look for open SSH ports to try them against, so the same guidance above of not exposing TMUI/REST to the internet applies to SSH. Outro I hope this was helpful. If your dealing with a compromised system, you may find my article on Inspecting a UCS file from a compromised BIG-IP to be helpful in recovering from a compromise. Finally, again, if you suspect that you have a compromised F5 system, you can Contact the F5 SIRT by opening up a service request and/or follow the guidance in K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system.
iHealth Upgrade Advisor: Making upgrades a little easier
Whether it is upgrading the firmware on a switch, the OS on a server, an important business application or the software on a BIG-IP, performing upgrades is something that makes almost all IT Admins and Network Engineers nervous. We’ve learned from (sometimes painful) experience that things don’t always go as planned. Good preparation greatly increases the likelihood that an upgrade will be successful, which is why F5 has created the iHealth Upgrade Advisor. Its goal is to provide an additional service from F5 that will complement your existing upgrade preparations, increasing the predictability of the upgrade while reducing your upgrade time. The iHealth Upgrade Advisor service provides a way for users to gain insight into potential issues with a BIG-IP upgrade before they attempt the upgrade. It provides guidance that is specific to a BIG-IP based on its configuration, the version of software it is currently running and the version you are planning to upgrade to. When an issue can be avoided by making a configuration change prior to upgrading, the Upgrade Advisor will tell you exactly what to change. For some issues, it will list the corrective actions to take after the upgrade. Demo Video This short video demonstrating the Upgrade Advisor shows you how to use it and some examples of the guidance it provides. Accessing the Upgrade Advisor The next time you are preparing to upgrade a BIG-IP, login to ihealth.f5.com, upload a .qkview file from that BIG-IP and then view the qkview after iHealth has analyzed it. The Upgrade Advisor can be accessed by clicking on its tab in the left-hand menu. Simply select the version of BIG-IP you are planning to upgrade to in the advisor and review the results. Here is a screenshot of the Upgrade Advisor: Give it a Try Try out the F5 upgrade Advisor today and let us know what you think using the feedback option (circled in red on the right side of the screenshot above).
iHealth API Part 4 - A Little More Code
In the last chapter, we wrote a quick bash script that used some tools to produce diagnostics summaries for qkviews. Now that you have some experience with the API, maybe you want to do a little bit more. In this article, we're going to explore the data in a little more detail to give you an idea of the kinds of things you can do with the API. Generally speaking, the API methods (sometimes called 'calls') are broken into two different broad categories, "group" or "collection" methods, and then methods that deal with a single qkview ("instance methods"). In this article, we're going to cover some of the collection methods, and how to manage your qkviews. In previous articles, we've restrained ourselves from going hog-wild and kept to the nice safe methods that are essentially read-only methods. That is, we just ask for information from the API, and then consume it and manipulate it locally, without making any changes to the data on the server. This is a nice safe way to get a handle on how the API works, without accidentally deleting anything, or changing an attribute of a QKView without meaning to. It's time to take the training wheels off, call in the goats, and go a little wild. Today, we'll be modifying an attribute of a QKView, or if we're feeling really crazy, we'll modify two of them, maybe even in the same HTTP request. My palms are sweating just writing this... Before we get that crazy, let's take a minute and talk about some collection methods first, and get that out of the way. Collection methods can be very handy, and very powerful, as they allow you to deal with your QKViews en masse. There are three collection methods that we'll talk about, the first one is a way to get a list of all your QKView IDs. This makes it possible to iterate over all the QKViews in your iHealth account while hunting for something. Remember how we got our list of goats in a barn? Similarly: GET qkview-analyzer/api/qkviews will return a list of qkview ids for every qkview in your ihealth account: What if we decide we want to get rid of all our QKViews? We could GET the list of IDs, and then iterate through them, DELETEing each one, but a collection method is much easier: DELETE qkview-analyzer/api/qkviews will wipe all the qkviews from your account. Use with caution, this is the 'nuke from orbit' option. We won't be using it in our projects for this article series, but I thought I'd mention it, as it is very handy for some situations. Another collection method that we *will* be using is this: POST qkview-analyzer/api/qkviews This method allows us to add to our collection of qkviews. Since we're using POST, remember that each time this is executed (even with the exact same qkview file), a new qkview object is created on the server. So you'll only need to do it once for a given qkview file, unless of course, you accidentally DELETE it from your collection, and want to add it again. I've never done anything like that, I'm just speaking hypothetically. So now armed with those collection methods, and with knowledge of our previous scripts that allowed us to examine QKView metadata, let's build up a little script that combines collection methods with some instance methods. Now, the sample qkview won't suffice for this exercise, as it is meant for demonstration purposes, and is used system-wide (and is thus read-only). For this exercise today, we'll be using a real qkview. This article is going to write two scripts. The first will perform an upload of a qkview file to iHealth, using many of the techniques that we learned in our first scripting session. Upload QKView script (note that all the scripts referenced in this series are included, but the one needed for this exercise is upload-qkview.sh) Notice how the authentication parts all look the same? What a great opportunity to refactor these scripts and build up an iHealth API library that our other scripts could use. Extra credit if you take that on! So using the same authentication methods, we need to be able to specify the location of the qkview file that we want to upload. Here is a sample qkview file for you to download and use for these exercises, or feel free to pull a qkview off your own gear, and use that. AskF5 explains how to get a QKView. Here is where the collection method to add a qkview to our collection happens: 96 function upload_qkview { 97 path="$1" 98 CURL_CMD="${CURL} ${ACCEPT_HEADER} ${CURL_OPTS} -F 'qkview=@${path}' -D /dev/stdout https://ihealth-api.f5.com/qkview-analyzer/api/qkviews" 99 [[ $DEBUG ]] && echo "${CURL_CMD}" >&2 100 out="$(eval "${CURL_CMD}")" 101 if [[ $? -ne 0 ]]; then 102 error "Couldn't retrieve diagnostics for ${qid}" 103 fi 104 location=$(echo "${out}" | grep -e '^Location:' | tr -d '\r\n') 105 transformed=${location/Location: /} 106 echo "${transformed}" 107 The upload will finish with an HTTP 302 redirect that points us to the new qkview. After we upload, we cannot just immediately ask for data about the qkview, as there is a certain amount of processing that happens before we can start grilling the server about it. While the server is busy doing all the prep work it needs to do in order to give you back data about the qkview, the server doesn't mind if we politely ask it about how things are going up there in the cloud. To do this, we use a GET request, and the server will respond with an HTTP status code that tells our script how things are going: our code: "ready yet?" server : "not yet" our code: "ready yet?" server : "not yet" sort of like kids on a car trip asking "are we there yet?" over and over and over again. Only instead of the driver getting enraged and turning up the radio to drown them out, our server just responds politely "not yet" until it's done processing. our code: "ready yet?" server : "yup, ready!" Then our code can go about it's business. In the programming trades, this called 'polling'. We poll the server until the server gives us the answer we want. our code: GET qkview-analyzer/api/qkviews/ server : HTTP 202 our code: GET qkview-analyzer/api/qkviews/ server : HTTP 202 our code: GET qkview-analyzer/api/qkviews/ server : HTTP 200 So that's how to add a qkview to your collection. Of course you might not get an HTTP 202 or 200 back, you might get something else, in which case something went wrong in either the upload or the processing. At that point, we should also bail out, and return an error to the runner of the script: 109 function wait_for_state { 110 url="$1" 111 count=0 112 CURL_CMD="${CURL} ${ACCEPT_HEADER} ${CURL_OPTS} -w "%{http_code}" ${url}" 113 [[ $DEBUG ]] && echo "${CURL_CMD}" >&2 114 _status=202 115 time_passed=0 116 while [[ "$_status" -eq 202 ]] && [[ $count -lt ${POLL_COUNT} ]]; do 117 _status="$(eval "${CURL_CMD}")" 118 count=$((count + 1)) 119 time_passed=$((count * POLL_WAIT)) 120 [[ $VERBOSE ]] && echo -ne "waiting (${time_passed} seconds and counting)\r" >&2 121 sleep ${POLL_WAIT} 122 done 123 printf "\nFinished in %s seconds\n" "${time_passed}" >&2 124 if [[ "$_status" -eq 200 ]]; then 125 [[ $VERBOSE ]] && echo "Success - qkview is ready" 126 elif [[ ${count} -ge ${POLL_COUNT} ]]; then 127 error "Timed out waiting for qkview to process" 128 else 129 error "Something went wrong with qkview processing, status: ${_status}" 130 fi 131 } The change-description.sh script (included in the zip linked above) will allow you to update the description field of any number of qkviews with a given chassis serial number. This script will use both a collection method (list all my qkview IDs), and an instance method on several qkviews to update some metadata associated with the qkview. We've introduced yet another verb into our working lexicon, PUT. We use PUT here, because it's a modification on a qkview that no matter how many times we perform it, will result in the same qkview state. This is called idempotency. Unlike, say our POST above in our upload script, which results in new qkviews every time you run it, this PUT may change the state of the qkview the first time, but subsequent identical PUT requests won't change the state further. So now we can submit QKViews, get diagnostic results, and change metadata about the QKView. So what else could we possibly do? The rest of this article series will explore the data in the API and how to retreive and process it.
