How the F5 SIRT Looks for Malware
Introduction Welcome to another article by F5 SIRT Security Engineer Kyle Fox. Today we explore the ways that F5 SIRT looks for malware on suspected compromised systems. We will explore some iHealth heuristics and one of the common ways malware is hidden on systems. First, we need to get some administrative matters out of the way. The F5 SIRT and F5 Support are not able to provide computer forensics services. There are a number of reasons for this including licensing, certifications and legal issues. We can assist in finding indicators of compromise, identifying suspect logs or data in QKViews, and answering questions from forensic analysts about F5 products and how they work. If you suspect that you have a compromised F5 system, you can Contact the F5 SIRT by opening up a service request and/or follow the guidance in K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system. iHealth Heuristics H511618: Malware may have infected the BIG-IP system This heuristic compares process names with a list of known-good process names on BIG-IP systems. It does not always alert when malware has infected the system and also sometimes it alerts on expected processes such as networked FIPS HSM drivers and others. If it is seen, more investigation may be needed. There are three other heuristics that also help identify how a system was compromised, or if your looking to secure a system, identify how to secure it: H444724: The management interface is allowing access from public IP addresses This heuristic is checking the IP addresses of connections to the management interface and looking for public IP addresses. This will identify when the BIG-IP is getting such access, even if such connections are being IP or port forwarded to a private IP address on the BIG-IP. H458565: Public IP addresses configured as a BIG-IP Self IP This heuristic will identify when public IP addresses are configured as a Self-IP and not properly secured using port lockdown. See K17333: Overview of port lockdown behavior (12.x - 17.x) for details on Port Lockdown. H727910: The configuration contains user accounts with insecure passwords This heuristic looks for default and common passwords in the user accounts. Typically this will hit on admin or root being set to something like "admin", "default", or "root". But there are a limited number of other common passwords it will look for. Remember to change the password on admin and root to a stronger password even if you disable those accounts. Typical Malware What the F5 SIRT has observed is the typical malware we see does flooding, cryptocurrency mining, or both. This malware is typically named with random characters and is hidden by deleting the executable after it is run. Attackers may stop at deploying such malware or may continue to try and infiltrate the victims networks from there. Typically they avoid implanting malware when using the F5 system to pivot into a network. One way we can look for this malware without using the iHealth heuristics is by looking for executables that are deleted, in the lsof command. They look like this: From iHealth -> Commands -> Standard -> UNIX -> Networking -> lsof -n (text only) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME [...] xDwJc 23367 root cwd DIR 253,12 4096 106664 /var/service/restjavad xDwJc 23367 root rtd DIR 253,9 1024 2 / xDwJc 23367 root txt REG 253,9 282 30733 /tmp/xDwJc (deleted) xDwJc 23367 root 0r CHR 1,3 0t0 1028 /dev/null xDwJc 23367 root 1r CHR 1,3 0t0 1028 /dev/null xDwJc 23367 root 2r CHR 1,3 0t0 1028 /dev/null xDwJc 23367 root 3u sock 0,7 0t0 2873878973 protocol: TCP xDwJc 23367 root 4u a_inode 0,10 0 6105 [eventfd] So, lets review what we see here. The main thing we are looking for is the line with the FD column named txt. This is the executable file the process is running from, and we see in the NAME column it says (deleted). In Linux, a file exists after it’s been deleted as long as it’s still open by one or more processes, so deleting the file does not cause the process to end. Malware writers have used this to hide malware from antimalware software that scans the filesystem for malicious executables. Let’s review all the columns in this listing: COMMAND - The command being executed. PID - The process ID, we will use this later. USER - The user the command is running as. FD - The file descriptor type. Here we see: cwd - Current working directory. rtd - Root directory. txt - Program executable. <number><letter> - file descriptors. The letter can be r for read, w for write and u for both. TYPE - Type of filesystem node associated with the file: DIR - Directory. REG - Regular File. CHR - Character special file, in this case a device called null. sock - A socket. a_inode - An anonymous inode (file system node) in this case a connection to the event forwarder daemon. DEVICE - The device numbers for the file. NODE - Node number of the file. NAME - The file name. From what we can see, the working directory is /var/service/restjavad. This strongly implies that the malware got on the system using an exploit for iControl REST, but it may also have gotten on the system through a longer route through some other part of TMUI. If you have a forensic analyst or otherwise want to analyze the malware, you can copy the executable even though it’s deleted by copying from /proc/<PID>/exe, in this case: cp /proc/23367/exe /var/tmp/malwarefile Typical Ways Intruders Break In In most cases of compromised systems, the attackers got in because the system had an Internet-facing Self-IP that was not secured. We recommend that you do not use Internet-facing Self-IPs for HA or administration, and to set Port Lockdown on Internet-facing Self-IPs to Allow None. See K17333: Overview of port lockdown behavior (12.x - 17.x) for details on Port Lockdown. Attackers typically use TMUI or iControl rest vulnerabilities to gain access to BIG-IP systems. The two major ones we see are CVE-2020-5902 and CVE-2022-1388, but many other vectors exist, so organizations should strive to keep their BIG-IPs software up to date. Attackers also try default passwords, and we have noted some systems that have default passwords in production. Make sure to change the admin and root passwords to strong ones on your BIG-IPs, even if you disable those accounts. Attackers who are trying passwords often will look for open SSH ports to try them against, so the same guidance above of not exposing TMUI/REST to the internet applies to SSH. Outro I hope this was helpful. If your dealing with a compromised system, you may find my article on Inspecting a UCS file from a compromised BIG-IP to be helpful in recovering from a compromise. Finally, again, if you suspect that you have a compromised F5 system, you can Contact the F5 SIRT by opening up a service request and/or follow the guidance in K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system.
iHealth Upgrade Advisor: Making upgrades a little easier
Whether it is upgrading the firmware on a switch, the OS on a server, an important business application or the software on a BIG-IP, performing upgrades is something that makes almost all IT Admins and Network Engineers nervous. We’ve learned from (sometimes painful) experience that things don’t always go as planned. Good preparation greatly increases the likelihood that an upgrade will be successful, which is why F5 has created the iHealth Upgrade Advisor. Its goal is to provide an additional service from F5 that will complement your existing upgrade preparations, increasing the predictability of the upgrade while reducing your upgrade time. The iHealth Upgrade Advisor service provides a way for users to gain insight into potential issues with a BIG-IP upgrade before they attempt the upgrade. It provides guidance that is specific to a BIG-IP based on its configuration, the version of software it is currently running and the version you are planning to upgrade to. When an issue can be avoided by making a configuration change prior to upgrading, the Upgrade Advisor will tell you exactly what to change. For some issues, it will list the corrective actions to take after the upgrade. Demo Video This short video demonstrating the Upgrade Advisor shows you how to use it and some examples of the guidance it provides. Accessing the Upgrade Advisor The next time you are preparing to upgrade a BIG-IP, login to ihealth.f5.com, upload a .qkview file from that BIG-IP and then view the qkview after iHealth has analyzed it. The Upgrade Advisor can be accessed by clicking on its tab in the left-hand menu. Simply select the version of BIG-IP you are planning to upgrade to in the advisor and review the results. Here is a screenshot of the Upgrade Advisor: Give it a Try Try out the F5 upgrade Advisor today and let us know what you think using the feedback option (circled in red on the right side of the screenshot above).
iHealth API Part 4 - A Little More Code
In the last chapter, we wrote a quick bash script that used some tools to produce diagnostics summaries for qkviews. Now that you have some experience with the API, maybe you want to do a little bit more. In this article, we're going to explore the data in a little more detail to give you an idea of the kinds of things you can do with the API. Generally speaking, the API methods (sometimes called 'calls') are broken into two different broad categories, "group" or "collection" methods, and then methods that deal with a single qkview ("instance methods"). In this article, we're going to cover some of the collection methods, and how to manage your qkviews. In previous articles, we've restrained ourselves from going hog-wild and kept to the nice safe methods that are essentially read-only methods. That is, we just ask for information from the API, and then consume it and manipulate it locally, without making any changes to the data on the server. This is a nice safe way to get a handle on how the API works, without accidentally deleting anything, or changing an attribute of a QKView without meaning to. It's time to take the training wheels off, call in the goats, and go a little wild. Today, we'll be modifying an attribute of a QKView, or if we're feeling really crazy, we'll modify two of them, maybe even in the same HTTP request. My palms are sweating just writing this... Before we get that crazy, let's take a minute and talk about some collection methods first, and get that out of the way. Collection methods can be very handy, and very powerful, as they allow you to deal with your QKViews en masse. There are three collection methods that we'll talk about, the first one is a way to get a list of all your QKView IDs. This makes it possible to iterate over all the QKViews in your iHealth account while hunting for something. Remember how we got our list of goats in a barn? Similarly: GET qkview-analyzer/api/qkviews will return a list of qkview ids for every qkview in your ihealth account: What if we decide we want to get rid of all our QKViews? We could GET the list of IDs, and then iterate through them, DELETEing each one, but a collection method is much easier: DELETE qkview-analyzer/api/qkviews will wipe all the qkviews from your account. Use with caution, this is the 'nuke from orbit' option. We won't be using it in our projects for this article series, but I thought I'd mention it, as it is very handy for some situations. Another collection method that we *will* be using is this: POST qkview-analyzer/api/qkviews This method allows us to add to our collection of qkviews. Since we're using POST, remember that each time this is executed (even with the exact same qkview file), a new qkview object is created on the server. So you'll only need to do it once for a given qkview file, unless of course, you accidentally DELETE it from your collection, and want to add it again. I've never done anything like that, I'm just speaking hypothetically. So now armed with those collection methods, and with knowledge of our previous scripts that allowed us to examine QKView metadata, let's build up a little script that combines collection methods with some instance methods. Now, the sample qkview won't suffice for this exercise, as it is meant for demonstration purposes, and is used system-wide (and is thus read-only). For this exercise today, we'll be using a real qkview. This article is going to write two scripts. The first will perform an upload of a qkview file to iHealth, using many of the techniques that we learned in our first scripting session. Upload QKView script (note that all the scripts referenced in this series are included, but the one needed for this exercise is upload-qkview.sh) Notice how the authentication parts all look the same? What a great opportunity to refactor these scripts and build up an iHealth API library that our other scripts could use. Extra credit if you take that on! So using the same authentication methods, we need to be able to specify the location of the qkview file that we want to upload. Here is a sample qkview file for you to download and use for these exercises, or feel free to pull a qkview off your own gear, and use that. AskF5 explains how to get a QKView. Here is where the collection method to add a qkview to our collection happens: 96 function upload_qkview { 97 path="$1" 98 CURL_CMD="${CURL} ${ACCEPT_HEADER} ${CURL_OPTS} -F 'qkview=@${path}' -D /dev/stdout https://ihealth-api.f5.com/qkview-analyzer/api/qkviews" 99 [[ $DEBUG ]] && echo "${CURL_CMD}" >&2 100 out="$(eval "${CURL_CMD}")" 101 if [[ $? -ne 0 ]]; then 102 error "Couldn't retrieve diagnostics for ${qid}" 103 fi 104 location=$(echo "${out}" | grep -e '^Location:' | tr -d '\r\n') 105 transformed=${location/Location: /} 106 echo "${transformed}" 107 The upload will finish with an HTTP 302 redirect that points us to the new qkview. After we upload, we cannot just immediately ask for data about the qkview, as there is a certain amount of processing that happens before we can start grilling the server about it. While the server is busy doing all the prep work it needs to do in order to give you back data about the qkview, the server doesn't mind if we politely ask it about how things are going up there in the cloud. To do this, we use a GET request, and the server will respond with an HTTP status code that tells our script how things are going: our code: "ready yet?" server : "not yet" our code: "ready yet?" server : "not yet" sort of like kids on a car trip asking "are we there yet?" over and over and over again. Only instead of the driver getting enraged and turning up the radio to drown them out, our server just responds politely "not yet" until it's done processing. our code: "ready yet?" server : "yup, ready!" Then our code can go about it's business. In the programming trades, this called 'polling'. We poll the server until the server gives us the answer we want. our code: GET qkview-analyzer/api/qkviews/ server : HTTP 202 our code: GET qkview-analyzer/api/qkviews/ server : HTTP 202 our code: GET qkview-analyzer/api/qkviews/ server : HTTP 200 So that's how to add a qkview to your collection. Of course you might not get an HTTP 202 or 200 back, you might get something else, in which case something went wrong in either the upload or the processing. At that point, we should also bail out, and return an error to the runner of the script: 109 function wait_for_state { 110 url="$1" 111 count=0 112 CURL_CMD="${CURL} ${ACCEPT_HEADER} ${CURL_OPTS} -w "%{http_code}" ${url}" 113 [[ $DEBUG ]] && echo "${CURL_CMD}" >&2 114 _status=202 115 time_passed=0 116 while [[ "$_status" -eq 202 ]] && [[ $count -lt ${POLL_COUNT} ]]; do 117 _status="$(eval "${CURL_CMD}")" 118 count=$((count + 1)) 119 time_passed=$((count * POLL_WAIT)) 120 [[ $VERBOSE ]] && echo -ne "waiting (${time_passed} seconds and counting)\r" >&2 121 sleep ${POLL_WAIT} 122 done 123 printf "\nFinished in %s seconds\n" "${time_passed}" >&2 124 if [[ "$_status" -eq 200 ]]; then 125 [[ $VERBOSE ]] && echo "Success - qkview is ready" 126 elif [[ ${count} -ge ${POLL_COUNT} ]]; then 127 error "Timed out waiting for qkview to process" 128 else 129 error "Something went wrong with qkview processing, status: ${_status}" 130 fi 131 } The change-description.sh script (included in the zip linked above) will allow you to update the description field of any number of qkviews with a given chassis serial number. This script will use both a collection method (list all my qkview IDs), and an instance method on several qkviews to update some metadata associated with the qkview. We've introduced yet another verb into our working lexicon, PUT. We use PUT here, because it's a modification on a qkview that no matter how many times we perform it, will result in the same qkview state. This is called idempotency. Unlike, say our POST above in our upload script, which results in new qkviews every time you run it, this PUT may change the state of the qkview the first time, but subsequent identical PUT requests won't change the state further. So now we can submit QKViews, get diagnostic results, and change metadata about the QKView. So what else could we possibly do? The rest of this article series will explore the data in the API and how to retreive and process it.iHealth API Part 3 - A Little Code
We finished the last article with exploring some of the data available in the API with a web browser that was helpful enough to render it in a mildly readable fashion. For most automation projects, however, we don't care if it's easy to read, as long as it's parseable and we can do something with the data that doesn't involve watching streams of xml scroll off the screen. For today, we'll be working with the data from the diagnostics section of the API: We'll use the same data from the API that we explored last time. GET /qkview-analyzer/api/qkviews/0/diagnostics For this script exercise, we'll be using a couple of tools working in the unix shell: bash 4.x xmlstarlet grep When the diagnostics run against a QKView, they either are considered a 'hit' where the diagnostic found an issue in the QKView, or a 'miss', where the diagnostic ran, but found no issue. We'll develop a little script that will provide a quick summary of our hits and misses for a given QKView. In the script we'll deal with the following steps: authentication diagnostics retrieval and asking for subsets of diagnostics diagnostics parsing and selective display Let's start out by writing a couple of functions that will deal with the housekeeping that we need to do in order to connect to the API. First thing we have to do is authenticate ourselves with the credentials that we used when we built our iHealth account. By authenticating, we will obtain a cookie in the HTTP response to our authentication request (if it's successful!) that we will send with every subsequent request to the API. This way, the API knows who we are, which qkviews in the system are ours, and knows that we've proven we're who we're claiming to be. This is currently done by sending a form POST to the login server containing our credentials, and stashing the cookies we get back for later use. curl provides exactly the sort of functionality that we need to perform API work, and bindings for the curl libraries are available for lots and lots of languages if you want to use something other than bash, or want to develop a bigger script later on. We set up some initial values for later use: 13 readonly CURL=/usr/bin/curl ... 23 RESPONSE_FORMAT=${FORMAT:-"xml"} ... 32 ACCEPT_HEADER="-H'Accept: application/vnd.f5.ihealth.api+${RESPONSE_FORMAT}'" Now that the housekeeping is done, we'll need an authentication function: 72 function authenticate { 73 user="$1" 74 pass="$2" 75 # Yup! Security issues here! we're eval'ing with user input. Don't put this code into a CGI script... 76 CURL_CMD="${CURL} --data-urlencode 'userid=${user}' --data-urlencode 'passwd=${pass}' ${CURL_OPTS} https://login.f5.com/resource/login 76 Action.jsp" 77 [[ $DEBUG ]] && echo ${CURL_CMD} 78 79 if [[ ! "$user" ]] || [[ ! "$pass" ]]; then 80 error "missing username or password" 81 fi 82 eval "$CURL_CMD" 83 rc=$? 84 if [[ $rc -ne 0 ]]; then 85 error "curl authentication request failed with exit code: ${rc}" 86 fi 87 88 if ! \grep -e "sso_completed.*1$" ${COOKIEJAR} > /dev/null 2>&1; then 89 error "Authentication failed, check username and password" 90 fi 91 [[ $VERBOSE ]] && echo "Authentication successful" >&2 92 } The way this script is written, it uses the setting of environment variables on the commandline in order to protect the innocent. This allows us to pass sensitive information into a script without it needing to be embedded for others to discover, or show up in the output of ps on a shared machine. You can hardcode it if you wish, but I'd recommend not doing so. Once authentication succeeds, then we go out and grab the diagnostics for the qkview we specified as an argument to the script: 94 function get_diagnostics_hits { 95 qid="$1" 96 CURL_CMD="${CURL} ${ACCEPT_HEADER} ${CURL_OPTS} https://ihealth-api.f5.com/qkview-analyzer/api/qkviews/${qid}/diagnostics?set=hit" 97 [[ $DEBUG ]] && echo "${CURL_CMD}" >&2 98 out="$(eval "${CURL_CMD}")" 99 if [[ $? -ne 0 ]]; then 100 error "Couldn't retrieve diagnostics for ${qid}" 101 fi 102 echo "$out" 103 } Notice how we have set up an Accept header in the request? This tells the API what format you want the response to be in. Diagnostics have a special role in the API, so they are available in a couple format: xml, json, pdf, and csv. Everything else in the API is only available in xml or json. Use the Accept header to specify how you want your response data. If you don't use a response header, then the API assumes you want XML, and returns XML. In this example we're asking for XML explicitly. Now that we have a big pile of XML, we only want a couple pieces of it, so make xmlstarlet dig through it and give us a nice display: 131 #perform some XML extraction, and print it out in a nice readable format 132 diagnostics_count=$(echo ${diagnostics} | ${XMLPROCESSOR} select -t -c 'string(/diagnostic_output/diagnostics/@hit_count)' -) 133 for ((i=1;i<=diagnostics_count;i++)); do 134 printf "%-10s : " $(echo ${diagnostics} | ${XMLPROCESSOR} select -t -c "string(//diagnostic[$i]/@name)" -) 135 printf "%s\n" "$(echo ${diagnostics} | ${XMLPROCESSOR} select -t -c "//diagnostic[$i]//h_header/node()" -)" 136 done See how easy that is? We can do the same thing with json if we want: 140 # in json 141 diagnostics_count=$(echo ${diagnostics} | ${JSONPROCESSOR} -r .diagnostics.hit_count) 142 for ((i=0;i<=diagnostics_count;i++)); do 143 printf "%-10s : " $(echo ${diagnostics} | ${JSONPROCESSOR} -r .diagnostics.diagnostic[$i].name) 144 printf "%s\n" "$(echo ${diagnostics} | ${JSONPROCESSOR} -r .diagnostics.diagnostic[$i].results.h_header)" 145 done If you need to deal with json at the commandline, and don't already know about it, jq is an incredibly powerful tool for handling json, and well worth your time exploring (see Working with iControl REST Data on the Command LIne for more details.) So now that it works, what do all the fields mean? Let's look at the json output and see what we're getting: { "diagnostics": { "filter": "hit", "diagnostic": [ { "name": "H371501", "output": [], "run_data": { "h_importance": "MEDIUM", "match": true }, "results": { "h_action": "Upgrade to version 10.2.2 or later.", "h_name": "H371501", "h_header": "A known issue causes the chmand process to leak memory on this BIG-IP platform", "h_summary": "Due to a known issue, the chassis manager daemon, chmand, leaks memory on BIG-IP systems which contain the Always-On Management (AOM) subsystem. The memory leak is constant regardless of the volume of traffic processed by the system.", "h_sols": [ "http://support.f5.com/kb/en-us/solutions/public/12000/900/sol12941.html" ] } }, name - the diagnostic name output - is an array containing any qkview specific information the diagnostics wants to tell you about run-data - data about the run including importance (low medium high critical) match (if it's a hit or not) results h_action - references to more materials h_name - the diagnostic name again h_header - a title h_summary - a more verbose explanation of the issue h_sols - links to solution articles that go into more depth Customizing the data that is shown in your summary is very easy, by adding the statements that dig into the diagnostics return data. One could, if one wished, build up a reporting system that showed the change in diagnostics results over time for a collection of qkviews if one were so inclined. The full script is available here.iHealth API Part 2 - An Introduction to REST
In the last article, we got some ideas of the kinds of data that iHealth can provide by exploring the wealth of information through a standard web UI. This is a quick easy way to get answers about a couple of your F5 machines, or to do a one-off check of something, or if you are looking for a particular problem, or working with F5 Support to resolve an issue or answer a question. If you have a couple machines, or maybe many many more, however, and you want to do something periodically, like, say, check for new Diagnostics results (remember, we generally update them once a week), then you'll quickly find that opening your browser, logging in, uploading the QKViews, and then reading through the Diagnostics is not necessarily the most efficient method. It can become annoying and tedious for even one or two machines. Since iHealth is a web application, it made the most sense to make the iHealth API a web API as well. There is a dogs breakfast of different types of web APIs, but one of the most prevalent conceptual frameworks in use today is called Representational State Transfer, or REST. The wikipedia entry on Roy Fielding has links to his dissertation if you want to up your street cred significantly. REST provides a nice clean way of retrieving, modifying and deleting things using HTTP in all (well most all) of it's glory. It's hidden from most users (as it should be), and unless you are a web developer, or someone who is dealing with web traffic a lot, there isn't any reason you would need to know this, but the HTTP protocol specifies a number of different ways to ask the server to do something. Now that you will be working with an HTTP API, you're going to have to know a little more about HTTP. Generally the two verbs that a developer will be familiar with is GET and POST. But HTTP has many other verbs, and we'll use most of them in this article series (GET, POST, PUT, and DELETE for now, although there are others), REST gives us a framework for using those verbs to make changes, or view the properties of things on the web. They generally do what they sound like: GET - get information (generally read-only) POST - create a new object (or modify an existing object repeatedly) PUT - modify an object idempotently so that multiple PUTs will result in the same state DELETE - delete an object This is all very abstract, so let's come up with an extremely contrived example: imagine a barn full of goats ( yes, my first job was as a farmhand ). Let's say that you had a robot that you wanted to perform various tasks relating to this barn full of goats. Maybe the first one would be a survey of all the goats in the barn. If the robot spoke REST, then you might send a text message to your robot that looked like this: GET /barn/1/goats Because the robot knows that in REST, that means that you want to know about all the goats in barn 1, your robot would trundle off (or fly, or crawl) to barn 1, and might come back with this data: goat 1: name: Dopey goat 2: name: Speedy Maybe then you wanted to know more about Speedy, so you'd text your robot this: GET /barn/1/goats/2 and your robot would gather up some more information about goat 2: goat 2: name : Speedy eats : Himalayan Blackberry hates : Dopey Okay, so what the hell do goats and barns have to do with iHealth? Well, if we substitute in QKViews for goats, we can start to see some information about our data in the same way that we were able to examine our herds of goats: GET /qkview-analyzer/api/qkviews Will show us a list of the qkviews that we have in our account: This is a sample, and there is a special QKView that isn't show in the listing, but that you should always have in your account, a QKView with an ID of 0. It's a sample QKView that we allow everyone access to in order to see how iHealth works with requireing you to upload anything of your own. So let's pick QKView 0 and load it up. We'll see some details about that QKView: GET /qkview-analyzer/api/qkviews/0 Feel free to poke around. At the bottom of the listing are some "discovery" URLs. Load them up in your browser; there is a ton to explore here, and you can get an idea of the kind of data you can work with without writing a single line of code. Code will come later, as you'll need it for things like adding to your QKView collections, and modifying existing QKView entries, which are tricky to do with just a browser.iHealth API Part 1 - An Introduction to iHealth
Welcome to the inaugural iHealth post (for the http nerds, yes, pun intended). This series will be an exploration and explanation of the iHealth Application Programming Interface (API), and how making the iHealth API a part of your datacenter ecosystem can bring you good fortune, reduce your downtime, help prevent emergency RMAs, and cook your emergency bacon. But before we get to the emergency bacon (spoiler: there is no emergency bacon), let's start with the basic building blocks and concepts that we'll be talking about, just so there is no regrettable misunderstandings later in the series. Let's start with the foundation of this series: iHealth. iHealth If you've spent much time with F5 gear (actual hardware or virtual editions it doesn't matter) in the last couple years, or had any interactions with the F5 support teams, you may have at least heard of iHealth. iHealth is a free, F5 hosted web application that provides deep inspection of BIG-IP, BIG-IQ, and Enterprise Manager system data. We want to help you monitor your systems, tune up your configurations, provide a distant early warning system, and help you keep on top of any issues or opportunities we can. The most important part of that deep inspection that iHealth provides are our Diagnostics. As of this writing, we've got over 900 diagnostic tests that are run against all that system data. Each diagnostic is specifically targeted to look for a very specific issue, or inform you about a particular configuration optimization, or warn you about a particular security vulnerability. We don't just warn you, or offer generic advice however. We also provide the specific items in your configuration that might be susceptible, or in need of a little love, and we provide extensive [Ask F5] solution articles to back up each Diagnostic to ensure that you have all the context, details, and knowledge to get things fixed up before they become a bigger problem. The F5 iHealth team releases new and updated Diagnostics every week, so the list of Diagnostics that are 'hits' (identify your gear as exhibiting the symptoms the diagnostic is looking for), may change from week to week. We get ideas for Diagnostics from all over F5, as well as from our customers (that's you). Got an idea of something you want iHealth to check for on all your machines? Jump into iHealth and hit the feedback button, and describe what you want to do. That feedback button creates a ticket that the F5 iHealth team will follow up on. The far-seeing among us might postulate that doing weekly checks of our gear might be a great way to keep our ship running tight with very little cost. That would be an astute observation indeed, and this article series will address that observation in some detail in the weeks to follow. iHealth has lots of other uses as well, but the Diagnostics are the crown jewel in the iHealth tiara. You can start using this tool, if you haven't already, by visiting https://ihealth.f5.com/. You might have to create a (free) account, but that's the only obstacle. We'll be depending on your access to your iHealth account for the duration of this series, so if you don't have an account, go sign up for one now. After you get your account, you can access https://ihealth.f5.com/ and you'll notice there is a sample QKView in your account already. We provision a sample in there to give you something to work with immediately, and, conveniently, it will give us a data set to use for the rest of this series. The UI is fairly extensive, and gives you lots of information about your machine, including things like End of Life notices for your hardware and software, if you've got a support contract, we let you know when that contract ends, as well as displaying lots of information about your F5 gear. Customizable performance graphs? Yup. Command-line output at the moment the QKView was taken? Yup. Diagnostics? Yup, we already covered that. Configuration information? Yup. So if all that data is already available, why am I bothering with an API? Well, the rest of the series should help illuminate the answer to that question, and you'll end up writing a basic application to make use of all the data iHealth provides, which should give you plenty of ideas about how iHealth in all it's forms can be an incredibly helpful tool.In 5 Minutes or Less - Enterprise Manager v3.0
In my 21st In 5 Video, I show you some of the new features available in Enterprise Manager v3.0, in 5 Minutes or Less. We cover the Centralized Analytics Module, iHealth integration and Multi-device configuration comparison. As a follow-up to release of BIG-IP v11.2, Enterprise Manager v3.0 is now available. In addition to support of v11.2, this release provides some key new capabilities that will help monitor and understand application performance across all BIG-IP devices. Simplify and automate the management of your F5 ADN infrastructure. ";" alt="" /> In 5 Minutes or Less - Enterprise Manager v3.0 ps Resources: F5 Enterprise Manager Overview Application Delivery Network Platform Management In 5 Minutes or Less Series (21 videos – over 2 hours of In 5 Fun) F5 Youtube Channel
F5 Friday: New Services from F5 Ease Migration and Upgrades
I get by with a little help from my friends… While cloud and virtualization primarily focus on improving the provisioning process, there is a lot more to managing a data center and its critical components than just deployment. There’s upgrades – both software and hardware – and migration to new solutions as well as tweaking knobs and buttons to optimize and troubleshoot issues. While public cloud computing may alleviate much of the pain associated with forward movement, private and hybrid environments as well as traditional data center models must face the reality of dealing with these admittedly often tedious tasks. It’s a foregone conclusion that new technology and devices like mobile, tablets, unified application delivery and cloud computing as well as an evolving threat spectrum put pressure on IT to maintain a healthy and modern set of services to ensure availability, performance, and security. As pressures increase on infrastructure services, vendors respond with new and or updated solutions to help IT combat the growing complexity of data center architectures. But sometimes, IT needs a little help from its friends to get there, and that’s where professional service organizations enter into the picture. One of F5’s top priorities is a world-class service organization. From implementation and ongoing support to migration and upgrades, our professional services organization continues to evaluate the technology landscape and address the most pressing issues facing IT through new offerings designed to ease those pain points introduced by a need to upgrade or migrate to new platforms. New Services Offerings from F5 F5 is introducing three new services offerings that address many of these issues. Each assessment covers four phases: planning, analysis, a detailed report, and review with recommendations. BIG-IP Local Traffic Manager (LTM) Upgrade Assessment Understand Your Infrastructure’s Readiness for Change The flexible infrastructures made possible by BIG-IP LTM can drive efficiencies, support business growth, and optimize new capabilities that become available as the infrastructure devices evolve. Nonetheless, version upgrades require planning and analytical validation that new functionality will align with the organization’s infrastructure vision. The BIG-IP LTM configuration is assessed in four broad categories: Platform, including current TMOS release level, device health, network configuration, and system monitoring and management Availability, including HA configuration, active/standby preferences, network redundancy, connection mirroring, and persistence settings Performance, including optimized service profiles, CPU throughput, simple F5 iRules scripting, virtual server types, and health monitors Security, including secure socket layer (SSL) cipher strengths, port lockdown settings, and administrative access configurations Firepass to BIG-IP Access Policy Manager (APM) Migration Assessment Migrate to BIG-IP APM for Faster, Flexible Access The rapid proliferation of mobile devices, an increasingly dispersed workforce, and the need to secure and optimize content delivery combine to make high-performance, high-concurrency remote access solutions crucial to organizations. Migrating now from a FirePass device to BIG-IP APM ensures your applications remain fast, secure, and available. BIG-IP APM provides a flexible, high-performance access and security solution within an agile infrastructure that will position your organization to effectively support today’s mobile workforce. The F5 Professional Services consultant reviews your current FirePass configuration and conducts a high-level design discussion to understand the target architecture requirements for meeting your organization’s remote access needs. The configuration review includes analysis of web services, landing URIs, authentication method, certificates, master and resource groups, and network, portal, and application access. Proactive Assessment Assess Your F5 Infrastructure Agility An F5 Proactive Assessment Service audits your F5® BIG-IP® products to ensure optimal configuration. Specifically, the Proactive Assessment Service reviews your current environment to uncover potential issues or areas for improvement and makes recommendations that help optimize F5 technologies. The result is an action plan designed to boost your BIG-IP platform performance, strengthen security, and increase availability. Network configuration is assessed with a comprehensive review of infrastructure characteristics in five categories: Operating system, including hotfix level and consistency within products and across BIG-IP device high-availability (HA) pairs Architecture, including virtual servers, pools, network address translation, and address resolution protocol (ARP) settings Security, including password policy, authentication methods, and network forwarding Availability, including fail-over, mirroring, HA configuration, health monitors, and backup policies Performance, including CPU performance graphs, memory consumption, and throughput Another great self-service resource can be found in iHealth, which enables you to verify the proper operation of your BIG-IP system and ensure your hardware and software function at peak efficiency. New to iHealth is a comparison feature that can assist with assessments as well as troubleshooting. iHealth requires registration, but is a free service from F5 designed to ease the support process as well as providing organizations with the means to self-support when desired. Additional Resources: F5 Professional Services F5 Professional Services Data Sheet iHealth Overview BIG-IP iHealth Data Sheet F5 Security Vignette: iHealth F5 Security Vignette: iHealth And Now, A Word About iHealth Need a Little Help Deploying IPv6? We’ve Got Your Back F5 Friday: Goodbye Defense in Depth. Hello Defense in Breadth. F5 Friday: Platform versus Product F5 Friday: Engineering, Experience, and Bacon?F5 Friday: How About a Little Vendor as a Service?
Automated configuration analysis that’s accessible as a service. How cool is that? Tuning infrastructure that’s designed to mediate for applications can be difficult, especially when there are a lot of knobs and buttons to push, pull, twist and turn. It’s often not convenient to wait for a field engineer or walk through a configuration on the phone with a support engineer and it’s certainly frustrating for everyone involved to discover after hours (or days) of searching that the source of a particular problem was a common misconfiguration. F5 aims to change that process with iHealth. IT as a SERVICE The concept of “IT as a Service” is based on self-service; the ability of customers – internal or external – to interact with services on their own that provide both business and technical value. In the world of cloud computing this is becoming synonymous with application and server infrastructure resource provisioning, but as a general theme IT as a Service is about enabling all aspects of IT – from provisioning to management to support – as services that are better able to support the volatile and growing environments in which applications are being deployed. But the focus of IT as a Service is often on the business stakeholder as the end-user, even though for myriad data center services and resources it is other IT folks who need the capability to self-manage, not necessarily business stakeholders. One of the key components of all scalable data center architectures is application delivery. Application delivery comprises a wide variety of technologies – from acceleration to security to optimization and load balancing – and the deployment of such technology often results in complex networking and application networking configurations. It is not always obvious whether a performance issue is the result of a misconfiguration at the network or application network layer, or whether a knob or button simply hasn’t been dialed up- or down. With years of application delivery tuning experience in thousands of organizations, F5 has an extensive knowledge base upon which to draw in virtually any situation. But sharing that knowledge can be difficult, and while F5 has documented many of these “best practices” in solution profiles and technical notes and deployment guides, often times just knowing where to start looking can be a challenge. That’s where iHealth comes in. CONFIGURATION ANALYSIS as a SERVICE An exclusive, online support service available to F5 customers and partners, iHealth is designed to assist in improving performance of application delivery infrastructure. Through performance monitoring and automated analysis, available as a web-based application, iHealth provides insight into BIG-IP deployments and can automatically identify a variety of performance issues and common misconfiguration. iHealth validates product configurations against best practices and displays heuristics data in an easy-to-understand format based on the BIG-IP interface. iHealth is also capable of alerting customers to available upgrades that optimize performance and help assess the impact of industry or regulatory changes, such as the effect on SSL performance when migrating from 1024-bit keys to 2048-bit keys. The iHealth service analyzes configuration and performance data, and prioritizes existing performance challenges. This automated capability leverages F5’s years of experience working with global organizations to provide convenient diagnostic and remediation information and offer detailed recommendations to address any issues. These recommendations also include links to the AskF5 Knowledge Base for detailed solutions and procedures. The BIG-IP iHealth Viewer displays clear summaries of hardware, software, failover configurations and Licensing details, as well as customizable graphs of performance metrics including: CPU usage Throughput Memory RAM cache Additionally, iHealth allows customers a convenient, accurate way in which they can share deployment configuration with F5 professionals should opening a support case become necessary. iHealth can quickly gather all relevant information and provide a concise snapshot for support professionals to work with, helping improve interactions between customers and F5, and ultimately leading to a swifter resolution. iHealth services are now available worldwide at no charge for all F5 customers with a current services maintenance agreement. Customers can access iHealth services directly at http://iHealth.f5.com. Happy analyzing! Related blogs & articles: iHealth Presentation F5 Friday: You’ll Catch More Bees with Honey(pots) F5 Friday: Efficient Long Distance Transfer of VMs with F5 BIG-IP WOM and NetApp Flexcache F5 Friday: Rackspace CloudConnect - Hybrid Architecture in Action All F5 Friday Entries on DevCentral F5 Friday: The 2048-bit Keys to the Kingdom F5 Friday: Gracefully Scaling Down F5 Friday: Beyond the VPN to VAN F5 Friday: Eavesdropping on Availability F5 Friday: Elastic Applications are Enabled by Dynamic Infrastructure The Other Hybrid Cloud Architecture The Goldfish EffectIn 5 Minutes or Less Video - F5's iHealth System
Maintaining your system and troubleshooting issues can be time-consuming and tedious. Whether you want to fix a problem, improve performance, or view your system’s running configuration, F5 BIG-IP iHealth™ can help you accomplish your task quickly and accurately. BIG-IP iHealth consists of BIG-IP iHealth Diagnostics and BIG-IP iHealth Viewer. BIG-IP iHealth Diagnostics identifies issues, including common configuration problems and known software issues. It also provides solutions and links to more information. With BIG-IP iHealth Viewer, you can see the status of your system at-a-glance, drill down for details, and view your network configuration. F5 iHealth system, an exclusive online support service available to F5 customers and partners, is designed to proactively improve the performance of application delivery infrastructures. iHealth is a web-based application that provides insight into BIG-IP product deployments through performance monitoring and automated analysis. iHealth gives users a succinct description of identified issues along with F5's recommendations for addressing them. This comprehensive analysis validates current product configurations against best practices, and displays heuristics data in an easy-to-understand format based on the familiar BIG-IP interface. In offering a proactive approach to troubleshooting, iHealth significantly decreases the time and effort necessary to diagnose and address technical issues. ";" alt="" /> F5’s iHealth System ps Resources: BIG-IP iHealth Product Datasheet (pdf) F5 Introduces Self-Service Support Capabilities and Diagnostic Tools New iHealth service offers improved visibility into BIG-IP environments helping streamline support Improve performance and uptime with F5 BIG-IP iHealth Diagnostics and Viewer DevCentral Weekly Podcast 153 – iHealth F5 Friday: How About a Little Vendor as a Service? iHealth SlideShare presentation F5 Services F5 Customer Support
