iHealth API Part 4 - A Little More Code
In the last chapter, we wrote a quick bash script that used some tools to produce diagnostics summaries for qkviews. Now that you have some experience with the API, maybe you want to do a little bit more. In this article, we're going to explore the data in a little more detail to give you an idea of the kinds of things you can do with the API. Generally speaking, the API methods (sometimes called 'calls') are broken into two different broad categories, "group" or "collection" methods, and then methods that deal with a single qkview ("instance methods"). In this article, we're going to cover some of the collection methods, and how to manage your qkviews. In previous articles, we've restrained ourselves from going hog-wild and kept to the nice safe methods that are essentially read-only methods. That is, we just ask for information from the API, and then consume it and manipulate it locally, without making any changes to the data on the server. This is a nice safe way to get a handle on how the API works, without accidentally deleting anything, or changing an attribute of a QKView without meaning to. It's time to take the training wheels off, call in the goats, and go a little wild. Today, we'll be modifying an attribute of a QKView, or if we're feeling really crazy, we'll modify two of them, maybe even in the same HTTP request. My palms are sweating just writing this... Before we get that crazy, let's take a minute and talk about some collection methods first, and get that out of the way. Collection methods can be very handy, and very powerful, as they allow you to deal with your QKViews en masse. There are three collection methods that we'll talk about, the first one is a way to get a list of all your QKView IDs. This makes it possible to iterate over all the QKViews in your iHealth account while hunting for something. Remember how we got our list of goats in a barn? Similarly: GET qkview-analyzer/api/qkviews will return a list of qkview ids for every qkview in your ihealth account: What if we decide we want to get rid of all our QKViews? We could GET the list of IDs, and then iterate through them, DELETEing each one, but a collection method is much easier: DELETE qkview-analyzer/api/qkviews will wipe all the qkviews from your account. Use with caution, this is the 'nuke from orbit' option. We won't be using it in our projects for this article series, but I thought I'd mention it, as it is very handy for some situations. Another collection method that we *will* be using is this: POST qkview-analyzer/api/qkviews This method allows us to add to our collection of qkviews. Since we're using POST, remember that each time this is executed (even with the exact same qkview file), a new qkview object is created on the server. So you'll only need to do it once for a given qkview file, unless of course, you accidentally DELETE it from your collection, and want to add it again. I've never done anything like that, I'm just speaking hypothetically. So now armed with those collection methods, and with knowledge of our previous scripts that allowed us to examine QKView metadata, let's build up a little script that combines collection methods with some instance methods. Now, the sample qkview won't suffice for this exercise, as it is meant for demonstration purposes, and is used system-wide (and is thus read-only). For this exercise today, we'll be using a real qkview. This article is going to write two scripts. The first will perform an upload of a qkview file to iHealth, using many of the techniques that we learned in our first scripting session. Upload QKView script (note that all the scripts referenced in this series are included, but the one needed for this exercise is upload-qkview.sh) Notice how the authentication parts all look the same? What a great opportunity to refactor these scripts and build up an iHealth API library that our other scripts could use. Extra credit if you take that on! So using the same authentication methods, we need to be able to specify the location of the qkview file that we want to upload. Here is a sample qkview file for you to download and use for these exercises, or feel free to pull a qkview off your own gear, and use that. AskF5 explains how to get a QKView. Here is where the collection method to add a qkview to our collection happens: 96 function upload_qkview { 97 path="$1" 98 CURL_CMD="${CURL} ${ACCEPT_HEADER} ${CURL_OPTS} -F 'qkview=@${path}' -D /dev/stdout https://ihealth-api.f5.com/qkview-analyzer/api/qkviews" 99 [[ $DEBUG ]] && echo "${CURL_CMD}" >&2 100 out="$(eval "${CURL_CMD}")" 101 if [[ $? -ne 0 ]]; then 102 error "Couldn't retrieve diagnostics for ${qid}" 103 fi 104 location=$(echo "${out}" | grep -e '^Location:' | tr -d '\r\n') 105 transformed=${location/Location: /} 106 echo "${transformed}" 107 The upload will finish with an HTTP 302 redirect that points us to the new qkview. After we upload, we cannot just immediately ask for data about the qkview, as there is a certain amount of processing that happens before we can start grilling the server about it. While the server is busy doing all the prep work it needs to do in order to give you back data about the qkview, the server doesn't mind if we politely ask it about how things are going up there in the cloud. To do this, we use a GET request, and the server will respond with an HTTP status code that tells our script how things are going: our code: "ready yet?" server : "not yet" our code: "ready yet?" server : "not yet" sort of like kids on a car trip asking "are we there yet?" over and over and over again. Only instead of the driver getting enraged and turning up the radio to drown them out, our server just responds politely "not yet" until it's done processing. our code: "ready yet?" server : "yup, ready!" Then our code can go about it's business. In the programming trades, this called 'polling'. We poll the server until the server gives us the answer we want. our code: GET qkview-analyzer/api/qkviews/ server : HTTP 202 our code: GET qkview-analyzer/api/qkviews/ server : HTTP 202 our code: GET qkview-analyzer/api/qkviews/ server : HTTP 200 So that's how to add a qkview to your collection. Of course you might not get an HTTP 202 or 200 back, you might get something else, in which case something went wrong in either the upload or the processing. At that point, we should also bail out, and return an error to the runner of the script: 109 function wait_for_state { 110 url="$1" 111 count=0 112 CURL_CMD="${CURL} ${ACCEPT_HEADER} ${CURL_OPTS} -w "%{http_code}" ${url}" 113 [[ $DEBUG ]] && echo "${CURL_CMD}" >&2 114 _status=202 115 time_passed=0 116 while [[ "$_status" -eq 202 ]] && [[ $count -lt ${POLL_COUNT} ]]; do 117 _status="$(eval "${CURL_CMD}")" 118 count=$((count + 1)) 119 time_passed=$((count * POLL_WAIT)) 120 [[ $VERBOSE ]] && echo -ne "waiting (${time_passed} seconds and counting)\r" >&2 121 sleep ${POLL_WAIT} 122 done 123 printf "\nFinished in %s seconds\n" "${time_passed}" >&2 124 if [[ "$_status" -eq 200 ]]; then 125 [[ $VERBOSE ]] && echo "Success - qkview is ready" 126 elif [[ ${count} -ge ${POLL_COUNT} ]]; then 127 error "Timed out waiting for qkview to process" 128 else 129 error "Something went wrong with qkview processing, status: ${_status}" 130 fi 131 } The change-description.sh script (included in the zip linked above) will allow you to update the description field of any number of qkviews with a given chassis serial number. This script will use both a collection method (list all my qkview IDs), and an instance method on several qkviews to update some metadata associated with the qkview. We've introduced yet another verb into our working lexicon, PUT. We use PUT here, because it's a modification on a qkview that no matter how many times we perform it, will result in the same qkview state. This is called idempotency. Unlike, say our POST above in our upload script, which results in new qkviews every time you run it, this PUT may change the state of the qkview the first time, but subsequent identical PUT requests won't change the state further. So now we can submit QKViews, get diagnostic results, and change metadata about the QKView. So what else could we possibly do? The rest of this article series will explore the data in the API and how to retreive and process it.iHealth API Part 3 - A Little Code
We finished the last article with exploring some of the data available in the API with a web browser that was helpful enough to render it in a mildly readable fashion. For most automation projects, however, we don't care if it's easy to read, as long as it's parseable and we can do something with the data that doesn't involve watching streams of xml scroll off the screen. For today, we'll be working with the data from the diagnostics section of the API: We'll use the same data from the API that we explored last time. GET /qkview-analyzer/api/qkviews/0/diagnostics For this script exercise, we'll be using a couple of tools working in the unix shell: bash 4.x xmlstarlet grep When the diagnostics run against a QKView, they either are considered a 'hit' where the diagnostic found an issue in the QKView, or a 'miss', where the diagnostic ran, but found no issue. We'll develop a little script that will provide a quick summary of our hits and misses for a given QKView. In the script we'll deal with the following steps: authentication diagnostics retrieval and asking for subsets of diagnostics diagnostics parsing and selective display Let's start out by writing a couple of functions that will deal with the housekeeping that we need to do in order to connect to the API. First thing we have to do is authenticate ourselves with the credentials that we used when we built our iHealth account. By authenticating, we will obtain a cookie in the HTTP response to our authentication request (if it's successful!) that we will send with every subsequent request to the API. This way, the API knows who we are, which qkviews in the system are ours, and knows that we've proven we're who we're claiming to be. This is currently done by sending a form POST to the login server containing our credentials, and stashing the cookies we get back for later use. curl provides exactly the sort of functionality that we need to perform API work, and bindings for the curl libraries are available for lots and lots of languages if you want to use something other than bash, or want to develop a bigger script later on. We set up some initial values for later use: 13 readonly CURL=/usr/bin/curl ... 23 RESPONSE_FORMAT=${FORMAT:-"xml"} ... 32 ACCEPT_HEADER="-H'Accept: application/vnd.f5.ihealth.api+${RESPONSE_FORMAT}'" Now that the housekeeping is done, we'll need an authentication function: 72 function authenticate { 73 user="$1" 74 pass="$2" 75 # Yup! Security issues here! we're eval'ing with user input. Don't put this code into a CGI script... 76 CURL_CMD="${CURL} --data-urlencode 'userid=${user}' --data-urlencode 'passwd=${pass}' ${CURL_OPTS} https://login.f5.com/resource/login 76 Action.jsp" 77 [[ $DEBUG ]] && echo ${CURL_CMD} 78 79 if [[ ! "$user" ]] || [[ ! "$pass" ]]; then 80 error "missing username or password" 81 fi 82 eval "$CURL_CMD" 83 rc=$? 84 if [[ $rc -ne 0 ]]; then 85 error "curl authentication request failed with exit code: ${rc}" 86 fi 87 88 if ! \grep -e "sso_completed.*1$" ${COOKIEJAR} > /dev/null 2>&1; then 89 error "Authentication failed, check username and password" 90 fi 91 [[ $VERBOSE ]] && echo "Authentication successful" >&2 92 } The way this script is written, it uses the setting of environment variables on the commandline in order to protect the innocent. This allows us to pass sensitive information into a script without it needing to be embedded for others to discover, or show up in the output of ps on a shared machine. You can hardcode it if you wish, but I'd recommend not doing so. Once authentication succeeds, then we go out and grab the diagnostics for the qkview we specified as an argument to the script: 94 function get_diagnostics_hits { 95 qid="$1" 96 CURL_CMD="${CURL} ${ACCEPT_HEADER} ${CURL_OPTS} https://ihealth-api.f5.com/qkview-analyzer/api/qkviews/${qid}/diagnostics?set=hit" 97 [[ $DEBUG ]] && echo "${CURL_CMD}" >&2 98 out="$(eval "${CURL_CMD}")" 99 if [[ $? -ne 0 ]]; then 100 error "Couldn't retrieve diagnostics for ${qid}" 101 fi 102 echo "$out" 103 } Notice how we have set up an Accept header in the request? This tells the API what format you want the response to be in. Diagnostics have a special role in the API, so they are available in a couple format: xml, json, pdf, and csv. Everything else in the API is only available in xml or json. Use the Accept header to specify how you want your response data. If you don't use a response header, then the API assumes you want XML, and returns XML. In this example we're asking for XML explicitly. Now that we have a big pile of XML, we only want a couple pieces of it, so make xmlstarlet dig through it and give us a nice display: 131 #perform some XML extraction, and print it out in a nice readable format 132 diagnostics_count=$(echo ${diagnostics} | ${XMLPROCESSOR} select -t -c 'string(/diagnostic_output/diagnostics/@hit_count)' -) 133 for ((i=1;i<=diagnostics_count;i++)); do 134 printf "%-10s : " $(echo ${diagnostics} | ${XMLPROCESSOR} select -t -c "string(//diagnostic[$i]/@name)" -) 135 printf "%s\n" "$(echo ${diagnostics} | ${XMLPROCESSOR} select -t -c "//diagnostic[$i]//h_header/node()" -)" 136 done See how easy that is? We can do the same thing with json if we want: 140 # in json 141 diagnostics_count=$(echo ${diagnostics} | ${JSONPROCESSOR} -r .diagnostics.hit_count) 142 for ((i=0;i<=diagnostics_count;i++)); do 143 printf "%-10s : " $(echo ${diagnostics} | ${JSONPROCESSOR} -r .diagnostics.diagnostic[$i].name) 144 printf "%s\n" "$(echo ${diagnostics} | ${JSONPROCESSOR} -r .diagnostics.diagnostic[$i].results.h_header)" 145 done If you need to deal with json at the commandline, and don't already know about it, jq is an incredibly powerful tool for handling json, and well worth your time exploring (see Working with iControl REST Data on the Command LIne for more details.) So now that it works, what do all the fields mean? Let's look at the json output and see what we're getting: { "diagnostics": { "filter": "hit", "diagnostic": [ { "name": "H371501", "output": [], "run_data": { "h_importance": "MEDIUM", "match": true }, "results": { "h_action": "Upgrade to version 10.2.2 or later.", "h_name": "H371501", "h_header": "A known issue causes the chmand process to leak memory on this BIG-IP platform", "h_summary": "Due to a known issue, the chassis manager daemon, chmand, leaks memory on BIG-IP systems which contain the Always-On Management (AOM) subsystem. The memory leak is constant regardless of the volume of traffic processed by the system.", "h_sols": [ "http://support.f5.com/kb/en-us/solutions/public/12000/900/sol12941.html" ] } }, name - the diagnostic name output - is an array containing any qkview specific information the diagnostics wants to tell you about run-data - data about the run including importance (low medium high critical) match (if it's a hit or not) results h_action - references to more materials h_name - the diagnostic name again h_header - a title h_summary - a more verbose explanation of the issue h_sols - links to solution articles that go into more depth Customizing the data that is shown in your summary is very easy, by adding the statements that dig into the diagnostics return data. One could, if one wished, build up a reporting system that showed the change in diagnostics results over time for a collection of qkviews if one were so inclined. The full script is available here.iHealth API Part 2 - An Introduction to REST
In the last article, we got some ideas of the kinds of data that iHealth can provide by exploring the wealth of information through a standard web UI. This is a quick easy way to get answers about a couple of your F5 machines, or to do a one-off check of something, or if you are looking for a particular problem, or working with F5 Support to resolve an issue or answer a question. If you have a couple machines, or maybe many many more, however, and you want to do something periodically, like, say, check for new Diagnostics results (remember, we generally update them once a week), then you'll quickly find that opening your browser, logging in, uploading the QKViews, and then reading through the Diagnostics is not necessarily the most efficient method. It can become annoying and tedious for even one or two machines. Since iHealth is a web application, it made the most sense to make the iHealth API a web API as well. There is a dogs breakfast of different types of web APIs, but one of the most prevalent conceptual frameworks in use today is called Representational State Transfer, or REST. The wikipedia entry on Roy Fielding has links to his dissertation if you want to up your street cred significantly. REST provides a nice clean way of retrieving, modifying and deleting things using HTTP in all (well most all) of it's glory. It's hidden from most users (as it should be), and unless you are a web developer, or someone who is dealing with web traffic a lot, there isn't any reason you would need to know this, but the HTTP protocol specifies a number of different ways to ask the server to do something. Now that you will be working with an HTTP API, you're going to have to know a little more about HTTP. Generally the two verbs that a developer will be familiar with is GET and POST. But HTTP has many other verbs, and we'll use most of them in this article series (GET, POST, PUT, and DELETE for now, although there are others), REST gives us a framework for using those verbs to make changes, or view the properties of things on the web. They generally do what they sound like: GET - get information (generally read-only) POST - create a new object (or modify an existing object repeatedly) PUT - modify an object idempotently so that multiple PUTs will result in the same state DELETE - delete an object This is all very abstract, so let's come up with an extremely contrived example: imagine a barn full of goats ( yes, my first job was as a farmhand ). Let's say that you had a robot that you wanted to perform various tasks relating to this barn full of goats. Maybe the first one would be a survey of all the goats in the barn. If the robot spoke REST, then you might send a text message to your robot that looked like this: GET /barn/1/goats Because the robot knows that in REST, that means that you want to know about all the goats in barn 1, your robot would trundle off (or fly, or crawl) to barn 1, and might come back with this data: goat 1: name: Dopey goat 2: name: Speedy Maybe then you wanted to know more about Speedy, so you'd text your robot this: GET /barn/1/goats/2 and your robot would gather up some more information about goat 2: goat 2: name : Speedy eats : Himalayan Blackberry hates : Dopey Okay, so what the hell do goats and barns have to do with iHealth? Well, if we substitute in QKViews for goats, we can start to see some information about our data in the same way that we were able to examine our herds of goats: GET /qkview-analyzer/api/qkviews Will show us a list of the qkviews that we have in our account: This is a sample, and there is a special QKView that isn't show in the listing, but that you should always have in your account, a QKView with an ID of 0. It's a sample QKView that we allow everyone access to in order to see how iHealth works with requireing you to upload anything of your own. So let's pick QKView 0 and load it up. We'll see some details about that QKView: GET /qkview-analyzer/api/qkviews/0 Feel free to poke around. At the bottom of the listing are some "discovery" URLs. Load them up in your browser; there is a ton to explore here, and you can get an idea of the kind of data you can work with without writing a single line of code. Code will come later, as you'll need it for things like adding to your QKView collections, and modifying existing QKView entries, which are tricky to do with just a browser.iHealth API Part 1 - An Introduction to iHealth
Welcome to the inaugural iHealth post (for the http nerds, yes, pun intended). This series will be an exploration and explanation of the iHealth Application Programming Interface (API), and how making the iHealth API a part of your datacenter ecosystem can bring you good fortune, reduce your downtime, help prevent emergency RMAs, and cook your emergency bacon. But before we get to the emergency bacon (spoiler: there is no emergency bacon), let's start with the basic building blocks and concepts that we'll be talking about, just so there is no regrettable misunderstandings later in the series. Let's start with the foundation of this series: iHealth. iHealth If you've spent much time with F5 gear (actual hardware or virtual editions it doesn't matter) in the last couple years, or had any interactions with the F5 support teams, you may have at least heard of iHealth. iHealth is a free, F5 hosted web application that provides deep inspection of BIG-IP, BIG-IQ, and Enterprise Manager system data. We want to help you monitor your systems, tune up your configurations, provide a distant early warning system, and help you keep on top of any issues or opportunities we can. The most important part of that deep inspection that iHealth provides are our Diagnostics. As of this writing, we've got over 900 diagnostic tests that are run against all that system data. Each diagnostic is specifically targeted to look for a very specific issue, or inform you about a particular configuration optimization, or warn you about a particular security vulnerability. We don't just warn you, or offer generic advice however. We also provide the specific items in your configuration that might be susceptible, or in need of a little love, and we provide extensive [Ask F5] solution articles to back up each Diagnostic to ensure that you have all the context, details, and knowledge to get things fixed up before they become a bigger problem. The F5 iHealth team releases new and updated Diagnostics every week, so the list of Diagnostics that are 'hits' (identify your gear as exhibiting the symptoms the diagnostic is looking for), may change from week to week. We get ideas for Diagnostics from all over F5, as well as from our customers (that's you). Got an idea of something you want iHealth to check for on all your machines? Jump into iHealth and hit the feedback button, and describe what you want to do. That feedback button creates a ticket that the F5 iHealth team will follow up on. The far-seeing among us might postulate that doing weekly checks of our gear might be a great way to keep our ship running tight with very little cost. That would be an astute observation indeed, and this article series will address that observation in some detail in the weeks to follow. iHealth has lots of other uses as well, but the Diagnostics are the crown jewel in the iHealth tiara. You can start using this tool, if you haven't already, by visiting https://ihealth.f5.com/. You might have to create a (free) account, but that's the only obstacle. We'll be depending on your access to your iHealth account for the duration of this series, so if you don't have an account, go sign up for one now. After you get your account, you can access https://ihealth.f5.com/ and you'll notice there is a sample QKView in your account already. We provision a sample in there to give you something to work with immediately, and, conveniently, it will give us a data set to use for the rest of this series. The UI is fairly extensive, and gives you lots of information about your machine, including things like End of Life notices for your hardware and software, if you've got a support contract, we let you know when that contract ends, as well as displaying lots of information about your F5 gear. Customizable performance graphs? Yup. Command-line output at the moment the QKView was taken? Yup. Diagnostics? Yup, we already covered that. Configuration information? Yup. So if all that data is already available, why am I bothering with an API? Well, the rest of the series should help illuminate the answer to that question, and you'll end up writing a basic application to make use of all the data iHealth provides, which should give you plenty of ideas about how iHealth in all it's forms can be an incredibly helpful tool.
