iControl REST + jq Cookbook - Part 1: Basics
In this first part, simplefiltersand basic command options are explained: Specifically, filters for extracting particular properties, counting a number of objects, and tabularizing the results. All the examples are from actual iControl REST calls: You may find some of them immediately applicable to your requirements. A filter is an instruction that describes how subsets of incoming JSON texts are parsed, extracted, modified and formatted. jq takes the filter string as a sole argument (other than the command options starting with - and input files). Since it is a single string, you need to single-quote it to protect from bash interpreting the filter contents. You can omit the single quotes for filters with no special characters, but it is highly recommended to use '' all the time. Let's get started. Format the entire JSON body An iControl REST GET request returns a single line JSON body without any line breaks or indentations, which is unfriendly for ordinary human eyes. To pretty-print the entire JSON body, pipe the response body to the filter "dot" ( . ). The following example is for formatting a response from the tmsh list ltm virtual node CentOS-internal20 equivalent call. curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq '.'. { "kind": "tm:ltm:node:nodestate", "name": "CentOS-internal20", "fullPath": "CentOS-internal20", ... # Skip "fqdn": { # A nested object "addressFamily": "ipv4", "autopopulate": "disabled", "downInterval": 5, "interval": "3600" }, ... # Skip "state": "up" } The . denotes the top level object. jq searches for the starting point of the top level object from the incoming text, and recursively prints all the components under it with indentations. Sort by property names To sort by the top-level property names (keys), add the --sort-keys option (the shortcut is -S ). curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq -S '.' { "address": "10.200.20.10", "connectionLimit": 0, "dynamicRatio": 1, ... } Extract a specific property You can reference any property directly under the top-level by appending the property name to the top-level dot. For example, to extract the value of fullPath property from the above call, run the command below: curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq '.fullPath' "CentOS-internal20" The fqdn property value is an object (nested object), hence the .fqdn filter yields the entire object including its surrounding curly braces. curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq '.fqdn' { "addressFamily": "ipv4", "autopopulate": "disabled", "downInterval": 5, "interval": "3600" } To reference a property inside the object, append another dot, then the property name. For example, for the downInterval property, run the command below: curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq '.fqdn.downInterval' 5 Note that a string value is double-quoted ( "" ) while a numeric value is not (seeRFC 8259). To remove the surrounding double quotes from string values, use the --raw-output option ( -r ). For example, to extract the value of addressFamily property inside fqdn , run the command below: curl -sku admin:admin https://localhost/mgmt/tm/ltm/node/CentOS-internal20 | jq -r '.fqdn.addressFamily' ipv4 Error Any input that is not properly JSON formatted causes error. In the following example, an incorrect password (xxx) is specified in the curl call. Because the BIG-IP returns an HTML response (instead of JSON), jq fails. # > /dev/null is added to show only the stderr from jq curl -sku admin:xxx https://<host>/mgmt/tm/sys/version | jq '.' > /dev/null parse error: Invalid numeric literal at line 1, column 10 A logical error is not reported. For example, the filter .fqdn.addressfamily is syntactically valid, but there is no such property in the fqdn property (must beFamily). When a property that does not present is referenced, jq yields null . curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq '.fqdn.addressfamily' null Substitute the authentication token to a shell variable The --raw-output option is handy when you need to directly use the value in a bash script. The example below is for extracting the authentication token from an iControl REST call POST /mgmt/shared/authn/login and substituting it to a shell variable TOKEN for later use. # Get the token through the '.token.token' filter and subsititute to the shell variable TOKEN TOKEN=`curl -sk https://<host>/mgmt/shared/authn/login -X POST -H "Content-Type: application/json" \ -d '{"username":"<username>", "password":"<password>", "loginProviderName":"tmos"}' | \ jq -r '.token.token'` # Check the token. Should not be enclosed in "" echo $TOKEN CL4SBLFG4GEOLQUFX4FTHSREDM # Run an iControl REST call using the token (e.g., the 'tmsh show sys version' equivalent) curl -sk https://<host>/mgmt/tm/sys/version -H "X-F5-Auth-Token: $TOKEN" | jq . { "kind": "tm:sys:version:versionstats", "selfLink": "https://localhost/mgmt/tm/sys/version?ver=15.1.1", ... } The iControl REST call returns a valid JSON body even the authentication fails with 401 as below: { "code": 401, "message": "Authentication failed.", "originalRequestBody": "{\"username\":\"satoshi\",\"loginProviderName\":\"tmos\",\"generation\":0,\"lastUpdateMicros\":0}", "referer": "192.168.184.1", "restOperationId": 6611604, "kind": ":resterrorresponse" } Since the response does not have .token.token property, the jq filter returns null . The subsequent call would fail because of the wrong credential. You can use the --exit-status option ( -e ) to detect such logical error. Normally, jq returns the exit code 0 even if the property is not present, however, with -e , it returns 1. curl -sk https://<host>/mgmt/shared/authn/login -X POST -H "Content-Type: application/json" \ -d '{"username":"<username>", "password":"<password>", "loginProviderName":"tmos"}' | jq -r '.token.token' null echo $? # The exit code indicates "Success". 0 curl -sk https://$HOST/mgmt/shared/authn/login -X POST -H "Content-Type: application/json" \ -d '{"username":"<username>", "password":"<password>", "loginProviderName":"tmos"}' | jq -er '.token.token' null $ echo $? # The exit code indicates 1 (non-zero) 1 Use $? in your script to exist when the token request fails. Using a slightly advanced technique, you can raise an error if .token.token is null using the if-then-else statement along with the error function (see theConditionals and Comparisonssection of the jq manual): ... | jq -r '.token.token | if . == null then error("no token") else . end' Extract multiple properties To extract multiple properties, concatenate them with comma ( , ). For example, to extract the virtual server's name ( .name ), destination address ( .destination ) and its pool ( .pool ) from a GET /mgmt/tm/ltm/virtual/vs response, run the command below: curl -sku admin:admin https://<host>/mgmt/tm/ltm/virtual/vs | jq '.name, .destination, .pool' "vs" "/Common/172.16.10.50:80" "/Common/Pool-CentOS80" The comma functions like Unix's tee: The stream of JSON text is individually fed into each designation, hence yielding three values. The leading . is necessary for each designation because you need to explicitly tell that it should reference from the top-level. Format the output You can concatenate three individual outputs to form a single line using the --join-output option ( -j ). The option also removed the double quotes around the strings, just like the --raw-output option. curl -sku admin:admin https://localhost/mgmt/tm/ltm/virtual/vs | jq -r '.name, .destination, .pool' vs/Common/172.16.10.50:80/Common/Pool-CentOS80 Probably, this is not exactly what you want. You can insert separator characters in between, separated by the commas as if they were properties. These designations also receive the input but output the specified literals without doing anything to the input. Here is an example: curl -sku admin:admin https://<host>/mgmt/tm/ltm/virtual/vs | \ jq -jr '.name, ."\t", destination, "\t", .pool, "\n"' vs /Common/172.16.10.50:80 /Common/Pool-CentOS80 Count a number of objects A GET request to a component generally returns a list (array) of objects. For example, GET /mgmt/tm/ltm/virtual returns a list of virtual servers and their properties (equivalent to tmsh list ltm virtual ). GET /mgmt/tm/ltm/pool returns a list of pools ( tmsh list ltm pool ). Here is an example from GET /mgmt/tm/ltm/pool : curl -sku admin:admin https://<host>/$HOST/mgmt/tm/ltm/pool | jq '.' { "kind": "tm:ltm:pool:poolcollectionstate", "selfLink": "https://localhost/mgmt/tm/ltm/pool?ver=15.1.1", "items": [ # the array starts { # 1st pool object starts "kind": "tm:ltm:pool:poolstate", "name": "Pool-CentOS80", ... }, # 1st pool object ends { # 2nd pool object starts "kind": "tm:ltm:pool:poolstate", "name": "Pool-CentOS443", ... } # 2nd pool object ends ] # the array ends } Each configuration object is an element in the items array. You can extract the entire items array by specifying .items as described in the previous examples. You can feed the resulting array to another processing filter through the pipe ( | ) - Just like passing the intermediate result to the next command in an Unix command chain. For example, you can count the number of pools by passing the .items array to the jq length function. curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | jq '.items | length' 3 As the result shows, this BIG-IP has three pool objects. Iterate through an array The .items[] filter yields individual object. The filter is fairly similar to the previous .items but has the additional [] . This [] is an iterator, which loops around the array and yields the elements one by one. These elements are independent of each other, or not structured. Let's try both .items and .items[] to see the difference. curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | jq '.items' [ # The value of .items is an array! { "kind": "tm:ltm:pool:poolstate", "name": "Pool-CentO8080", ... }, { "kind": "tm:ltm:pool:poolstate", "name": "Pool-CentOS80", ... }, { "kind": "tm:ltm:pool:poolstate", "name": "Pool-CentOS443", ... } ] # The array closing ] $ curl -sku $PASS https://$HOST/mgmt/tm/ltm/pool | jq '.items[]' { # Three independent elements, hence no []. "kind": "tm:ltm:pool:poolstate", "name": "Pool-CentO8080", ... } # No comma { "kind": "tm:ltm:pool:poolstate", "name": "Pool-CentOS80", ... Because of these differences, the length function counts the number of properties in each pool object, hence yields 26 three times (from the three pools): curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | jq '.items[] | length' 26 # The length of Pool-CentO8080 26 # The length of Pool-CentOS80 26 # The length of Pool-CentOS443 Since .items[] yields three pool objects separately, you can now extract each individual property inside the object: curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | jq -r '.items[].fullPath' /Common/Pool-CentO8080 /Common/Pool-CentOS80 /Common/Pool-CentOS443 If you also want to extract the monitor names for each object, you first iterate through the pool object ( .items[] ), then feed the output (each object) to the comma separated list of properties through the pipe. curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | jq '.items[] | .fullPath, .monitor' /Common/Pool-CentO8080 /Common/gateway_icmp /Common/Pool-CentOS80 /Common/http /Common/Pool-CentOS443 /Common/https Obtain the full pool information Printing the pool name and monitor in one line would be nicer. You might have already tried the --join-output option, however, unfortunately, it prints all values in a single line. curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | \ jq -j '.items[] | .fullPath, .monitor' /Common/Pool-CentO8080/Common/gateway_icmp/Common/Pool-CentOS80/Common/http/Common/Pool-CentOS443/Common/https You should use the join function for concatenating the output per pool. curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | \ jq -r '.items[] | [.fullPath, .monitor] | join("\t")' /Common/Pool-CentO8080 /Common/gateway_icmp /Common/Pool-CentOS80 /Common/http /Common/Pool-CentOS443 /Common/https Please note the [] surrounding the two properties. This is for creating an array containing two elements (fullPath and monitor values). This pre-processing is required because the join function can concatenates only array elements. Now, let's get the pool name, monitor and its members. The information on the associated pool members is stored in the membersReference property but it is normally just a link. Here's an example from GET /mgmt/tm/ltm/pool/Pool-CentOS80 : curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool/Pool-CentOS80 | \ jq -r '.membersReference' { "link": "https://localhost/mgmt/tm/ltm/pool/~Common~Pool-CentOS80/members?ver=15.1.2", "isSubcollection": true } To obtain the actual information such as pool names, add the ?expandSubcollections=true query option to the REST call: curl -sku $PASS https://$HOST/mgmt/tm/ltm/pool/Pool-CentOS80?expandSubcollections=true | \ jq -r '.membersReference' { "link": "https://localhost/mgmt/tm/ltm/pool/~Common~Pool-CentOS80/members?ver=15.1.2", "isSubcollection": true, "items": [ { "kind": "tm:ltm:pool:members:membersstate", "name": "CentOS-internal20:80", ... }, { "kind": "tm:ltm:pool:members:membersstate", "name": "CentOS-internal30:80", ... } ] } Because there could be more than one pool members in a pool, the member objects are stored in an array ( .items ). You can reference the names by applying the .membersReferences.items[].name filter. Now, combining all the above yields the filter for generating the pool information in a tabular format. curl -sku $PASS https://$HOST/mgmt/tm/ltm/pool?expandSubcollections=true | \ jq -r '.items[] | [.fullPath, .monitor, .membersReference.items[].name] | join("\t")' /Common/Pool-CentO8080 /Common/gateway_icmp CentOS-internal20:8080 /Common/Pool-CentOS80 /Common/http CentOS-internal20:80 CentOS-internal30:80 /Common/Pool-CentOS443 /Common/https CentOS-internal20:443 CentOS-internal30:443 Continue toPart 2iControl REST + jq Cookbook - Part 2: Intermediate
Part 2 deals with slightly complex use cases: Manipulating array elements, looking for strings that match a certain criteria and handling both property names (keys) and values to generate a tabularized output. iControl REST sometimes returns a bare output from the underlying tmsh (e.g., monitor stats). Toward the end, filters for such row strings are discussed. Additionally, methods for converting Unix epoch timestamps are presented. If you find some jq notations difficult, please revisit thePart 1. Extract an array element A single array may contain a large number of elements. For example, the records property of an LTM internal data-group (e.g., tmsh list ltm data-group internal aol ) stores as many entries as you want in a single array. Here's an example from the AOL internal data-group: curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | jq '.records' [ { "name": "64.12.96.0/19", "data": "" }, ... { "name": "207.200.112.0/21", "data": "" } ] You can selectively extract a single element by specifying the array index number to the records property. The index number starts from 0, so [1] means the 2nd element. curl -sku $PASS https://<host>/mgmt/tm/ltm/data-group/internal/aol | jq '.records[1]' { # 0th record element "name": "195.93.16.0/20", "data": "" } You can specify multiple indexes separated by , . In the example below, only the name properties are extracted by additional | .name . curl -sku $PASS https://<host>/mgmt/tm/ltm/data-group/internal/aol | \ jq -r '.records[2, 3, 5] | .name' 195.93.48.0/22 # The name property value from 2nd 195.93.64.0/19 # The name property value from 3rd 198.81.0.0/22 # The name property value from 5th Note that jq does not raise any error for an out-of-range index: It silently returns null (just like referencing a non-existing property). curl -sku $PASS https://<host>/mgmt/tm/ltm/data-group/internal/aol | jq '.records[100]' null Slice an array You can extract a subset of array by slicing: Just specify the start (inclusive) and stop (exclusive) indexes separated by colon ( : ) inside the [] . For example, to extract the names (IP addresses) from the top 4 elements in the AOL data-group records, run the following command: curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | \ jq -r '.records[0:4] | .[].name' 64.12.96.0/19 # The name property value from 0th 195.93.16.0/20 195.93.48.0/22 195.93.64.0/19 # The name property value from 3rd You might have noticed the subtle difference from the previous comma separated indexes: The additional [] is prepended to the .name filter. This is because the comma separated indexing ( .records[2, 3, 5] ) yields three independent objects (with the name and data properties), while the slice generates an array containing the record objects. Because it is an array, you need to iterate through it to process each record. When the start index number is omitted, zero is used ( [0:4] and [:4] are identical). When the end index number is omitted, it assumes the end of the array. The following example extracts the last 4 IP addresses from the AOL data-group (it currently contains 14 records) - from the 10th to the 13th. curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | \ jq -r '.records[10:] | .[].name' 205.188.146.144/30 # The name property value from 10th 205.188.192.0/20 205.188.208.0/23 207.200.112.0/21 # The name property value from last Applying the slice to a string yields its substring. For example, you can extract the first 8 characters from "iControlREST" like this: $ echo '"iControlREST"' | jq '.[:8]' # Piping "iControlREST" to jq "iControl" Find the element Having said that, we seldom look for elements by their index numbers. We normally look for a pattern. For example, you want to find the data value of the "useragent6" in the sys_APM_MS_Office_OFBA_DG internal data-group. curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/sys_APM_MS_Office_OFBA_DG | \ jq -r '.records[] | [.name, .data] | join("\t")' ie_sp_session_sharing_enabled 0 ie_sp_session_sharing_inactivity_timeout 60 ofba_auth_dialog_size 800x600 useragent1 microsoft data access internet publishing provider useragent2 office protocol discovery useragent3 microsoft office useragent4 non-browser useragent5 msoffice 12 useragent6 microsoft-webdav-miniredir # Looking for this one useragent7 webdav-miniredir useragent9 ms frontpage 1[23456789] useragent10 onenote You can extract only the specific element using the select function. Specify the condition for the selection in the function argument. Usual conditional statements that yields a Boolean can be used: For example, the == operator returns true if two operands are equal. echo '["BIG-IP", "BIG-IQ"]' | jq '.[] | . == "BIG-IP"' true # BIG-IP is BIG-IP false # BIG-IQ is not BIG-IP The select function outputs the input only when the condition in the argument is true. echo '["BIG-IP", "BIG-IQ"]' | jq '.[] | select(. == "BIG-IP")' "BIG-IP" With these techniques, you can extract the record (object) that contains the "useragent6" in the name property: curl -sku $PASS https://<host>/mgmt/tm/ltm/data-group/internal/sys_APM_MS_Office_OFBA_DG | \ jq -r '.records[] | select(.name == "useragent6")' { "name": "useragent6", "data": "microsoft-webdav-miniredir" } The following comparison operators can be used in jq: != , > , >= , < and <= . Find matching elements The select function accepts anything as the conditional statement as long as it returns true or false . For example, the startswith and endswith functions. They compare the input against their argument and return a Boolean. Unlike == , where you explicitly compare the incoming data ( . ) and the string, you only need to specify the string to compare. Let's find the IP addresses that start with "195" in the AOL internal data-group. curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | \ jq -r '.records[] | .name | select(startswith("195"))' 195.93.16.0/20 195.93.48.0/22 195.93.64.0/19 195.93.96.0/19 Here is an example for endswith , looking for the "/20" subnets: curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | \ jq -r '.records[].name | select(endswith("/20"))' 195.93.16.0/20 198.81.16.0/20 205.188.112.0/20 205.188.192.0/20 Regular expressions? Yes, jq comes with standard regular expression functions such as test and match . Here's an example for finding IP addresses that contain 96, dot and any digits: curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | \ jq -r '.records[] | .name | select(test("96.\\d+"))' 64.12.96.0/19 195.93.96.0/19 The jq regular expression is Perl-compatible, however note that backslashes must be escaped (e.g., \\d+ instead of just \d+ ). For more information on the regular expression feature, refer tojq manual. The RegEx functions are available only when the jq was compiled with the regular expression libraryOniguruma. Without the library, you will receive the jq: error (at <stdin>:0): jq was compiled without ONIGURUMA regex libary. match/test/sub and related functions are not available error upon the use of any RegEx function. The jq on a BIG-IP doesnotcome with the library. Extract property names and values Sometimes, you need to obtain both property names (keys) and their values, however, the .[] filter only yields the values. To get both, filter the incoming objects with the to_entries function (it acts like the Python's items() ). For example, GET /mgmt/tm/ltm/monitor/http/http ( tmsh list ltm monitor http http equivalent) returns the following object: curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http | jq '.' { "kind": "tm:ltm:monitor:http:httpstate", "name": "http", ... } The to_entries generates an array. Its elements are objects and each object consists of two properties: "key" and "value". The key property keeps the original property name while the value property stores the value of the property. curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http | jq 'to_entries' [ { # Object from the 0th property "key": "kind", # 0th property name "value": "tm:ltm:monitor:http:httpstate" # 0th property value }, { "key": "name", "value": "http" }, ... ] By piping the result to .key, .value , you now have both property names and values. curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http | \ jq -r 'to_entries[] | .key, .value' kind # 0th property name tm:ltm:monitor:http:httpstate # 0th property value name # 1st property name http # 1st property value ... Obviously, it's much better to have the key and value in the same line. You can do so by using the array constructor [] and the join function as described inPart 1. curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http | \ jq -r 'to_entries[] | [.key, .value] | join(" ... ")' kind ... tm:ltm:monitor:http:httpstate name ... http fullPath ... http generation ... 0 selfLink ... https://localhost/mgmt/tm/ltm/monitor/http/http?ver=15.1.2 adaptive ... disabled adaptiveDivergenceType ... relative ... Extract a property from a deeply nested object Some iControl REST responses are deeply nested. For example, the BIG-IP version number is found in the 6th level in a GET /mgmt/tm/sys/version response: curl -sku admin:admin https://<host>/mgmt/tm/sys/version | jq '.' { "kind": "tm:sys:version:versionstats", "selfLink": "https://localhost/mgmt/tm/sys/version?ver=15.1.1", "entries": { # top level "https://localhost/mgmt/tm/sys/version/0": { # 2nd level "nestedStats": { # 3rd level "entries": { # 4th level "Build": { # 5th level "description": "0.0.6" # 6th level }, ... "Version": { "description": "15.1.1" # HERE! } ... Obviously, you can reference the description property in the Version object by chaining the property names with dots from top to bottom (you need to double-quote the URL-like property as it contains sensitive characters). curl -sku admin:admin https://<host>/mgmt/tm/sys/version | \ jq '.entries."https://localhost/mgmt/tm/sys/version/0".nestedStats.entries.Version.description' "15.1.1" That's too tedious (and error prone). In this case, you can use [] as a catch-all property for the 2nd, 3rd and 4th levels. As you have seen, the iterator [] yields multiple elements in an array or object, however, because these levels only contain one property, it is as good as referencing specifically. curl -sku admin:admin https://<host>/mgmt/tm/sys/version | \ jq '.entries[][][].Version.description' "15.1.1" Tabularize all the version information Producing a tabularized version information should be easy by now: Combination of the -r option and the to_entries and join functions that you have already seen. curl -sku admin:admin https://<host>/mgmt/tm/sys/version | \ jq -r '.entries[][][] | to_entries[] | [.key, .value.description] | join("\t")' Build 0.0.6 Date Thu Oct 8 02:52:59 PDT 2020 Edition Point Release 0 Product BIG-IP Title Main Package Version 15.1.1 The .entries[][][] yields the 5th level objects: e.g., "Build": {"description": "0.0.6"} . The to_entries function then generates an object with the key and value properties: The key is "Build" and the "value" is {"description":"0.0.6"} in this case. You can access the data by referencing .key and .value.description . Process raw monitor output Some iControl REST calls return the raw output from tmsh. For example, GET /mgmt/tm/ltm/monitor/http/http/stats returns a blob of lines from tmsh show ltm monitor http http in the .apiRawValues.apianonymous property. curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http/stats | \ jq '.apiRawValues.apiAnonymous' "---------------------------\nLTM::Monitor /Common/http \n------------------ ..." The output is a single string with the line feeds expressed in the literal "\n" (backslash + n). You can simply output it using the --raw-output option. curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http/stats | \ jq -r '.apiRawValues.apiAnonymous' --------------------------- LTM::Monitor /Common/http --------------------------- Destination: 10.200.20.10:80 State time: up for 8hrs:9mins:13sec | Last error: N/A @2021.01.06 08:44:24 ... Hmmm ... This is too easy for the intermediate level. Let's achieve the same goal in a harder way: Replace the \n (2F 6E) with LF (0A) using the string replacement (substitution) function sub . sub requires the regular expression library. curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http/stats | \ jq -r '.apiRawValues.apiAnonymous | sub("\\n"; "\n")' --------------------------- LTM::Monitor /Common/http --------------------------- Destination: 10.200.20.10:80 State time: down for 6hrs:3mins:51sec | Last error: No successful responses received before deadline. @2020.10.30 10:49:33 ... The first argument of the sub function is afromstring and the second one is atostring. The argument separator is ; (semicolon). Note that you need to escape the literal backslash by another backslash ( \\n ). You can also use regular expressions. Format 'list sys global-settings' output Another response that needs massaging might be the one from tmsh list sys global-settings file-blacklist-path-prefix . As you can see below, it returns a single string, consisting of paths that are enclosed in {} . tmsh list sys global-settings file-blacklist-path-prefix sys global-settings { file-blacklist-path-prefix "{/shared/3dns/} {/shared/bin/} {/shared/core/} ... } Naturally, the equivalent iControl REST call returns a very long string. We can check how long it is by passing the string to the length function. It returns a number of characters when a string is fed (a number of elements or properties in an array or object): curl -sku admin:admin https://$HOST/mgmt/tm/sys/global-settings | \ jq '.fileBlacklistPathPrefix | length' 469 # 469 characters long You can split the string at each whitespace using the split function (this is not a regular expression function). The function generates an array consisting of substrings. curl -sku admin:admin https://<host>/mgmt/tm/sys/global-settings | \ jq -r '.fileBlacklistPathPrefix | split(" ")' [ "{/shared/3dns/}", "{/shared/bin/}", "{/shared/core/}", ... ] If you think the curly braces are obnoxious, you can remove them by feeding it to the gsub regular expression function: a sibling of sub that performs substitution globally. gsub requires the regular expression library. curl -sku admin:admin https://<host>/mgmt/tm/sys/global-settings| \ jq -r '.fileBlacklistPathPrefix | split(" ") | .[] | gsub("{|}"; "")' /shared/3dns/ /shared/bin/ /shared/core/ ... The first argument for gsub is afromstring. Here, a regular expression ("matches either { or } ") is used. The second argument is atostring. Here, any { or } is globally replaced with an empty string. Again, note that the argument separator is ; . You can use the sub command for global replacement by specifying the g (global) flag in the third argument. curl -sku admin:admin https://<host>/mgmt/tm/sys/global-settings | \ jq -r '.fileBlacklistPathPrefix | split(" ") | .[] | sub("{|}"; ""; "g")' Convert Unix epoch time Some time related property values are expressed in Unix epoch in microseconds (10 -6 ). For example, the lastUpdateMicros and expirationMicros properties in an authentication token object represent the timestamps for last update and expiration in microseconds. curl -sku admin:admin https://<host>/mgmt/shared/authz/tokens/LKMRTZ3TZY5PFHJI5HZNRCDUEM | \ jq -r '.lastUpdateMicros, .expirationMicros' 1604038137666701 # Last Update date/time in μs 1604039337667000 # Expiration date/time in μs You can convert the numbers to human readable format using the todate function. curl -sku admin:admin https://<host>/mgmt/shared/authz/tokens/LKMRTZ3TZY5PFHJI5HZNRCDUEM | \ jq -r '.lastUpdateMicros, .expirationMicros | ./(1000*1000) | todate' 2020-10-30T06:08:57Z 2020-10-30T06:28:57Z The expression in the middle of the filter is an arithmetic computation: It divides the incoming time ( . , which is in microsecond) by 10 6 . This is required because the todate function expects the Unix epoch time in seconds. The literal 1000*1000 looks ugly, but it is better than 1000000 (IMHO). If you prefer, you can use the pow function to compute 10 to the power of negative 6. $ echo 1604039337667000 | jq '. * pow(10; -6) | todate' "2020-10-30T06:28:57Z" jq is equipped with standard C math functions (such as cos and log ). Refer to thejq manual. Continue toPart 3.iControl REST + jq Cookbook - Part 3: Advanced Topics
In Part 3, some useful filters from actual customer cases are presented: i.e., replication of an existing object and data-group modification. The article also shows easy ways to summarize the iControl REST log files. In the last section, a method to access the Github API to retrieve information aboutF5 Automation Toolchain. Replicate an object with slight modifications Sometimes you want to replicate a configuration object with a slight changes to a few properties. jq can do that for you. Just call a GET request against the source object, pipe the response to a jq filter to modify the values of properties, and POST the result back to the BIG-IP. In the example that follows, I will create a virtual server object from the existing virtual "vs" with the name, destination and pool properties changed. Let's check the existing properties first. curl -sku admin:admin https://<host>/mgmt/tm/ltm/virtual/vs | \ jq -r '.name, .destination, .pool' vs /Common/172.16.10.50:80 /Common/Pool-CentOS80 Use the = assignment operator to modify the name property value, say to "vs8080". jq outputs the entire JSON text with only that property value changed. curl -sku admin:admin https://<host>/mgmt/tm/ltm/virtual/vs | \ jq '.name="vs8080"' { "kind": "tm:ltm:virtual:virtualstate", "name": "vs8080", # name changed. "fullPath": "vs", ... By repeatedly piping the result to the next assignment, you can modify other properties too: i.e., curl -sku admin:admin https://<host>/mgmt/tm/ltm/virtual/vs | \ jq '.name="vs8080" | .destination="/Common/172.16.10.50:8080" | .pool="/Common/Pool-CentO8080"' { "kind": "tm:ltm:virtual:virtualstate", "name": "vs8080", # name changed ... "destination": "/Common/172.16.10.50:8080", # destination changed ... "pool": "/Common/Pool-CentO8080", # pool changed ... Redirect the output to a file (e.g., virtual8080.json ). Post it to the BIG-IP to create the new virtual server object. curl -sku admin:admin https://<host>/mgmt/tm/ltm/virtual \ -H "Content-type: application/json" -X POST \ -d@virtual8080.json The virtual8080.json file contains some references to the original configurations: For example, the selfLink property is still pointing to the virtual vs, and the poolReferences to the Pool-CentOS80. No worry. iControl REST will relink the paths to the new objects. Optionally, you can selectively remove the properties using the del function. Specify the property names you would like to get rid of in the argument. Use the comma ( , ) for multiple properties. curl -sku admin:admin https://<host>/mgmt/tm/ltm/virtual/vs | \ jq '.name="vs8080" | .destination="/Common/172.16.10.50:8080" | .pool="/Common/Pool-CentO8080" | del(.selfLink, .poolReference, .policiesReference, .profilesReference)' Note that you do not add\to the end of second line. This part is not a shell. Modify internal data-group records One of the frequently asked questions is an iControl REST method to partially add/modify/delete data-group records. Unfortunately, there is none. You need to obtain the current record set, modify it locally, then submit a PATCH request. jq can make this process easier. You can get the records property value from the following jq filter. As you can see, the call returns an array. curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/TestDg | \ jq '.records' [ { "name": "name", "data": "Cha Siu Bao" }, { "name": "type", "data": "large" } ] You can add a record object (consisting of the name and data properties) to the array using the + operator. Note that the right-hand operand must also be an array. curl -sku $PASS https://$HOST/mgmt/tm/ltm/data-group/internal/TestDg | \ jq '.records + [{"name": "price", "data": 7}]' [ { "name": "name", "data": "Cha Siu Bao" }, { "name": "type", "data": "large" }, { "name": "price", "data": 7 } ] The patch data you want to send is an object that consists of the records property with the name and value: i.e., "records":[ {...}, {...}, {...}] . You can create the object by specifying the property name, colon ( : ) and the value and enclosing them with the {} . curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/TestDg | \ jq '{"records": (.records + [{"name": "price", "data": 7}])}' { "records": [ { "name": "name", "data": "Cha Siu Bao" }, { "name": "type", "data": "large" }, { "name": "price", "data": 7 } ] } Note that the parentheses around the value are required to keep the value in one blob. Now, pipe the output to a file (here tmpData ) and PATCH it to the endpoint. curl -sku $PASS https://$HOST/mgmt/tm/ltm/data-group/internal/TestDg \ -X PATCH -H "Content-type: application/json" -d@tmpData | jq . { "kind": "tm:ltm:data-group:internal:internalstate", "name": "TestDg", "fullPath": "TestDg", "generation": 77, "selfLink": "https://localhost/mgmt/tm/ltm/data-group/internal/TestDg?ver=15.1.2", "type": "string", "records": [ { "name": "name", "data": "Cha Siu Bao" }, { "name": "price", "data": "7" }, { "name": "type", "data": "large" } ] } Use the select function to remove a record by name. The function selectively yields the objects if the statement is the argument returns true. To exclude an object, use != . In the following example, the record with the name "price" is removed. curl -sku $PASS https://$HOST/mgmt/tm/ltm/data-group/internal/TestDg | \ jq '{"records": [(.records[] | select(.name != "price"))]}' { "records": [ { "name": "name", "data": "Cha Siu Bao" }, { "name": "type", "data": "large" } ] } Now, pipe the output to a file and PATCH it. curl -sku $PASS https://$HOST/mgmt/tm/ltm/data-group/internal/TestDg \ -X PATCH -H "Content-type: application/json" \ -d@tmpData | jq . { "kind": "tm:ltm:data-group:internal:internalstate", "name": "TestDg", "fullPath": "TestDg", "generation": 79, "selfLink": "https://localhost/mgmt/tm/ltm/data-group/internal/TestDg?ver=15.1.2", "type": "string", "records": [ { "name": "name", "data": "Cha Siu Bao" }, { "name": "type", "data": "large" } ] } Similarly, you can replace a record using the if-then-else-end syntax. The if statement accepts standard conditional operators such as == . Note that jq's if-then-else-end always requires the else block. You need to explicitly put . in the else block to output the existing value. Also, it requires end at the end of statement. The following example modifies the object's data value to "medium" if the name property value is "type": $ curl -sku $PASS https://$HOST/mgmt/tm/ltm/data-group/internal/TestDg | \ jq '{"records": [(.records[] | if .name == "type" then .data = "medium" else . end)]}' { "records": [ { "name": "name", "data": "Cha Siu Bao" }, { "name": "type", "data": "medium" } ] } Process restjavad-autit.log /var/log/restjavad-audit.*.log (where * is a number) keeps records of REST calls as shown in the example below (manually edited for readability): [I][784][12 May 2020 05:17:34 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"GET","uri":"http://localhost:8100/mgmt/tm/sys/available", "status":400,"from":"com.f5.rest.common.RestAvailabilityEntry"} Successful GETs (200) are not recorded by default. To include these calls, change the value of the audit.logging.FileHandler.level property in /etc/restjavad.log.conf to FINE . The log messages are handy for gathering the iControl REST statistics: How often do the calls fail? Which endpoint is used most extensively? Such intelligence is helpful when you need to investigate performance issues or intermittent failure. First, use grep to extract only the audit records as the file may contain other messages (such as boot markers). Then, apply cut to trim the first timestamp part. Now you have a sequence of JSON texts. fgrep '"user":' restjavad-audit.0.log | cut -d ' ' -f6- {"user":"local/admin","method":"GET","uri":"http://localhost:8100/mgmt/tm/sys/tmm-stat","status":400,"from":"192.168.184.10"} {"user":"local/admin","method":"GET","uri":"http://localhost:8100/mgmt/tm/sys/disk/stat","status":400,"from":"192.168.184.10"} {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/tm/util/bash","status":200,"from":"192.168.184.10"} ... With the help of sort and uniq, you can generate a statistical report on the methods and status: fgrep '"user":' restjavad-audit.0.log | cut -d ' ' -f6- | \ jq -r '[.method, (.status|tostring)] | join(" ")' | \ sort | uniq -c 3 DELETE 200 2160 GET 400 20 GET 401 14 GET 404 1 GET 500 1 GET 503 21 PATCH 200 12 PATCH 400 5 PATCH 403 147 POST 200 2 POST 202 46 POST 400 14 POST 401 4 POST 403 4 POST 404 2 PUT 202 7 PUT 400 The first part of the filter extracts the method and status properties. The status value is a number, hence it must be converted to string using the tostring function (needs the parentheses to group the two operations). Note that the join function only accepts an array with string elements. Similarly, you can obtain an endpoint-based summary. In the following example, long URLs are summarized into a component level (e.g., endpoints with actual virtual server object names are reduced to just /mgmt/tm/ltm/virutal ) by picking only the first four path components (mgmt, tm, ltm and virtual). fgrep '"user":' restjavad-audit.0.log | cut -d ' ' -f6- | \ jq -r '.uri | split("/")[3:7] | join("/")' | \ sort | uniq -c 1 mgmt/cm/autodeploy/software-image-downloads 64 mgmt/shared/authn/login 2 mgmt/shared/authz/roles 14 mgmt/shared/authz/tokens 9 mgmt/shared/authz/users ... The first part of the filter extracts the .uri property. The values are paths in string. The split function in the next stage splits the string by / character and the following slice ( [3:7] ) extracts only the first four path elements. Why starting from 3? Because the URIs start from https://localhost:8010/... and the first meaningful path element (mgmt) starts from the 3rd (counting from 0). Process restjavad-api-usage.log Another source of iControl REST statistics is /var/log/restjavad-api-usage.log . The file keeps the access counts per endpoint and per method in JSON format since the initial deployment of the system. The restjavad-api-usage feature was introduced in BIG-IP v13.1. The property apiUsageStartTimeMicrosUtc tells you when it started to tally the accesses. The value is a Unix epoch in microseconds. cat restjavad-api-usage.json | \ jq -r '.apiUsageStartTimeMicrosUtc | ./(1000*1000) | todate' 2019-08-25T22:05:46Z The usage property keeps the access counts in an array. Each array element is an object that contains the usage per endpoint. Access counts are categorized by the access sources: external : Accesses from outside (via management interface). internal : Accesses from inside the box (via localhost) messages : iControl REST internal message passing. The above access source properties keep the counts for each access method in an array: i.e., GET, POST, PUT, DELETE, PATCH and OPTIONS. Unlike the audit log, it counts the accesses irrespective of success or failure. The example below shows the 1000th element, which, in this case, contains the access counts for the /mgmt/cm endpoint. cat restjavad-api-usage.json | jq -r '.usage[1000]' # 1000th element { "path": "/mgmt/cm", # The endpoint "external": [ # Access from outside 1, # GET 0, # POST 0, # PUT 0, # DELETE 0, # PATCH 0 # OPTIONS ], "internal": [ # Access from inside (localhost) 0, 0, 0, 0, 0, 0 ], "message": [ # Interthread communications 0, 0, 0, 0, 0, 0 ] } The example below extracts the number of GET requests against /mgmt/cm from external clients: cat restjavad-api-usage.json | jq -r '.usage[1000].external[0]' 1 jq can do more for you. The example below extracts the sum of the external access counts (for all the methods) for the endpoints that start with /mgmt/tm/sys . cat restjavad-api-usage.json | \ jq -r '.usage[] | select(.path | startswith("/mgmt/tm/sys")) | [.path, (.external | add | tostring)] | join("\t")' /mgmt/tm/sys 4 /mgmt/tm/sys/clock 1 /mgmt/tm/sys/cpu 1 /mgmt/tm/sys/crypto 1 /mgmt/tm/sys/crypto/cert-order-manager 2 /mgmt/tm/sys/crypto/csr 1 /mgmt/tm/sys/crypto/key 6 /mgmt/tm/sys/db 2930 /mgmt/tm/sys/disk 5 /mgmt/tm/sys/disk/application-volume 1 /mgmt/tm/sys/disk/directory 9 /mgmt/tm/sys/file/external-monitor 7 /mgmt/tm/sys/global-settings 35 /mgmt/tm/sys/ha-group 2 /mgmt/tm/sys/hardware 4 /mgmt/tm/sys/host-info 1 /mgmt/tm/sys/provision 5 /mgmt/tm/sys/ready 16 /mgmt/tm/sys/software 4 /mgmt/tm/sys/software/image 1 /mgmt/tm/sys/tmm-info 1 /mgmt/tm/sys/ucs 4 /mgmt/tm/sys/version 94 Let's explore this filter. The select function extracts the usage object with the endpoints that start with /mgmt/tm/sys . .path | startswith() in the select argument feeds the .path value to the startswith function, which returns true when the input starts with the string in the function argument. The .external part of (.external | add | tostring) filter extracts the array with 6 numbers. The add function adds them together, then the last tostring does the number to string conversion (because join requires string elements). Get a list of the latest Automation Toolchain F5 Automation Toolchain is a collection of remote deployment services (SeeF5 Automation Toolchain). The software, which you need to install on BIG-IP to utilize the service(s), is updated with enhancements and bug fixes fairly often, hence you may want to check Github from time to time to stay up to date (The Github pages areAS3,DOandTSrespectively). You can automate the checking by accessing the following URI. AS3 (Application Services 3): https://api.github.com/repos/F5Networks/f5-appsvcs-extension/releases/latest DO (Declarative Onboarding): https://api.github.com/repos/F5Networks/f5-declarative-onboarding/releases/latest TS (Telemetry Streaming): https://api.github.com/repos/F5Networks/f5-telemetry-streaming/releases/latest A GET request to the URL returns a JSON formatted text that contains information about the latest software. Here is an example from AS3. curl -sk https://api.github.com/repos/F5Networks/f5-appsvcs-extension/releases/latest | \ jq -r '.' { ... "node_id": "MDc6UmVsZWFzZTM0MjA3MjM4", "tag_name": "v3.24.0", "target_commitish": "master", "name": "v3.24.0", "draft": false, "prerelease": false, "created_at": "2020-11-20T02:03:49Z", "published_at": "2020-11-20T02:12:39Z", ... } The top-level properties show the generic information about the software package. The version number is shown in the name property and the date of publication is found in the published_at property. Here's a script to grab these information for all the services: for service in f5-appsvcs-extension f5-declarative-onboarding f5-telemetry-streaming; do echo -n "$service ..." curl -sk https://api.github.com/repos/F5Networks/$service/releases/latest | \ jq -r '[.name, .published_at] | join(" ")' done f5-appsvcs-extension ...v3.24.0 2020-11-20T02:12:39Z f5-declarative-onboarding ...v1.17.0 2020-11-20T01:45:51Z f5-telemetry-streaming ...v1.16.0 2020-11-20T01:22:22Z Conclusion iControl REST was introduced in BIG-IP version 11.4 and it is now a must-know feature for automated configurations and maintenance. It represents the information in JSON, hence jq is a natural choice for casual data processing. If you did not find the method you need in this series, please take a look at thejq manual. You should be able to find the functions you are after. Having said so, jq is not as flexible as a general-purpose programming language. If you have complex requirements, you should consider using iControl REST SDKs such asF5 Python SDK.iControl REST + jq Cookbook - Readme
This series describes convenient methods for extracting meaningful information from lengthy and often verbose iControl REST responses using a third party tool called jq. To extract specific data from an iControl REST JSON text response, such as a virtual server name or pool member IP address, most often, you would try good old Unix utilities such as grep, sed or awk. You would pipe multiple commands if a single command does not satisfy you. If you have ever craved for a neater way, consider using jq - A tool specifically designed for parsing and filtering JSON texts. jq is bundled with BIG-IP. If your client system does not have one, you can install it via a package management tool such as yum or apt. Alternatively, you can download the binary from Download jq. You can copy it to any directory as it does not have any dependency. Although jq for Windows is available, Windows command prompt is picky about quotes and escapes. The jq filter examples in the documents are for bash, hence you may need to tweak them to satisfy peculiar Window's needs. Consider enabling Windows Subsystem for Linux (WSL) to use bash on Windows. If you are unfamiliar with JSON, please take a look at RFC 8259: "The JavaScript Object Notation (JSON) Data Interchange Format". Don't worry. It's just 16 pages long. Table of Contents Part01 - Basics Format the entire JSON body Sort by property names Extract a specific property Error Substitute the authentication token to a shell variable Extract multiple properties Format the output Count a number of objects Iterate through an array Obtain the full pool information Part02 - Intermediate [To be published] Extract an array element Slice an array Find the element Find matching elements Extract property names and values Extract a property from deeply nested object Tabularize all the version information Process raw monitor output Format 'list sys global-settings' output Convert Unix epoch time Part03 - Advanced topics [To be published] Replicate an object with slight modifications Modify internal data-group records Process restjavad-autit.log Process restjavad-api-usage.log Get a list of the latest Automation Toolchain Examples The examples in this series use curl for sending iCotnrol REST requests. The response JSON body is fed into the jq's stdin via pipe as below. curl -sku admin:admin https://<host>/mgmt/tm/sys/version | jq '.' The -s curl option is for disabling its progress bar (--silent). The -k option is for skipping server certificate verification (--insecure). The -u is to provide the user name and password (--user): The examples use the user 'admin' with the password 'admin'. Please change them. The <host> in the URL is the management IP of the target BIG-IP. The example output is shown directly below the call: curl -sku $PASS https://$HOST/mgmt/tm/sys/version | jq '.' {# output starts from here "kind": "tm:sys:version:versionstats", "selfLink": "https://localhost/mgmt/tm/sys/version?ver=15.1.2", "entries": { "https://localhost/mgmt/tm/sys/version ...# Skipped References jq 1.6 Manual. The current latest version is 1.6. The one bundled in BIG-IP is 1.5. Download jq. curl.1 the man page. iControl REST Home, F5 Clouddocs. iControl REST Cookbook - Virtual Server (ltm virtual), Devcentral. iControl REST Cookbook - LTM policy (ltm policy), DevCentral. iControl REST Cookbook - Virtual Server Profile (LTM Virtual Profiles), DevCentral. K13225405: Common iControl REST API command examples, AskF5. The JavaScript Object Notation (JSON) Data Interchange Format, RFC 8259, IETF. Windows Subsystem for Linux, Microsoft.