iControl REST + jq Cookbook - Part 2: Intermediate

Part 2 deals with slightly complex use cases: Manipulating array elements, looking for strings that match a certain criteria and handling both property names (keys) and values to generate a tabularized output. iControl REST sometimes returns a bare output from the underlying tmsh (e.g., monitor stats). Toward the end, filters for such row strings are discussed. Additionally, methods for converting Unix epoch timestamps are presented.

If you find some jq notations difficult, please revisit the Part 1.

Extract an array element

A single array may contain a large number of elements. For example, the records property of an LTM internal data-group (e.g., tmsh list ltm data-group internal aol) stores as many entries as you want in a single array. Here's an example from the AOL internal data-group:

curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | jq '.records'
[
  {
    "name": "64.12.96.0/19",
    "data": ""
  },
  ...
  {
    "name": "207.200.112.0/21",
    "data": ""
  }
]

You can selectively extract a single element by specifying the array index number to the records property. The index number starts from 0, so [1] means the 2nd element.

curl -sku $PASS https://<host>/mgmt/tm/ltm/data-group/internal/aol | jq '.records[1]'
{                                                              # 0th record element
  "name": "195.93.16.0/20",
  "data": ""
}

You can specify multiple indexes separated by ,. In the example below, only the name properties are extracted by additional | .name.

curl -sku $PASS https://<host>/mgmt/tm/ltm/data-group/internal/aol | \
  jq -r '.records[2, 3, 5] | .name'
195.93.48.0/22                                                 # The name property value from 2nd
195.93.64.0/19                                                 # The name property value from 3rd
198.81.0.0/22                                                  # The name property value from 5th

Note that jq does not raise any error for an out-of-range index: It silently returns null (just like referencing a non-existing property).

curl -sku $PASS https://<host>/mgmt/tm/ltm/data-group/internal/aol | jq '.records[100]'
null

Slice an array

You can extract a subset of array by slicing: Just specify the start (inclusive) and stop (exclusive) indexes separated by colon (:) inside the []. For example, to extract the names (IP addresses) from the top 4 elements in the AOL data-group records, run the following command:

curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | \
  jq -r '.records[0:4] | .[].name'
64.12.96.0/19                                                  # The name property value from 0th
195.93.16.0/20
195.93.48.0/22
195.93.64.0/19                                                 # The name property value from 3rd

You might have noticed the subtle difference from the previous comma separated indexes: The additional [] is prepended to the .name filter. This is because the comma separated indexing (.records[2, 3, 5]) yields three independent objects (with the name and data properties), while the slice generates an array containing the record objects. Because it is an array, you need to iterate through it to process each record.

When the start index number is omitted, zero is used ([0:4] and [:4] are identical). When the end index number is omitted, it assumes the end of the array. The following example extracts the last 4 IP addresses from the AOL data-group (it currently contains 14 records) - from the 10th to the 13th.

curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | \
  jq -r '.records[10:] | .[].name'
205.188.146.144/30                                             # The name property value from 10th
205.188.192.0/20
205.188.208.0/23
207.200.112.0/21                                               # The name property value from last

Applying the slice to a string yields its substring. For example, you can extract the first 8 characters from "iControlREST" like this:

$ echo '"iControlREST"' | jq '.[:8]'                           # Piping "iControlREST" to jq
"iControl"

Find the element

Having said that, we seldom look for elements by their index numbers. We normally look for a pattern. For example, you want to find the data value of the "useragent6" in the sys_APM_MS_Office_OFBA_DG internal data-group.

curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/sys_APM_MS_Office_OFBA_DG | \
  jq -r '.records[] | [.name, .data] | join("\t")'
ie_sp_session_sharing_enabled   0
ie_sp_session_sharing_inactivity_timeout        60
ofba_auth_dialog_size   800x600
useragent1      microsoft data access internet publishing provider
useragent2      office protocol discovery
useragent3      microsoft office
useragent4      non-browser
useragent5      msoffice 12
useragent6      microsoft-webdav-miniredir                     # Looking for this one
useragent7      webdav-miniredir
useragent9      ms frontpage 1[23456789]
useragent10     onenote

You can extract only the specific element using the select function. Specify the condition for the selection in the function argument. Usual conditional statements that yields a Boolean can be used: For example, the == operator returns true if two operands are equal.

echo '["BIG-IP", "BIG-IQ"]' | jq '.[] | . == "BIG-IP"'
true                                                           # BIG-IP is BIG-IP
false                                                          # BIG-IQ is not BIG-IP

The select function outputs the input only when the condition in the argument is true.

echo '["BIG-IP", "BIG-IQ"]' | jq '.[] | select(. == "BIG-IP")'
"BIG-IP"

With these techniques, you can extract the record (object) that contains the "useragent6" in the name property:

curl -sku $PASS https://<host>/mgmt/tm/ltm/data-group/internal/sys_APM_MS_Office_OFBA_DG | \
  jq -r '.records[] | select(.name == "useragent6")'
{
  "name": "useragent6",
  "data": "microsoft-webdav-miniredir"
}

The following comparison operators can be used in jq: !=, >, >=, < and <=.

Find matching elements

The select function accepts anything as the conditional statement as long as it returns true or false. For example, the startswith and endswith functions. They compare the input against their argument and return a Boolean. Unlike ==, where you explicitly compare the incoming data (.) and the string, you only need to specify the string to compare. Let's find the IP addresses that start with "195" in the AOL internal data-group.

curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | \
  jq -r '.records[] | .name | select(startswith("195"))'
195.93.16.0/20
195.93.48.0/22
195.93.64.0/19
195.93.96.0/19

Here is an example for endswith, looking for the "/20" subnets:

curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | \
  jq -r '.records[].name | select(endswith("/20"))'
195.93.16.0/20
198.81.16.0/20
205.188.112.0/20
205.188.192.0/20

Regular expressions? Yes, jq comes with standard regular expression functions such as test and match. Here's an example for finding IP addresses that contain 96, dot and any digits:

curl -sku admin:admin https://<host>/mgmt/tm/ltm/data-group/internal/aol | \
  jq -r '.records[] | .name | select(test("96.\\d+"))'
64.12.96.0/19
195.93.96.0/19

The jq regular expression is Perl-compatible, however note that backslashes must be escaped (e.g., \\d+ instead of just \d+). For more information on the regular expression feature, refer to jq manual.

The RegEx functions are available only when the jq was compiled with the regular expression library Oniguruma. Without the library, you will receive the jq: error (at <stdin>:0): jq was compiled without ONIGURUMA regex libary. match/test/sub and related functions are not available error upon the use of any RegEx function. The jq on a BIG-IP does not come with the library.

Extract property names and values

Sometimes, you need to obtain both property names (keys) and their values, however, the .[] filter only yields the values. To get both, filter the incoming objects with the to_entries function (it acts like the Python's items()). For example, GET /mgmt/tm/ltm/monitor/http/http (tmsh list ltm monitor http http equivalent) returns the following object:

curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http | jq '.'
{
  "kind": "tm:ltm:monitor:http:httpstate",
  "name": "http",
  ...
}

The to_entries generates an array. Its elements are objects and each object consists of two properties: "key" and "value". The key property keeps the original property name while the value property stores the value of the property.

curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http | jq 'to_entries'
[
  {                                                            # Object from the 0th property
    "key": "kind",                                             # 0th property name
    "value": "tm:ltm:monitor:http:httpstate"                   # 0th property value
  },
  {
    "key": "name",
    "value": "http"
  },
  ...
]

By piping the result to .key, .value, you now have both property names and values.

curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http | \
  jq -r 'to_entries[] | .key, .value'
kind                                                           # 0th property name
tm:ltm:monitor:http:httpstate                                  # 0th property value
name                                                           # 1st property name
http                                                           # 1st property value
...

Obviously, it's much better to have the key and value in the same line. You can do so by using the array constructor [] and the join function as described in Part 1.

curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http | \
  jq -r 'to_entries[] | [.key, .value] | join(" ... ")'
kind ... tm:ltm:monitor:http:httpstate
name ... http
fullPath ... http
generation ... 0
selfLink ... https://localhost/mgmt/tm/ltm/monitor/http/http?ver=15.1.2
adaptive ... disabled
adaptiveDivergenceType ... relative
...

Extract a property from a deeply nested object

Some iControl REST responses are deeply nested. For example, the BIG-IP version number is found in the 6th level in a GET /mgmt/tm/sys/version response:

curl -sku admin:admin https://<host>/mgmt/tm/sys/version | jq '.'
{
  "kind": "tm:sys:version:versionstats",
  "selfLink": "https://localhost/mgmt/tm/sys/version?ver=15.1.1",
  "entries": {                                                 # top level
    "https://localhost/mgmt/tm/sys/version/0": {               # 2nd level
      "nestedStats": {                                         # 3rd level
        "entries": {                                           # 4th level
          "Build": {                                           # 5th level
            "description": "0.0.6"                             # 6th level
          },
          ...
          "Version": {
            "description": "15.1.1"                            # HERE!
          }
          ...

Obviously, you can reference the description property in the Version object by chaining the property names with dots from top to bottom (you need to double-quote the URL-like property as it contains sensitive characters).

curl -sku admin:admin https://<host>/mgmt/tm/sys/version | \
  jq '.entries."https://localhost/mgmt/tm/sys/version/0".nestedStats.entries.Version.description'
"15.1.1"

That's too tedious (and error prone). In this case, you can use [] as a catch-all property for the 2nd, 3rd and 4th levels. As you have seen, the iterator [] yields multiple elements in an array or object, however, because these levels only contain one property, it is as good as referencing specifically.

curl -sku admin:admin https://<host>/mgmt/tm/sys/version | \
  jq '.entries[][][].Version.description'
"15.1.1"

Tabularize all the version information

Producing a tabularized version information should be easy by now: Combination of the -r option and the to_entries and join functions that you have already seen.

curl -sku admin:admin https://<host>/mgmt/tm/sys/version | \
  jq -r '.entries[][][] | to_entries[] | [.key, .value.description] | join("\t")'
Build   0.0.6
Date    Thu Oct  8 02:52:59 PDT 2020
Edition Point Release 0
Product BIG-IP
Title   Main Package
Version 15.1.1

The .entries[][][] yields the 5th level objects: e.g., "Build": {"description": "0.0.6"}. The to_entries function then generates an object with the key and value properties: The key is "Build" and the "value" is {"description":"0.0.6"} in this case. You can access the data by referencing .key and .value.description.

Process raw monitor output

Some iControl REST calls return the raw output from tmsh. For example, GET /mgmt/tm/ltm/monitor/http/http/stats returns a blob of lines from tmsh show ltm monitor http http in the.apiRawValues.apianonymous property.

curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http/stats | \
  jq '.apiRawValues.apiAnonymous'
"---------------------------\nLTM::Monitor /Common/http \n------------------ ..."

The output is a single string with the line feeds expressed in the literal "\n" (backslash + n). You can simply output it using the --raw-output option.

curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http/stats | \
  jq -r '.apiRawValues.apiAnonymous'
---------------------------
 LTM::Monitor /Common/http
---------------------------
   Destination:  10.200.20.10:80
   State time:   up for 8hrs:9mins:13sec
   |   Last error:   N/A @2021.01.06 08:44:24
...

Hmmm ... This is too easy for the intermediate level. Let's achieve the same goal in a harder way: Replace the \n (2F 6E) with LF (0A) using the string replacement (substitution) function sub.

sub requires the regular expression library.

curl -sku admin:admin https://<host>/mgmt/tm/ltm/monitor/http/http/stats | \
  jq -r '.apiRawValues.apiAnonymous | sub("\\n"; "\n")'
---------------------------
 LTM::Monitor /Common/http
---------------------------
   Destination:  10.200.20.10:80
   State time:   down for 6hrs:3mins:51sec
   |   Last error:   No successful responses received before deadline. @2020.10.30 10:49:33
...

The first argument of the sub function is a from string and the second one is a to string. The argument separator is ; (semicolon). Note that you need to escape the literal backslash by another backslash (\\n). You can also use regular expressions.

Format 'list sys global-settings' output

Another response that needs massaging might be the one from tmsh list sys global-settings file-blacklist-path-prefix. As you can see below, it returns a single string, consisting of paths that are enclosed in {}.

tmsh list sys global-settings file-blacklist-path-prefix
sys global-settings {
    file-blacklist-path-prefix "{/shared/3dns/} {/shared/bin/} {/shared/core/} ...
}

Naturally, the equivalent iControl REST call returns a very long string. We can check how long it is by passing the string to the length function. It returns a number of characters when a string is fed (a number of elements or properties in an array or object):

curl -sku admin:admin https://$HOST/mgmt/tm/sys/global-settings | \
  jq '.fileBlacklistPathPrefix | length'
469                                                            # 469 characters long

You can split the string at each whitespace using the split function (this is not a regular expression function). The function generates an array consisting of substrings.

curl -sku admin:admin https://<host>/mgmt/tm/sys/global-settings | \
  jq -r '.fileBlacklistPathPrefix | split(" ")'
[
  "{/shared/3dns/}",
  "{/shared/bin/}",
  "{/shared/core/}",
  ...
]

If you think the curly braces are obnoxious, you can remove them by feeding it to the gsub regular expression function: a sibling of sub that performs substitution globally.

gsub requires the regular expression library.

curl -sku admin:admin https://<host>/mgmt/tm/sys/global-settings| \
  jq -r '.fileBlacklistPathPrefix | split(" ") | .[] | gsub("{|}"; "")'
/shared/3dns/
/shared/bin/
/shared/core/
...

The first argument for gsub is a from string. Here, a regular expression ("matches either { or }") is used. The second argument is a to string. Here, any { or } is globally replaced with an empty string. Again, note that the argument separator is ;. You can use the sub command for global replacement by specifying the g (global) flag in the third argument.

curl -sku admin:admin https://<host>/mgmt/tm/sys/global-settings | \
  jq -r '.fileBlacklistPathPrefix | split(" ") | .[] | sub("{|}"; ""; "g")'

Convert Unix epoch time

Some time related property values are expressed in Unix epoch in microseconds (10^-6). For example, the lastUpdateMicros and expirationMicros properties in an authentication token object represent the timestamps for last update and expiration in microseconds.

curl -sku admin:admin https://<host>/mgmt/shared/authz/tokens/LKMRTZ3TZY5PFHJI5HZNRCDUEM | \
  jq -r '.lastUpdateMicros, .expirationMicros'
1604038137666701                                               # Last Update date/time in μs
1604039337667000                                               # Expiration date/time in μs

You can convert the numbers to human readable format using the todate function.

curl -sku admin:admin https://<host>/mgmt/shared/authz/tokens/LKMRTZ3TZY5PFHJI5HZNRCDUEM | \
  jq -r '.lastUpdateMicros, .expirationMicros | ./(1000*1000) | todate'
2020-10-30T06:08:57Z
2020-10-30T06:28:57Z

The expression in the middle of the filter is an arithmetic computation: It divides the incoming time (., which is in microsecond) by 10⁶. This is required because the todate function expects the Unix epoch time in seconds.

The literal 1000*1000 looks ugly, but it is better than 1000000 (IMHO). If you prefer, you can use the pow function to compute 10 to the power of negative 6.

$ echo 1604039337667000 | jq '. * pow(10; -6) | todate'
"2020-10-30T06:28:57Z"

jq is equipped with standard C math functions (such as cos and log). Refer to the jq manual.

Continue to Part 3.