iControl REST + jq Cookbook - Part 1: Basics
In this first part, simple filters and basic command options are explained: Specifically, filters for extracting particular properties, counting a number of objects, and tabularizing the results. All the examples are from actual iControl REST calls: You may find some of them immediately applicable to your requirements.
A filter is an instruction that describes how subsets of incoming JSON texts are parsed, extracted, modified and formatted. jq takes the filter string as a sole argument (other than the command options starting with - and input files). Since it is a single string, you need to single-quote it to protect from bash interpreting the filter contents. You can omit the single quotes for filters with no special characters, but it is highly recommended to use '' all the time.
Let's get started.
Format the entire JSON body
An iControl REST GET request returns a single line JSON body without any line breaks or indentations, which is unfriendly for ordinary human eyes. To pretty-print the entire JSON body, pipe the response body to the filter "dot" (.). The following example is for formatting a response from the tmsh list ltm virtual node CentOS-internal20 equivalent call.
curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq '.'.
{
"kind": "tm:ltm:node:nodestate",
"name": "CentOS-internal20",
"fullPath": "CentOS-internal20",
... # Skip
"fqdn": { # A nested object
"addressFamily": "ipv4",
"autopopulate": "disabled",
"downInterval": 5,
"interval": "3600"
},
... # Skip
"state": "up"
}
The . denotes the top level object. jq searches for the starting point of the top level object from the incoming text, and recursively prints all the components under it with indentations.
Sort by property names
To sort by the top-level property names (keys), add the --sort-keys option (the shortcut is -S).
curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq -S '.'
{
"address": "10.200.20.10",
"connectionLimit": 0,
"dynamicRatio": 1,
...
}
Extract a specific property
You can reference any property directly under the top-level by appending the property name to the top-level dot. For example, to extract the value of fullPath property from the above call, run the command below:
curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq '.fullPath' "CentOS-internal20"
The fqdn property value is an object (nested object), hence the .fqdn filter yields the entire object including its surrounding curly braces.
curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq '.fqdn'
{
"addressFamily": "ipv4",
"autopopulate": "disabled",
"downInterval": 5,
"interval": "3600"
}
To reference a property inside the object, append another dot, then the property name. For example, for the downInterval property, run the command below:
curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq '.fqdn.downInterval' 5
Note that a string value is double-quoted ("") while a numeric value is not (see RFC 8259). To remove the surrounding double quotes from string values, use the --raw-output option (-r). For example, to extract the value of addressFamily property inside fqdn, run the command below:
curl -sku admin:admin https://localhost/mgmt/tm/ltm/node/CentOS-internal20 | jq -r '.fqdn.addressFamily' ipv4
Error
Any input that is not properly JSON formatted causes error. In the following example, an incorrect password (xxx) is specified in the curl call. Because the BIG-IP returns an HTML response (instead of JSON), jq fails.
# > /dev/null is added to show only the stderr from jq curl -sku admin:xxx https://<host>/mgmt/tm/sys/version | jq '.' > /dev/null parse error: Invalid numeric literal at line 1, column 10
A logical error is not reported. For example, the filter.fqdn.addressfamily is syntactically valid, but there is no such property in the fqdn property (must be Family). When a property that does not present is referenced, jq yields null.
curl -sku admin:admin https://<host>/mgmt/tm/ltm/node/CentOS-internal20 | jq '.fqdn.addressfamily' null
Substitute the authentication token to a shell variable
The --raw-output option is handy when you need to directly use the value in a bash script. The example below is for extracting the authentication token from an iControl REST call POST /mgmt/shared/authn/login and substituting it to a shell variable TOKEN for later use.
# Get the token through the '.token.token' filter and subsititute to the shell variable TOKEN
TOKEN=`curl -sk https://<host>/mgmt/shared/authn/login -X POST -H "Content-Type: application/json" \
-d '{"username":"<username>", "password":"<password>", "loginProviderName":"tmos"}' | \
jq -r '.token.token'`
# Check the token. Should not be enclosed in ""
echo $TOKEN
CL4SBLFG4GEOLQUFX4FTHSREDM
# Run an iControl REST call using the token (e.g., the 'tmsh show sys version' equivalent)
curl -sk https://<host>/mgmt/tm/sys/version -H "X-F5-Auth-Token: $TOKEN" | jq .
{
"kind": "tm:sys:version:versionstats",
"selfLink": "https://localhost/mgmt/tm/sys/version?ver=15.1.1",
...
}
The iControl REST call returns a valid JSON body even the authentication fails with 401 as below:
{
"code": 401,
"message": "Authentication failed.",
"originalRequestBody": "{\"username\":\"satoshi\",\"loginProviderName\":\"tmos\",\"generation\":0,\"lastUpdateMicros\":0}",
"referer": "192.168.184.1",
"restOperationId": 6611604,
"kind": ":resterrorresponse"
}
Since the response does not have .token.token property, the jq filter returns null. The subsequent call would fail because of the wrong credential. You can use the --exit-status option (-e) to detect such logical error. Normally, jq returns the exit code 0 even if the property is not present, however, with -e, it returns 1.
curl -sk https://<host>/mgmt/shared/authn/login -X POST -H "Content-Type: application/json" \
-d '{"username":"<username>", "password":"<password>", "loginProviderName":"tmos"}' |
jq -r '.token.token'
null
echo $? # The exit code indicates "Success".
0
curl -sk https://$HOST/mgmt/shared/authn/login -X POST -H "Content-Type: application/json" \
-d '{"username":"<username>", "password":"<password>", "loginProviderName":"tmos"}' |
jq -er '.token.token'
null
$ echo $? # The exit code indicates 1 (non-zero)
1
Use $? in your script to exist when the token request fails.
Using a slightly advanced technique, you can raise an error if .token.token is null using the if-then-else statement along with the error function (see the Conditionals and Comparisons section of the jq manual):
... | jq -r '.token.token | if . == null then error("no token") else . end'
Extract multiple properties
To extract multiple properties, concatenate them with comma (,). For example, to extract the virtual server's name (.name), destination address (.destination) and its pool (.pool) from a GET /mgmt/tm/ltm/virtual/vs response, run the command below:
curl -sku admin:admin https://<host>/mgmt/tm/ltm/virtual/vs | jq '.name, .destination, .pool' "vs" "/Common/172.16.10.50:80" "/Common/Pool-CentOS80"
The comma functions like Unix's tee: The stream of JSON text is individually fed into each designation, hence yielding three values. The leading . is necessary for each designation because you need to explicitly tell that it should reference from the top-level.
Format the output
You can concatenate three individual outputs to form a single line using the --join-output option (-j). The option also removed the double quotes around the strings, just like the --raw-output option.
curl -sku admin:admin https://localhost/mgmt/tm/ltm/virtual/vs | jq -r '.name, .destination, .pool' vs/Common/172.16.10.50:80/Common/Pool-CentOS80
Probably, this is not exactly what you want. You can insert separator characters in between, separated by the commas as if they were properties. These designations also receive the input but output the specified literals without doing anything to the input. Here is an example:
curl -sku admin:admin https://<host>/mgmt/tm/ltm/virtual/vs | \ jq -jr '.name, ."\t", destination, "\t", .pool, "\n"' vs /Common/172.16.10.50:80 /Common/Pool-CentOS80
Count a number of objects
A GET request to a component generally returns a list (array) of objects. For example, GET /mgmt/tm/ltm/virtual returns a list of virtual servers and their properties (equivalent to tmsh list ltm virtual). GET /mgmt/tm/ltm/pool returns a list of pools (tmsh list ltm pool). Here is an example from GET /mgmt/tm/ltm/pool:
curl -sku admin:admin https://<host>/$HOST/mgmt/tm/ltm/pool | jq '.'
{
"kind": "tm:ltm:pool:poolcollectionstate",
"selfLink": "https://localhost/mgmt/tm/ltm/pool?ver=15.1.1",
"items": [ # the array starts
{ # 1st pool object starts
"kind": "tm:ltm:pool:poolstate",
"name": "Pool-CentOS80",
...
}, # 1st pool object ends
{ # 2nd pool object starts
"kind": "tm:ltm:pool:poolstate",
"name": "Pool-CentOS443",
...
} # 2nd pool object ends
] # the array ends
}
Each configuration object is an element in the items array. You can extract the entire items array by specifying .items as described in the previous examples. You can feed the resulting array to another processing filter through the pipe (|) - Just like passing the intermediate result to the next command in an Unix command chain. For example, you can count the number of pools by passing the .items array to the jq length function.
curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | jq '.items | length' 3
As the result shows, this BIG-IP has three pool objects.
Iterate through an array
The .items[] filter yields individual object. The filter is fairly similar to the previous .items but has the additional []. This [] is an iterator, which loops around the array and yields the elements one by one. These elements are independent of each other, or not structured.
Let's try both .items and .items[] to see the difference.
curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | jq '.items'
[ # The value of .items is an array!
{
"kind": "tm:ltm:pool:poolstate",
"name": "Pool-CentO8080",
...
},
{
"kind": "tm:ltm:pool:poolstate",
"name": "Pool-CentOS80",
...
},
{
"kind": "tm:ltm:pool:poolstate",
"name": "Pool-CentOS443",
...
}
] # The array closing ]
$ curl -sku $PASS https://$HOST/mgmt/tm/ltm/pool | jq '.items[]'
{ # Three independent elements, hence no [].
"kind": "tm:ltm:pool:poolstate",
"name": "Pool-CentO8080",
...
} # No comma
{
"kind": "tm:ltm:pool:poolstate",
"name": "Pool-CentOS80",
...
Because of these differences, the length function counts the number of properties in each pool object, hence yields 26 three times (from the three pools):
curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | jq '.items[] | length' 26 # The length of Pool-CentO8080 26 # The length of Pool-CentOS80 26 # The length of Pool-CentOS443
Since .items[] yields three pool objects separately, you can now extract each individual property inside the object:
curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | jq -r '.items[].fullPath' /Common/Pool-CentO8080 /Common/Pool-CentOS80 /Common/Pool-CentOS443
If you also want to extract the monitor names for each object, you first iterate through the pool object (.items[]), then feed the output (each object) to the comma separated list of properties through the pipe.
curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | jq '.items[] | .fullPath, .monitor' /Common/Pool-CentO8080 /Common/gateway_icmp /Common/Pool-CentOS80 /Common/http /Common/Pool-CentOS443 /Common/https
Obtain the full pool information
Printing the pool name and monitor in one line would be nicer. You might have already tried the --join-output option, however, unfortunately, it prints all values in a single line.
curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | \ jq -j '.items[] | .fullPath, .monitor' /Common/Pool-CentO8080/Common/gateway_icmp/Common/Pool-CentOS80/Common/http/Common/Pool-CentOS443/Common/https
You should use the join function for concatenating the output per pool.
curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool | \
jq -r '.items[] | [.fullPath, .monitor] | join("\t")'
/Common/Pool-CentO8080 /Common/gateway_icmp
/Common/Pool-CentOS80 /Common/http
/Common/Pool-CentOS443 /Common/https
Please note the [] surrounding the two properties. This is for creating an array containing two elements (fullPath and monitor values). This pre-processing is required because the join function can concatenates only array elements.
Now, let's get the pool name, monitor and its members. The information on the associated pool members is stored in the membersReference property but it is normally just a link. Here's an example from GET /mgmt/tm/ltm/pool/Pool-CentOS80:
curl -sku admin:admin https://<host>/mgmt/tm/ltm/pool/Pool-CentOS80 | \
jq -r '.membersReference'
{
"link": "https://localhost/mgmt/tm/ltm/pool/~Common~Pool-CentOS80/members?ver=15.1.2",
"isSubcollection": true
}
To obtain the actual information such as pool names, add the ?expandSubcollections=true query option to the REST call:
curl -sku $PASS https://$HOST/mgmt/tm/ltm/pool/Pool-CentOS80?expandSubcollections=true | \
jq -r '.membersReference'
{
"link": "https://localhost/mgmt/tm/ltm/pool/~Common~Pool-CentOS80/members?ver=15.1.2",
"isSubcollection": true,
"items": [
{
"kind": "tm:ltm:pool:members:membersstate",
"name": "CentOS-internal20:80",
...
},
{
"kind": "tm:ltm:pool:members:membersstate",
"name": "CentOS-internal30:80",
...
}
]
}
Because there could be more than one pool members in a pool, the member objects are stored in an array (.items). You can reference the names by applying the .membersReferences.items[].name filter.
Now, combining all the above yields the filter for generating the pool information in a tabular format.
curl -sku $PASS https://$HOST/mgmt/tm/ltm/pool?expandSubcollections=true | \
jq -r '.items[] | [.fullPath, .monitor, .membersReference.items[].name] | join("\t")'
/Common/Pool-CentO8080 /Common/gateway_icmp CentOS-internal20:8080
/Common/Pool-CentOS80 /Common/http CentOS-internal20:80 CentOS-internal30:80
/Common/Pool-CentOS443 /Common/https CentOS-internal20:443 CentOS-internal30:443
Continue to Part 2
Published Jan 21, 2021
Version 1.0
No CommentsBe the first to comment