LTM: HTTP Monitoring with POST Request

I recently discovered that you can use LTM's built in HTTP monitor template to send a POST request to a server to verify its health. It's surprisingly straightforward in comparison with the external monitor approach I've used in the past. This article will cover the details you'll need to properly implement an HTTP POST monitor using the built in monitor template.

The built in HTTP monitor template

The HTTP monitor template provides a way to send a single request to the server, monitor the response for an expected string, and mark the pool member down if the expected response isn't seen within the configured timeout. The options of interest for this exercise are the Send string and the Receive string. 

The Send string is where the request is configured.  It should include the method (in this case, POST), the URI to request from the server, the HTTP version, some standard HTTP headers (including Basic Auth, if applicable), and in this case, the POST body.

The server should respond in a predictable way to this request each time it is issued.  The Receive string should include a phrase or non-dictionary word that will appear only in that expected response. The Receive string can use basic regular expressions to accommodate per-server or other minor variations in the expected response.

Capturing the the transaction

The first thing you'll need to do, as with any monitor, is to identify and capture the transaction you'd like to replicate in order to validate the health of your application servers.  You can run tcpdump to watch live production traffic and capture the target request and response, but it's safer to run an isolated test request at the command line to more exactly replicate the action of the LTM monitor.

The approach I recommend uses both techniques.  First capture a trace of production traffic so you have a known-good working example with all the expected headers.  Then log into the LTM command line and use telnet or cURL to send the same request to the server along with the observed headers, and monitor the response, adjusting the request until the expected response is received.

Here's the cURL syntax you'll need to use for command line request submission:

 curl -N -H <headers> -u <user:password> -d <post body> http://<pool_mmeber_IP>:<pool_mmeber_port><URI>

You'll need to examine the actual data that's being passed, including whitespace characters.  A packet trace can be used to look directly at the conversation on the wire. Capture on the server-facing interface.  To examine only production traffic, filter for the IP/port of the pool member, excluding the non-floating self IP on the server-facing VLAN:

 tcpdump -nni <server_vlan> -Xs 0 host <pool_member_IP> and port <pool_member_port> and not host <internal_non-floating_selfIP>

(Filtering out the non-floating self IP prevents monitor traffic from appearing in the trace.)

To examine your command line request, filter instead for the non-floating self IP on the server-facing VLAN and the IP/port of the pool member:

 tcpdump -nni <server_vlan> -Xs 0 host <internal_non-floating_selfIP> and host <pool_member_IP> and port <pool_member_port> 

Here's a sample transaction trace for a SOAP/xml request:

Once you have identified the transaction and validated the required headers, you are ready to build your custom monitor.

Creating the custom HTTP monitor

The information you've gathered so far should be everything you need to configure the custom HTTP monitor. Start by creating a monitor of type "http" in the LTM GUI (Local Traffic -> Monitors -> Create).  Configure Interval and Timeout as appropriate for your application. Set Username and Password if Basic Auth credentials are required by the server. Then you'll need to dig into your transaction trace data to construct the Send string containing all the request details.

For the Send string, the HTTP method must be specified along with the URI and HTTP version v1.1 on the same line, followed by an encoded LF/CR (\r\n), then the Host header (required for HTTP 1.1).  For the trace above, we'll start with:

 POST /XAIApp/xaiserver HTTP/1.1\r\nHost: 191.168.1.1:4200

Add the other headers, all separated by a single encoded CR/LF.  (Line breaks are only for display. No literal carriage returns are inserted into the Send string.)  Include the Basic Auth header, if applicable, even though it is specified elsewhere in the monitor template.  With headers, the Send string now looks like this:

POST /XAIApp/xaiserver HTTP/1.1\r\nHost: 191.168.1.1:4200\r\n Authorization: Basic c3lzdXNlcjpzeXN1c2VyMDA= \r\n
Pragma: no-cache\r\nAccept: */*\r\nContent-Length: 673\r\nContent-Type: application/x-www-form-urlencoded

Finally add the POST body, preceded by a double CR/LF to indicate the end of the headers. If the POST body contains double quotes or braces, you will need to escape them with a preceding backslash so they are interpreted correctly by LTM. The completed Send string now looks like this:

POST /XAIApp/xaiserver HTTP/1.1\r\nHost: 191.168.1.1:4200\r\n Authorization: Basic c3lzdXNlcjpzeXN1c2VyMDA= \r\n
Pragma: no-cache\r\nAccept: */*\r\nContent-Length: 673\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n
<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"urn:schemas-xmlsoap-org:envelope\">
<SOAP-ENV:Body><SSvcInstallationOptionsMaintenance dateTimeTagFormat=\"CdxDateTime\"  transactionType =\"READ\" ><SSvcInstallationOptionsMaintenanceService>
<SSvcInstallationOptionsMaintenanceHeader InstallationOptionID=\"1\"/><SSvcInstallationOptionsMaintenanceDetails InstallationOptionID=\"1\">
<InstallMsg><InstallMsgHeader InstallationOptionID=\"1\"/></InstallMsg><InstallAlg><InstallAlgHeader InstallationOptionID=\"1\"/></InstallAlg>
<Module></Module></SSvcInstallationOptionsMaintenanceDetails></SSvcInstallationOptionsMaintenanceService></SSvcInstallationOptionsMaintenance>
</SOAP-ENV:Body></SOAP-ENV:Envelope>

To complete the monitor definition, enter the phrase or string expected in a healthy server response into the Receive string field. For the trace above, we'll use:

 Standard automatic payment creation

The resulting monitor configuration will look like this:

monitor http-post-soapenv {
   defaults from http
   interval 10
   timeout 31
   password "password"
   recv "Standard automatic payment creation"
   send "POST /XAIApp/xaiserver HTTP/1.1\r\nHost: 191.168.1.1:4200\r\nAuthorization: Basic c3lzdXNlcjpzeXN1c2VyMDA= \r\n
Pragma: no-cache\r\nAccept: */*\r\n Content-Length: 673\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n
<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"urn:schemas-xmlsoap-org:envelope\">
<SOAP-ENV:Body><SSvcInstallationOptionsMaintenance dateTimeTagFormat=\"CdxDateTime\"  transactionType =\"READ\" ><SSvcInstallationOptionsMaintenanceService>
<SSvcInstallationOptionsMaintenanceHeader InstallationOptionID=\"1\"/><SSvcInstallationOptionsMaintenanceDetails InstallationOptionID=\"1\">
<InstallMsg><InstallMsgHeader InstallationOptionID=\"1\"/></InstallMsg><InstallAlg><InstallAlgHeader InstallationOptionID=\"1\"/></InstallAlg>
<Module></Module></SSvcInstallationOptionsMaintenanceDetails></SSvcInstallationOptionsMaintenanceService></SSvcInstallationOptionsMaintenance>
</SOAP-ENV:Body></SOAP-ENV:Envelope>"
   username "user"
}

Related information

These 2 previous articles are specific to external monitors, but contain some more useful details about command line testing and capturing traffic:
   LTM External Monitors: The Basics
   LTM External Monitors: Troubleshooting