Convert curl command to BIG-IP Monitor Send String

[Eric_Chen]

Problem this snippet solves:

Convert curl commands into a HTTP or HTTPS monitor Send String.

How to use this snippet:

Save this code into a Python script and run as a replacement for curl.

Related Article: How to create custom HTTP monitors with Postman, curl, and Python

An example would be:

% python curl_to_send_string.py -X POST -H "Host: api.example.com" -H "User-Agent: Custom BIG-IP Monitor" -H "Accept-Encoding: identity" -H "Connection: Close" -H "Content-Type: application/json" -d '{
  "hello":
      "world"
}
' "http://10.1.10.135:8080/post?show_env=1"
SEND STRING:
POST /post?show_env=1 HTTP/1.1\r\nHost: api.example.com\r\nUser-Agent: Custom BIG-IP Monitor\r\nAccept-Encoding: identity\r\nConnection: Close\r\nContent-Type: application/json\r\n\r\n{\n  \"hello\":\n      \"world\"\n}\n

Code :

Python 2.x

import getopt
import sys
import urllib

optlist, args = getopt.getopt(sys.argv[1:], 'X:H:d:')
flat_optlist = dict(optlist)

method = flat_optlist.get('-X','GET')

(host,uri) = urllib.splithost(urllib.splittype(args[0])[1])
protocol = 'HTTP/1.1'

headers = ["%s %s %s" %(method, uri, protocol)]
headers.extend([h[1] for h in optlist if h[0] == '-H'])
if not filter(lambda x: 'host:' in x.lower(),headers):
    headers.insert(1,'Host: %s' %(host))


send_string =  "\\r\\n".join(headers)
send_string += "\\r\\n\\r\\n"

if '-d' in flat_optlist:
    send_string += flat_optlist['-d'].replace('\n','\\n')
send_string = send_string.replace("\"", "\\\"")
print "SEND STRING:"
print send_string

Python 3.x

import getopt
import sys

from urllib.parse import splittype, urlparse

optlist, args = getopt.getopt(sys.argv[1:], 'X:H:d:')
flat_optlist = dict(optlist)

method = flat_optlist.get('-X','GET')

parts = urlparse(splittype(args[0])[1])
(host,uri) = (parts.hostname,parts.path)
protocol = 'HTTP/1.1'

headers = ["%s %s %s" %(method, uri, protocol)]
headers.extend([h[1] for h in optlist if h[0] == '-H'])
if not filter(lambda x: 'host:' in x.lower(),headers):
    headers.insert(1,'Host: %s' %(host))


send_string =  "\\r\\n".join(headers)
send_string += "\\r\\n\\r\\n"

if '-d' in flat_optlist:
    send_string += flat_optlist['-d'].replace('\n','\\n')
send_string = send_string.replace("\"", "\\\"")
print("SEND STRING:")
print(send_string)

Tested this on version:

11.5

[Danny_Arroyo]

Eric,

I just tried your script on v11.4.1 and it worked perfectly. Thanks!!!

[Buddy_Edwards_1]

I'd like to say first that this is a great piece of code that has saved me a lot of headaches so thank you for creating it. The original script runs great in Python2 but had need for it to run natively in Python3 as the interpreter. This is a modified version that works in Python3.

    Code

import getopt
import sys
from urllib.parse import splithost, splittype

optlist, args = getopt.getopt(sys.argv[1:], 'X:H:d:')
flat_optlist = dict(optlist)

method = flat_optlist.get('-X', 'GET')

(host, uri) = splithost(splittype(args[0])[1])
protocol = 'HTTP/1.1'

headers = ["%s %s %s" % (method, uri, protocol)]
headers.extend([h[1] for h in optlist if h[0] == '-H'])
if not fil`text`ter(lambda x: 'host:' in x.lower(), headers):
    headers.insert(1, 'Host: %s' % (host))

send_string = "\\r\\n".join(headers)
send_string += "\\r\\n\\r\\n"

if '-d' in flat_optlist:
    send_string += flat_optlist['-d'].replace('\n', '\\n')
send_string = send_string.replace("\"", "\\\"")
print("SEND STRING: " + send_string)

[Eric_Chen]

Awesome! Thank you for sharing.

[scyonix_360392]

Hi I tried this but health monitoring is showing down, Logs says its a malformed request

this is my curl command which works as expected:

curl -X POST -H 'content-type: application/json' -H 'host: host.domain.com' -d '{"queryParam":"cleaning","start":"0","rows":"20","searchType":["ALL"],"facetSelection":["SC:catalogs"],"userRoles":[""]}'

here is the equivalent send string created using the script which is not working:

POST /globalSearch/basicSearch HTTP/1.1\r\ncontent-type: application/json\r\nhost: host.domain.com\r\n\r\n{\“queryParam\“:\“cleaning\“,\“start\“:\“0\“,\“rows\“:\“20\“,\“searchType\“:[\“ALL\“],\“facetSelection\“:[\“SC:catalogs\“],\“userRoles\“:[\“\”]}

Can anyone find out what the issue is?

[Eric_Chen]

I wonder whether your web server is expecting a content-length. Try

POST /globalSearch/basicSearch HTTP/1.1\r\ncontent-type: application/json\r\nhost: host.domain.com\r\nContent-Length: 120\r\n\r\n{\“queryParam\“:\“cleaning\“,\“start\“:\“0\“,\“rows\“:\“20\“,\“searchType\“:[\“ALL\“],\“facetSelection\“:[\“SC:catalogs\“],\“userRoles\“:[\“\”]}

[scyonix_360392]

no luck, does the braces need to be escaped?

[Eric_Chen]

In your curl example above was it http or https? Similar question about whether your monitor was configured to match the same protocol.

[scyonix_360392]

Curl is http and the protocol monitor is also http

[Eric_Chen]

Weird. It looks fine. tcpdump could verify whether any differences. Sending to the same port?

[Eric_Chen]

Also. What is the response size?

[scyonix_360392]

response size is 39Kb.

This is the actual request from the logs and i am getting HTTP 400 but curl works just fine:

send=POST /globalSearch/basicSearch HTTP/1.1\x0d\x0acontent-type: application/json\x0d\x0ahost: host.domain.com\x0d\x0aContent-Length: 120\x0d\x0a\x0d\x0a{\x0a"queryParam":"cleaning",\x0a"start":"0",\x0a "rows":"20",\x0a"searchType":[\x0a"ALL"\x0a],\x0a"facetSelection":[\x0a"SC:catalogs"\x0a ],\x0a"userRoles":[""\x0a ]\x0a}\x0a

[Eric_Chen]

I think you’re hitting this limit (5kb response): https://support.f5.com/csp/article/K3451

You could try adding a range header or modify your query to get less data back.

[scyonix_360392]

My response string appears in the 1st line of the response which is within 5,120 bytes so i doubt this is the problem

[Eric_Chen]

What version are you running, here's what I get from my monitor log. Running TMOS 13.1.0.5

[0][6132] 2018-05-04 01:07:41.595594: ID 46 :(_send_active_service_ping): writing [ tmm?=false td=true tr=false addr=::ffff:192.168.1.187:80 srcaddr=::ffff:192.168.1.250%0:56436 ] send=POST /test.json HTTP/1.1\x0d\x0acontent-type: application/json\x0d\x0ahost: host.domain.com\x0d\x0a\x0d\x0a{\\xe2\x80\x9cqueryParam\\xe2\x80\x9c:\\xe2\x80\x9ccleaning\\xe2\x80\x9c,\\xe2\x80\x9cstart\\xe2\x80\x9c:\\xe2\x80\x9c0\\xe2\x80\x9c,\\xe2\x80\x9crows\\xe2\x80\x9c:\\xe2\x80\x9c20\\xe2\x80\x9c,\\xe2\x80\x9csearchType\\xe2\x80\x9c:[\\xe2\x80\x9cALL\\xe2\x80\x9c],\\xe2\x80\x9cfacetSelection\\xe2\x80\x9c:[\\xe2\x80\x9cSC:catalogs\\xe2\x80\x9c],\\xe2\x80\x9cuserRoles\\xe2\x80\x9c:[\\xe2\x80\x9c\\xe2\x80\x9d]}

Are you using the GUI to create the monitor or using iControl REST?

[Accenture_Lorea]

Hello,

In this example, you are trying to send the monitoring traffic in http but in port tcp/8080 right ? But where are you mentioning that. I want to perform the same type of GET request but in a different port. Where should I mention that for a monitor.

[Eric_Chen]

By default a monitor will use the IP:PORT that is configured for the pool member. If you need to target a specific different port than what the service is using than you can specify a "Alias Service Port" as part of the monitor.

[Accenture_Lorea]

Yes Eric, I just found out that couple of minutes back. But thanks for the information..