HTTP Payload Collection

Problem this snippet solves:

iRule demonstrating the basic approach for collection and manipulation of HTTP payload data.

Note: Data collection and manipulation can require a large amount of memory and CPU processing for each connection. LTM is not the optimal place to perform HTTP payload manipulation unless you can take advantage of the streaming replacement functionality.

For general payload examination, payload collection is limited by memory allocation constraints to a 32MB maximum (4Mb in pre-v11.x). Character set conversions and regex operations can quadruple the payload size in memory, so the functional maximum collection limit for payload manipulation is 8MB (1Mb in pre-v11.x). (See AskF5 SOL6578 for details).

Even though it is possible to collect up to store up to 32Mb of data in a variable, it is not recommended to do this without careful consideration. If each connection can use 32Mb of memory, TMM can quickly run out of memory.

Forcing the server to HTTP/v1.0 prevents chunking & ensures a Content-Length header is sent. The first example uses that value to set the collection size to the smaller of Content-Length or 1MB/4MB limit.

For payloads larger than 1MB, the entire payload may be processed by iteratively collecting and releasing max sized payload sections and decrementing a counter on each pass by # of bytes processed until the entire content length has been processed. Note that pattern matching may be compromised for patterns that straddle collection boundaries. The second example demonstrates this approach.

How to use this snippet:

This iRule requires LTM v10. or higher.

Code :

# The following example gives the framework to examine and manipulate the first 1MB of each response:

when HTTP_REQUEST {
  # Don't allow data to be chunked
  if { [HTTP::version] eq "1.1" } {
    if { [HTTP::header is_keepalive] } {
      HTTP::header replace "Connection" "Keep-Alive"
    }
    HTTP::version "1.0"
  }
}

# The following example includes collection counter / recollect logic to process payloads > 1MB in size. (Untested, may not be optimal -- please update with improvements.)

when HTTP_REQUEST {
  # Don't allow data to be chunked
  if { [HTTP::version] eq "1.1" } {
    if { [HTTP::header is_keepalive] } {
      HTTP::header replace "Connection" "Keep-Alive"
    }
    HTTP::version "1.0"
  }
  set collected 0
}

when HTTP_RESPONSE {
  # Get the content length so we can request the data to be
  # processed in the HTTP_RESPONSE_DATA event.
  if { [HTTP::header exists "Content-Length"] } {
    set content_length [HTTP::header "Content-Length"]
  } else {
    set content_length 0
  }
  # content_length of 0 indicates chunked data (of unknown size)
  if { $content_length > 0 && $content_length < 1048577 } {
    set collect_length $content_length
  } else {
    set collect_length 1048576
  }  
  log local0.info "Content Length: $content_length   Collect length: $collect_length"
  if { $collect_length > 0 } {
    HTTP::collect $collect_length
  }
}

when HTTP_RESPONSE_DATA {
  
  HTTP::release
  if { $content_length > 0 } {
    # for unchunked data, calculate remaining length & re-collect if necessary
    # The HTTP_RESPONSE_DATA event will be triggered again when each collection is complete
    set collected [expr {$collected + $collect_length}]
    set remaining [expr {$content_length - $collected}]
    if { $remaining > 0 } {
      if { $remaining < $collect_length } {
        set collect_length $remaining
      }
      HTTP::collect $collect_length
    }
  } else {
    # chunked responses data, continue collecting in 1MB chunks.  Watch for hanging responses here.
    # The HTTP_RESPONSE_DATA event will be triggered again when each collection is complete
    HTTP::collect $collect_length
  }
}


when HTTP_RESPONSE {
  # Get the content length so we can request the data to be
  # processed in the HTTP_RESPONSE_DATA event.
  if { [HTTP::header exists "Content-Length"] && [HTTP::header "Content-Length"] < 1048577 } {
    set content_length [HTTP::header "Content-Length"]
  } else {
    set content_length 1048576
  }
  log local0.info "Content Length: $content_length"
  if { $content_length > 0 } {
     HTTP::collect $content_length
  }
}

when HTTP_RESPONSE_DATA {
  
  HTTP::release
}

Tested this on version: