Node.js HTTP Message Body: Transfer-Encoding and Content-Length in LineRate lrs/virtualServer Module

Summary:

(refer to AskF5 solution article 15884)

In the LineRate scripting environment we consult the Content-Length header of HTTP messages entering the system (see the LineRate Developer Scripting Guide under the topic "Writing Past the Content-Length"). Therefore when performing transformations to an HTTP message body, be it a request or response, we must ensure that if a Content-Length header is present in the original message we send accurate information reflecting our changes in the message we transmit to the end host.

Full Story:

When I first began investigating what node.js was I stumbled upon a video presentation by Ryan Dahl (node's creator) from 2011.  If you fast-forward to about 14:00 in the video, Ryan is talking about the node HTTP module.  One of the features Ryan mentions is that the HTTP module's “server” object provides HTTP/1.1 responses.  Along with the Keep-Alive header, the node HTTP server automatically "chunks" the response; that is, the server will automatically apply the "Transfer-Encoding: chunked" header to a response.  Ryan explains that this enables node to automatically handle variable length responses. This also enables you to call the write() method multiple times on an active HTTP response object without prior knowledge of the total length of the response you want to send.  This is a great feature when you know all of your clients will be fully HTTP/1.1 compliant.

The LineRate application proxy’s scripting API provides for manipulating the HTTP data path, and when acting as part of a full-proxy architecture the data path requires the management of both HTTP server functionality as well as an HTTP client functionality.  This enables users of LineRate to make transformations to HTTP responses that originate from other servers ("real-servers" in LROS configuration terms).  Since LineRate stands between the origin servers and the HTTP clients we must take care to preserve application behavior, regardless of the version of HTTP the server provides.  Herein lies one of the differences in the design of the LineRate lrs/virtualServer module and the node HTTP module's server objects.

Bringing the previous paragraphs together: when you apply content transformations to HTTP message bodies using the lrs/virtualServer module, you must be conscious of the original message's framing method; that is, does the message use "Transfer-Encoding: chunked", or does it use a "Content-Length:" header value to delineate boundaries of HTTP message bodies?  The LineRate Scripting Developer Guide addresses this issue, and it is briefly outlined in a recent AskF5 solution article: Overview of Node.js auto-chunking behavior on LineRate systems.

In order to better understand this behavior I built a small node module that appends some chosen content to the end of an HTTP response:

This script makes use of the Node.js 'Modules' API, where it adds the object name “ResponseAppender” to the "exports" namespace.  In order to use this script as a module on your LineRate system simply copy this code as a file to the directory /home/linerate/data/scripting/, and then 'require' the module from within an in-line script where we call 'new' on the call to the exported function, passing three parameters: the virtual server we want to use the module with as the first parameter, an arbitrary string as the second parameter, and a boolean as the third parameter ('true' means we correct the value of the Content-Length to the final response if present, 'false' means we leave the headers exactly as the real-server provided them).  Here is an example of a partial LROS configuration with a virtual server and inline script that makes use of this module (assuming the above module file is named 'response_appender.js'):

With the above configuration objects (and assuming "test-vip" and "some-server" refer to appropriate corresponding configuration objects) we can easily toggle the "FIX_HEADER" value from the LineRate Manager web GUI, or by using the CLI from the configuration context with the command:

LROS(config-script:appender)# source edit vim
## or
LROS(config-script:appender)# source edit emacs

Once this scenario is configured we should be able to observe the following:

CASE #1: FIX_HEADER == false, and the real-server provides a Content-Length header

Here we would expect to see no change in a browser when we receive the response from the virtual server.  Because the Content-Length header is preserved form the original response the browser stops short of reading the appended content.  However, if we use a tool like tcpdump or wireshark to capture the network stream, we can actually see the appended string being sent over the wire.

CASE #2: FIX_HEADER == true, and the real-server provides a Content-Length header

This time because FIX-HEADER is true we actually calculate a new Content-Length value that includes the byte count of the appended string.  Now if we make a request to the test-vip with a browser we expect to the the pop-up text "SURPRISE!" because now the browser knows to read the additional bytes of the appended string.

CASE #3: the real-server provides the response with Transfer-Encoding: chunked or TE: chunked:

In this case nothing changes.  We can observe that LineRate behaves exactly like the node HTTP server objects, where any content written to the response stream is properly framed as an HTTP/1.1 chunk.  We expect the response to result in a browser displaying the "SURPRISE!" pop-up message.

Now that you know how it works, download your own copy of LROS, get a free tier license and try it yourself!