Forum Discussion
Craig_Jackson_2
Nimbostratus
NimbostratusSpurious CRLF before HTTP verb
We're running into a problem with a spurious \r\n being sent to the server immediately preceding a HTTP verb. This is in a OneConnect environment.
The problem occurs because IE (6 and 7 at least) sometimes puts a \r\n after transmitting the body of an HTTP POST. (I haven't figured out exactly when this happens, but it seems to be more common when going through a proxy.) This is evidently a hoary problem with IE -- it's mentioned in section 4.1 of the HTTP 1.1 RFC. Logically, in an HTTP 1.1 connection-reuse environment, this \r\n gets prefixed onto the HTTP verb of the next transaction. The HTTP spec says that there are some browsers that do this, and that this illegal behavior but servers should accept it.
We're using OneConnect so that the F5 Bigip will look at each transaction in the reused connection and properly perform cookie persistence on each one. When we were running 9.1.1 HF3, seeing such a spurious \r\n would cause the Bigip to terminate the connection prematurely, resulting in the clients seeing broken pages. At the suggestion of F5 Support, we upgraded to 9.3, and the F5 no longer delivers broken pages.
The nature of the change which we saw when we upgraded to 9.3 is that the \r\n is now tacked onto the front of the next transaction, which is how it appears in logical sequence. Unfortunately, we've got an old application which is still running on a 1999-era web server (Netscape Enterprise Server 3.6) It can't handle the spurious CRLF in front of its verb -- it gives a 500 error.
F5 Support says that the new (9.3) behavior is proper, but this can be handled with an iRule. Of course, F5 Support doesn't support the most valuable aspect of the F5 product line, iRules, so they sent me to devcentral.
So, how can I detect this spurious \r\n in an iRule, and how do I remove it? Will it be in front of the verb in an HTTP_REQUEST event, or will I need to work at the TCP level? Do I need to use some sort of a server-side event? I'm a bit at a loss.
Note that I've written a number of iRules and I've been programming TCL on and off for 10 years. I'm just having trouble figuring out how to detect this condition.
Craig Jackson
- I think you might be able to get around this by disbling pipelining (this is found in the http profile). This will effectively close the connection instead of sending the other request with the leading \r\n. Its not a particularly desirable behavior but may be a reasonable workaround for you guys (is it?). If not then we could try some thing more trickier.
Craig_Jackson_2Yes, we could do that. If we did that, we wouldn't need OneConnect, either. It's certainly a viable workaround, but it seems drastic.
Craig
John_Gouk_10641OK, had a quick grock of iRules. Here's a sketch, with totally incorrect syntax (!):
If it's a POST
do HTTP::collect [HTTP::content-length] (there's examples of this - collects the declared length)
when HTTP-REQUEST-DATA is called,
do HTTP::release to send good request on its way
Start TCP::collect (2) to pick up any stray CRLF bytes
on TCP-
check if CRLF, if it is, throw it away (how do you do this? TCP::payload with some special nulling value?), else do Release and we're done...
Any comments?
Cheers
J- Craig_Jackson_2Seems like it might work, although I don't know how it would interact with OneConnect.
There's some handwaving in there, too. - John_Gouk_10641Right. Spotted an error in the iRule that meant it was not matching on the bad packets - fixed below.
However, although it now logs that it is replacing data, it doesn't seem to, in that I'm still getting the error, and traces of traffic to/from the server still show packets with CRLF as the first 2 chars, as left behind from a POST.
Anybody out there?!
Cheers
J
when HTTP_REQUEST {
if { [HTTP::method] eq "POST"} {
log local0. "HTTP_REQUEST triggered"
log local0. "host: [HTTP::host]"
if { [HTTP::header exists "Content-Length"] } {
set content_length [HTTP::header "Content-Length"]
} else {
set content_length 4294967295
}
log local0. "CRLF Rule:contentLength: $content_length"
if { $content_length > 0 } {
HTTP::collect $content_length
}
}
}
when HTTP_REQUEST_DATA {
log local0. "HTTP_REQUEST_DATA triggered"
HTTP::release
TCP::collect 2
}
when CLIENT_DATA {
binary scan [TCP::payload 2] H* hex
log local0. "CLIENT_DATA triggered: data $hex"
if { $hex eq "0d0a" } {
log local0. "replacing data"
TCP::payload replace 0 0 0
}
TCP::release
} - Craig_Jackson_2Thanks. It still seems more complicated than it should be. It seems like you could look at [HTTP::method] or something.
I just put in an "iRules on Demand" request for this -- we'll see what they say.
Craig - Did you guys ever try disabling 'pipelining' in the HTTP profile. That should get around having to do any rule modifications.
- I was going to recommend following bl0ndie's suggestion to disable pipelining, and I see that suggestion has been made again. I'd sure give it a shot, given the source of the recommendation.
(Piplelining and OneConnect are enabled separately. OneConnect is a backend KeepAlive mechanism. Piplelining and keepalives both represent efficiencies to the basic HTTP mechanisms, and are often confused, but are not the same thing.)
Failing that, I was going to recommend further that you might want to try collecting Content-Length + 2, then perform payload replacement with only the first Content-Length bytes. I think that way might be more likely to send the extra 2 bytes to the bit bucket. (Assumption there is that the 2nd collect operation may be holding the "extra data" in the collect buffer, then sending it along with the next request on the same keepalive connection.)
HTH
/deb - Craig_Jackson_2I actually haven't gotten around to trying to turn off "HTTP Pipelining", by which I believe F5 means "HTTP connection reuse". (Strictly speaking, "HTTP pipelining" is HTTP connection reuse where the client sends in several HTTP requests in quick succession without waiting for a response.)
I was trying to avoid this. I'd prefer to have the benefit of the connection reuse between the browser and the F5. But I'm going to test it.
Note that the only reason why I've got OneConnect turned on is to get the F5 to do the cookie persistence correctly. Otherwise, I really don't care whether reuse is done between the LTM and the server. - Actually, pipelining and keepalives are different optimizations.
OneConnect uses Keep-Alive connections, which is connection re-use -- the same TCP connection may be used to process multiple HTTP requests.
Pipelining, on the other hand, is what you describe: the client can send a new request on the same connection without waiting for the response from the previous request(s).
Pipelining requires a keepalive connection, but keepalives do not require pipelining.
OneConnect is enabled by applying a oneconnect profile to the virtual server.
Pipelining is enabled by default in the HTTP profile, and can be disabled by modifying the default profile and applying the new profile to the virtual server along with the oneconnect profile.
/deb
