Forum Discussion
Craig_Jackson_2
Nimbostratus
NimbostratusSpurious CRLF before HTTP verb
We're running into a problem with a spurious \r\n being sent to the server immediately preceding a HTTP verb. This is in a OneConnect environment.
The problem occurs because IE (6 and 7 at least) sometimes puts a \r\n after transmitting the body of an HTTP POST. (I haven't figured out exactly when this happens, but it seems to be more common when going through a proxy.) This is evidently a hoary problem with IE -- it's mentioned in section 4.1 of the HTTP 1.1 RFC. Logically, in an HTTP 1.1 connection-reuse environment, this \r\n gets prefixed onto the HTTP verb of the next transaction. The HTTP spec says that there are some browsers that do this, and that this illegal behavior but servers should accept it.
We're using OneConnect so that the F5 Bigip will look at each transaction in the reused connection and properly perform cookie persistence on each one. When we were running 9.1.1 HF3, seeing such a spurious \r\n would cause the Bigip to terminate the connection prematurely, resulting in the clients seeing broken pages. At the suggestion of F5 Support, we upgraded to 9.3, and the F5 no longer delivers broken pages.
The nature of the change which we saw when we upgraded to 9.3 is that the \r\n is now tacked onto the front of the next transaction, which is how it appears in logical sequence. Unfortunately, we've got an old application which is still running on a 1999-era web server (Netscape Enterprise Server 3.6) It can't handle the spurious CRLF in front of its verb -- it gives a 500 error.
F5 Support says that the new (9.3) behavior is proper, but this can be handled with an iRule. Of course, F5 Support doesn't support the most valuable aspect of the F5 product line, iRules, so they sent me to devcentral.
So, how can I detect this spurious \r\n in an iRule, and how do I remove it? Will it be in front of the verb in an HTTP_REQUEST event, or will I need to work at the TCP level? Do I need to use some sort of a server-side event? I'm a bit at a loss.
Note that I've written a number of iRules and I've been programming TCL on and off for 10 years. I'm just having trouble figuring out how to detect this condition.
Craig Jackson
14 Replies
Craig_Jackson_2Ok, I'm somewhat more confused now. How can the LTM control whether the browser pipelines or doesn't?
What exactly are the semantics of the pipelining switch? Does it change a response header? Does it cause the connection to be closed if it is turned off? (If so, what distinguishes it from connection reuse between the browser and the LTM?)
I guess I need to spend some more time staring at the HTTP RFC.
Craig- Craig_Jackson_2Just checked the HTTP 1/1 RFC. It suggests that it's something that the client MAY do. It appears that the only way for the the server to stop it is to close the connection after delivering the response to the first request. (The client is suppose to be prepared to retry.)
So I suppose that the "pipelining" switch could cause the LTM to close the connection if a request was received before a previous response had been delivered. However, it would seem that such a switch would allow connection reuse if the requests were not received prematurely. (I.e. if a strict request/response discipline was maintained over the reused connection.) Is this correct? If so, how would that help my connection-reuse problem? - OneConnect is a keepalive mechanism that operates between the LTM and the servers. It is intended to minimize backend TCP handshake overhead by re-using serverside connections for multiple requests. The client side of the connection is not managed by OneConnect.
Disabling pipelining @ LTM will have the effect of closing the connection on the client side once the response is sent, hence bl0ndie's suggestion: If the connection is closed, any data associated with it cannot be prepended to the next request (because it would be on a new connection.)
HTH
/deb - Good to know that this worked out for you. Just to set things straight, turning off the pipelining should not have any performance for connections that don't pipeline the requests (they will get connection pooled / oneconnected just like all the requests). For the ones that do happen to pipeline, we will honor the first pipelined request but will close the connection when the response is sent. As you know from the RFC clients are supposed to re-try the requests (only non idempotent requests should be pipelined) therefore it should not break the application. It does however add the over head of the new connection especially if the virtual is shared with other applications that do proper pipelining.
The reason why I suggested this in the first place is because the irule necessary to do achieve this get a lot trickier than what you have. You can't solve this from HTTP because we deliberately prevent you from modifying data that's not part of the current request. Since the extra CRLF's 'appear' as data for the next request, the HTTP rules can’t touch it.
This does not mean that you can't do this though. You could hook into the TCP events and parse it before the HTTP layer has had a chance to process the data. I did not want to cross that bridge until you had a chance to try disabling pipeling. The only down side to this approach is that parsing data at the TCP layer will affect all traffic where as disabling the pipeling affects just the pipelined connections.
With that said, we are looking at fixing this in subsequest release. If you would like to track this fix or need it sooner go ahead and feel free to open a ticket with support and tell them bl0ndie sent you. In the mean time here is a similar example that demonstrates parsing at the TCP layer.
Click here
