Forum Discussion
Thomas_Knutson_
Nimbostratus
NimbostratusRandom HTTP 400 Errors
We recently implemented two 3600 model devices with LTM and ASM running on them. We are primarily running tomcat applications on our back end servers on TCP 8080, and terminating the SSL requests on the F5 device. We have the standard "stream" profile assigned to our virtual server, and an iRule to convert http:// refernces to https:// references. We have an ASM profile assigned, but it is set to transparent mode.
When users visit our site, they get random 400 errors after clicking on a link, or clicking the back button in the browser. Issuing a page reload, or copying the link and pasting it into a new tab on the browser enables the page to load. When a 400 status code is recieved, the request never makes it to the backend pool server. We have changed the "chunking" method in the HTTP profile, and changed how we are handling the stream. Looking for some input on potential causes of the problem, as we are very new to these devices. We can provide more detailed information if needed.
hooleylistCirrostratus
Hi Thomas,
Are you using an iRule to selectively enable the stream profile rewriting? If not, that's the first thing I'd try. You can use an iRule like this:From: http://devcentral.f5.com/wiki/default.aspx/iRules/stream__expression Example which replaces http:// with https:// in response content Prevents server compression in responses when HTTP_REQUEST { Disable the stream filter for all requests STREAM::disable } when HTTP_RESPONSE { Check if response type is text if {[HTTP::header value Content-Type] contains "text"}{ Replace http:// with https:// STREAM::expression {@http://@https://@} Enable the stream filter for this response only STREAM::enable } }
Aaron
Thomas_Knutson_Posted By hoolio on 05/13/2011 01:07 PM Hi Thomas,
Are you using an iRule to selectively enable the stream profile rewriting? If not, that's the first thing I'd try. You can use an iRule like this:From: http://devcentral.f5.com/wiki/default.aspx/iRules/stream__expression Example which replaces http:// with https:// in response content Prevents server compression in responses when HTTP_REQUEST { Disable the stream filter for all requests STREAM::disable } when HTTP_RESPONSE { Check if response type is text if {[HTTP::header value Content-Type] contains "text"}{ Replace http:// with https:// STREAM::expression {@http://@https://@} Enable the stream filter for this response only STREAM::enable } }
Aaron
We are using that exact iRule, and getting the same issue. We were using a generic "stream" profile instead of an iRule, and changed that on our test VIP today, and still encountered the same problems.We do have two "trunks" set up, and they have a very high amount of drops on incoming requests. Were not sure if that is related or not, but our switch ports don't have any errors on them, so it appears to just be on the F5 side.
- hooleylistCan you reproduce the issue? If so, I'd try capturing a tcpdump on LTM and use Fiddler2 to log the clientside HTTP. You can also add a debug event to the end of the iRule to log when a replacement is done:
From: http://devcentral.f5.com/wiki/default.aspx/iRules/stream__expression Example which replaces http:// with https:// in response content Prevents server compression in responses when HTTP_REQUEST { Disable the stream filter for all requests STREAM::disable set log_line "[IP::client_addr]:[TCP::client_port]: [HTTP::method] request to [HTTP::host][HTTP::uri]" log local0. $log_line } when HTTP_RESPONSE { Check if response type is text if {[HTTP::header value Content-Type] contains "text"}{ Replace http:// with https:// STREAM::expression {@http://@https://@} Enable the stream filter for this response only STREAM::enable } } when STREAM_MATCHED { log local0. "$log_line, matched [STREAM::match]" }
Aaron - Thomas_Knutson_Posted By hoolio on 05/13/2011 01:19 PM
Can you reproduce the issue? If so, I'd try capturing a tcpdump on LTM and use Fiddler2 to log the clientside HTTP. You can also add a debug event to the end of the iRule to log when a replacement is done:
Aaron
We are able to reproduce the issue on a consistent basis. Just using fiddler2 to log the clientside HTTP, we see some strange behavior. The errors are usually related to a .pdf file, that opens in the browser, and contains links. What I see in fiddler2 on good and bad connections is as follows.BAD:
1.) Visit page and generate report (pdf) that loads in a frame on the page. Fiddler2 reports the connection to the domain as a 200, and all looks good.
2.) Click on a link in the report (this is a pdf) that should load the detailed pdf, inside of a frame on the page, but a 400 is shown where that section of the frame is. Fiddler reports no client side requests or traffic during this time.
3.) Right click on the section of the frame where content should be, and select "refresh" and all content loads correctly. Fiddler2 shows an initial connect to the site (HTTP 200) and then the actual request which is a 302.
GOOD:
1.) Visit page and generate report (pdf) that loads in a frame on the page. Fiddler2 reports the connection to the domain as a 200, and all looks good.
2.) Click on a link in the report (this is a pdf) that then loads a detailed pdf inside of a frame on the page and all looks good. Fiddler reports the 302 redirect.
We have not done a TCP dump yet, and are looking into all possible culprits of the problem. We recently transitioned these services from an older netcontinum device, which did not cause the problem. As of this time we have a fairly generic setup, with minimal iRules, and ASM is in transparent mode. We do have an iRule that handles these specific reports, and sends them to a different set of back end server pools. The iRule for this is listed below (some content is edited out however)
when HTTP_REQUEST {
if { [HTTP::uri] contains "/specific-uri-that-conflicts-with-uri-below" } {
return
} elseif { [HTTP::uri] contains "/report-uri/" } {
pool report-pool_reports_web
} elseif { [HTTP::uri] contains "/other-report-uri/" } {
pool other-report-pool_reports_web
snat automap
} elseif { [HTTP::uri] contains "/path-we-block/" } {
drop
}
}
- Thomas_Knutson_We believe that we have resolve the issue at this time. I will outline the cause/resolution below in case anyone else has the same problem.
Cause:
On all of our virtual servers, and pool members we are using cookie persistance. While trying to diagnose the problem, we started to look into "oneconnect" profiles, and found a note in the article here http://support.f5.com/kb/en-us/solu...l7208.html
Resolution:
We created a new oneconnect profile, and assigned in to the virtual servers that were effected by this. We used the following link as our reference http://support.f5.com/kb/en-us/solu...l7964.html
After applying a oneconnect profile to our virtual servers, the random 400 errors seem to no longer occur. If I understand the article correctly, this would only occur if you are using "cookie" or "universal" persistance methods.
- hooleylistNice work in figuring that out. In general, you should use OneConnect whenever you're modifying the load balancing or persistence on a per-HTTP request basis. In addition to SOL7964, you can get more info on this wiki page:
http://devcentral.f5.com/wiki/default.aspx/AdvDesignConfig/oneconnect.html
Aaron