Forum Discussion
Wil_Schultz_101
Nimbostratus
Jul 30, 2007Reselect pool on status code 500 or above...
So we run some Tomcat servers, when we undeploy the server will send a 503 that states that the application is unavailable. I do have a check that looks for a static page every 5 seconds and fails at 16 seconds, however for that 16 second period people from the outside world see this 503 error.
I would like to set it up to where any status code above 500 reselects but I seem to have an order of operation problem. Maybe someone that i saner than I might have a suggestion?
Here is what I have so far:
when HTTP_RESPONSE {
if { [HTTP::status] > 499 }{
set ::hostfailed 1
log local0. "Setting up ::hostfailed."
}
}
when LB_SELECTED {
if { $::hostfailed == 1 } {
LB::detach
LB::reselect [LB::server pool]
log local0. "LB failed, re-load balancing to pool."
}
}
This irule fails because LB_SELECTED us called after HTTP_RESPONSE and ::hostfailed is not set.
22 Replies
- Colin_Walker_12Historic F5 AccountYou can get to the ltm logs via the GUI as well under System -> Logs.
Colin - JJ_47859
Nimbostratus
This was a good thread. I'm testing wschultz iRule and it works well except that we are using least connections for load balancing and every so often the iRule will redirect to the maintenance page because it may every http request to the same server that was throwing http 500 error messages. Can I use an array or some other technique to ensure that the LB::reselect will choose a different pool member?
Thanks - Deb_Allen_18Historic F5 AccountI think the easiest approach is to add an LB_SELECTED event in which you change the load balancing method to round robin & re-select in the same pool for retries only:
HTHwhen LB_SELECTED { if { $myretry >= 1 } { LB::mode rr LB::reselect pool $mypool } }
/deb - aalford_564
Nimbostratus
Instead of redirecting to an external URL, is there a way to choose a new pool after it has been determined that all servers in the original pool are throwing 500+ errors? I would prefer not to waste an external IP on a maintenance page, if possible. - hoolio
Cirrostratus
I think you could modify Deb's last snippet to select a new pool:when LB_SELECTED { if { $myretry >= 1 } { LB::reselect pool $my_new_pool } }
You can test this by making requests which only generate a 500 response and see if the new pool is used.
Note that there is a known issue with LB::resleect. Check the wiki page (Click here) for related solutions which describe the problem.
Aaron - sam_81897
Nimbostratus
So I am trying to do the following with the iRule which was submitted by Wil Schultz:
1. Client Sends HTTP Post to F5
2. F5 Proxies connection to Server1
3. Server1 is shutting down, replies with HTTP Status Code 5XX
4. F5 Catches 503, redirects request to another Server2 in pool
5. Server2 fulfills the request sends HTTP200 OK to F5
6. F5 Relays 200OK to client.
The main thing is that I do not want the client to see the 500 error. Is there additional logic that I need to implement to ensure the client does not see the 500 error? The iRule I am using is as follows:this counter is used to limit the number of retrieswhen CLIENT_ACCEPTED {set retries 0}Try a new node if we have retried the connection once, set the method to rrThis is just an idea right nowwhen LB_SELECTED {if { $retries >= 1 } {log "Reselecting from node=[LB::server addr] in pool=[LB::server pool]"set mypool [LB::server pool]LB::mode rrLB::reselect pool $mypool}}we need to capture the request data to send the request again after receiving a 500 errorwhen HTTP_REQUEST {set myhttprequest [HTTP::request]}This is the guts here, examine each HTTP response. If you get an HTTP 500 error do the following: Check to ensure we have not tried multiple times Detach from the current node Select a new node from the current pool when HTTP_RESPONSE {if { [HTTP::status] starts_with "5" } {log "$myhttprequest has generated a [HTTP::status]"if { ($retries < [active_members [LB::server pool]] ) } {incr retriesHTTP::retry $myhttprequest log "500 error received. HTTP request $myhttprequest has been retried to [LB::server addr]."}}} Mark the node down and select a new node on LB failure.when LB_FAILED {log "Failure of node=[LB::server addr] in pool=[LB::server pool]"set mypool [LB::server pool]LB::downLB::mode rrLB::reselect pool $mypoool} - hoolio
Cirrostratus
That should work. The only change I can see you should make is to limit retries to GET requests. Else, you'd need to collect the HTTP payload for POSTs and append it to the request headers and then call that combined variable with HTTP::retry. That could eat up a lot of memory.
You can set a variable in HTTP_REQUEST if HTTP::method eq "GET" and then check that in your if statement looking for a 5xx status:when HTTP_REQUEST { if {[HTTP::method] eq "GET"}{ set request_headers [HTTP::request] set method "GET" } else { set method "" } } ... when HTTP_RESPONSE { if { [HTTP::status] starts_with "5" and $method eq "GET"} { ... }
Also, if the retries exceeds the number of pool members, you could send back a generic HTTP 200.
Aaron - sam_81897
Nimbostratus
We are working with the below rule now, but we are getting some errors, but I am having trouble finding information about the error:
Oct 5 10:41:21 local/tmm err tmm[2617]: 011f0007:3: http_process_state_early_100continue - Invalid action EV_TCL_CLOSE during ST_HTTP
_EARLY_100CONTINUE (Server side: vip=*vipname* profile=*profile* pool=*pool*when HTTP_REQUEST { set my_url [HTTP::host][HTTP::uri] log "Request for URL=$my_url received." if { [HTTP::cookie BIGIP] contains "Redirect" } { set retries [findstr [HTTP::cookie BIGIP] "Redirect_" 9 1] } else { set retries 0 } } This is the guts here, examine each HTTP response. If you get an HTTP 500 error do the following: Check to ensure we have not tried too many times force current node down Select a new node from the current pool when HTTP_RESPONSE { if { [HTTP::status] starts_with "5" && $retries < [active_members [LB::server pool]] } { log "$my_url has generated a [HTTP::status]. Sending 307 for response." incr retries LB::down HTTP::respond 307 Location "http://$my_url" Set-Cookie "BIGIP=Redirect_$retries" Connection "Close" } elseif { [HTTP::status] starts_with "5" && $retries >= [active_members [LB::server pool]] } { set retries 0 HTTP::respond 404 Set-Cookie "BIGIP=Not_Found" Connection "Close" } if { ($retries < [active_members [LB::server pool]] ) } { incr retries HTTP::retry $myhttprequest log "500 error received. HTTP request $myurl has been retried to [LB::server addr]." } } Mark the node down and select a new node on LB failure. when LB_FAILED { log "Failure of node=[LB::server addr] in pool=[LB::server pool]" set mypool [LB::server pool] LB::down LB::mode rr LB::reselect pool $mypool } - sam_81897
Nimbostratus
I am really having trouble preventing the status code being transmitted with my new irule my data stream looks like:POST /Services/TransReportIIS/HttpServiceHoster.aspx HTTP/1.1 Content-Type: text/xml Host: Content-Length: 927 Expect: 100-continue HTTP/1.0 307 Temporary Redirect Location: Set-Cookie: BIGIP=Redirect_1 Server: BigIP Connection: close Content-Length: 0 HTTP/1.1 503 Service Unavailable Content-Type: text/html; charset=us-ascii Server: Microsoft-HTTPAPI/2.0 Date: Wed, 05 Oct 2011 16:39:12 GMT Connection: close Content-Length: 326 - hoolio
Cirrostratus
For a non-GET request, can you check for a 5xx status in HTTP_RESPONSE and send a redirect or response from the iRule (using HTTP::redirect or HTTP::respond)? I don't think it's worthwhile to buffer every request payload in order to retry them for 5xx responses.
Aaron
Help guide the future of your DevCentral Community!
What tools do you use to collaborate? (1min - anonymous)Recent Discussions
Related Content
DevCentral Quicklinks
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
Discover DevCentral Connects