Forum Discussion
JamesS_40157
Nimbostratus
Nimbostratus404 re-load balance in pool
Hi all,
Firstly apologies if this has already been covered - I've tried searching the forums and google but can't find exactly what I'm looking for. I'm also very new to irules so please forgive my ignorance here!
We have a pool of servers that in theory should have the same content on each but in practice they do not - that is to say at any one time 1 or 2 servers could potentially serve a 404 on a piece of content.
What I'm after is a rule to say that if the load balancer receives a 404 error, to transparently re-balance to the next node, and then re-balance again to the next node until it has tried all servers in the pool. At this point I'd like it to serve a 404 to the client if it cannot find the content anywhere. To me this seems like a trivial thing a load balancer should do but I can't see to find info on it anywhere!
I've had a go at writing the rule however it does not work perfectly (using LB:reselect) as this relies I belive on the load balancer mechanism which is currently set to round-robin. Therefore it will not be foolproof to ensuring it tries each and every node in the pool for a single client request.
This is the code I have ( mainly copied from an older post on this forum)
when CLIENT_ACCEPTED {
set retries 0
}
when HTTP_REQUEST {
set request [HTTP::request]
}
when HTTP_RESPONSE {
if { [HTTP::status] eq "404"} {
if { $retries < [active_members mediapool] } {
incr retries
HTTP::retry $request
} else {
HTTP::respond 404
}
}
}
This does not guarantee to try each member of the pool per client request. I'm sure I'm close but can't see the final step...could someone please help me out? Many thanks!
5 Replies
hoolioCirrostratus
Hi James,
That's a nice rule to start with...
You could try to track each server that's been requested and failed, but that's going to get expensive in terms of memory and CPU to process the used members for every request which generates a 404. If you wanted to try, you could use [active_members -list] (Click here) if you're running 9.4.2 or higher. You'd need to save the available members to a list and then move them to a tried/failed list.
Also, HTTP::request only returns the HTTP headers from a request. And it would be very expensive in terms of memory to try to collect every request payload to potentially use if there is a 404 response. So you might be better off checking the request method using HTTP::method and only saving the request headers for GET requests. You could then add a check in HTTP_RESPONSE for a 404 status to a GET request:when CLIENT_ACCEPTED { set retries 0 } when HTTP_REQUEST { if {[HTTP::method] eq "GET"}{ Save the request headers set request [HTTP::request] } else { Not going to retry this request, so unset the variable unset request } } when HTTP_RESPONSE { Check if the response is a 404 and it was a response to a GET request if { [HTTP::status] == 404 && [info exists request]} { Check if we've tried fewer times than number of pool members if { $retries < [active_members mediapool] } { Retry request incr retries HTTP::retry $request } else { Give up and send a 404 We could just do nothing and send back the application's response... HTTP::respond 404 } } }
Aaron- hoolioWhat about resetting the retries to 0 in an else clause if this is false: ' {[HTTP::status] == 404 && [info exists request]}'?
Aaron - hoolioCan you add logging to the iRule and post the log output when a failure occurs? Try to include the client IP:port in the log statements so you can track what requests were made over the same TCP connection.
I'm not sure if there is another approach that would work.
Aaron 
JamesS_40157Hi hoolio - yes I added logs however I know they were definitely different connections as I initiated my tests from 2 separate phsyical servers/IP's. If I started to request 2 different bits of content they would get 404's every now and then which we can't have. I'm going to start to look at a different way of doing this..if I get anywhere I'll post it as I'm sure this is a reasonable thing a load balancer should do!
Cheers- hoolioActually, if the servers don't all have the same content and the load balancing is done based on round robin it's possible that a request could be re-load balanced to servers that don't have the content. I think the only way to guarantee the request would be tried against all the servers until one is found that serves the content would be to explicitly check each server using [active_members -list].
Another option for improving this might be to use RAM caching. Assuming you didn't cache 404 responses, you could reduce the number of requests to the pool by answering the requests from cache on LTM.
Aaron
