Forum Discussion

Roman_80473's avatar
Roman_80473
Icon for Nimbostratus rankNimbostratus
Oct 17, 2011

How to detect if pool member is down?

Hi folks,

 

 

I need to detect if pool member is down (crashed, hung, out of memory), take it out of the mix, write details to the log (node ip, name, port), and resend traffic back to the original pool before health monitor detects a failure.

 

So, inside my iRule, I have two events to capture server failure:

 

1. HTTP_RESPONSE checks if server's alive, but not responding to requests:

 

when HTTP_RESPONSE {

 

if { [HTTP::status] >= 500 } {

 

log local0.debug "Node [LB::server addr]:[LB::server port] is not responding, mark down, resend traffic ..."

 

LB::down

 

resend cached request

 

HTTP::retry $LB_request

 

}

 

}

 

 

 

2. LB_FAILED checks if server's completely dead:

 

when LB_FAILED {

 

log local0.debug "node [LB::server addr]:[LB::server port] is dead, mark down, resend traffic ..."

 

LB::down

 

resend cached request

 

HTTP::retry $LB_request

 

}

 

 

 

When I shutdown one of the members, the logic seems to be somewhat working (no blank page in the browser), but I see this in the logs:

 

01220001:3: TCL error: My_LB_Rule - Error: HTTP::retry may not be used in client-side event LB_FAILED (line 1) invoked from within "HTTP::retry $LB_request"

 

 

Is there a better way to accomplish this?

 

 

Any help is greatly appreciated

 

Roman
application delivery
dev
devops
iRules
  • Michael_Yates's avatar
    Michael_Yates
    Icon for Nimbostratus rankNimbostratus
    Oct 17, 2011
    Hi Roman,

     

     

    The LB_FAILED Event should occur prior to the HTTP_RESPONSE Event, so the HTTP::retry in your response event should suffice for what you are trying to do.

     

     

    See this Tech Tip: http://devcentral.f5.com/Tutorials/TechTips/tabid/63/articleType/ArticleView/articleId/344/iRules-Event-Order.aspx

     

     

    Hope this helps.
  • Roman_80473's avatar
    Roman_80473
    Icon for Nimbostratus rankNimbostratus
    Oct 17, 2011
    Hi Michael,

     

     

    If node is shutdown, http_response is never executed. Same goes for suspended/out of memory situation (server responds with 500 errors) when the lb_failed is not executed. I think I need both events present in iRule. I just need to figure out how to resend traffic back in each event.

     

     

    Thanks Roman
  • Michael_Yates's avatar
    Michael_Yates
    Icon for Nimbostratus rankNimbostratus
    Oct 17, 2011
    Hi Roman,

    No way to execute the HTTP::retry in the LB_FAILED Event. There are ways around it though.

    You can set the HTTP::request in a variable in the HTTP_REQUEST Event and if the LB_FAILED Event occurs you can perform an LB::down, persist none, and LB::reselect to restart the process instead of dying. Setting a retry flag to know if you need to resend the previous request could then be processed in the HTTP_RESPONSE.

    Something like this: 

     
when HTTP_REQUEST {
log local0. "HTTP Request Event."
set httprequest [HTTP::request]
set retry 0
}
when LB_SELECTED {
log local0. "LB Selected Event [LB::server addr] : [LB::server port]"
}
when LB_FAILED {
log local0. "LB Failed Event [LB::server addr] : [LB::server port]"
LB::down
persist none
LB::reselect
set retry 1
}
when HTTP_RESPONSE {
log local0. "HTTP Response Event."
if { $retry == 1 } {
set retry 0
HTTP::retry $httprequest
}
}

    Hope this helps.
  • Roman_80473's avatar
    Roman_80473
    Icon for Nimbostratus rankNimbostratus
    Oct 19, 2011
    Hi Michael,

     

     

    I think I got it now. Thank you very much for your help

     

     

    Roman

     

  • Gopal_Prakriya_'s avatar
    Gopal_Prakriya_
    Icon for Nimbostratus rankNimbostratus
    Dec 28, 2012
    I would like to page myself by sending a text message on LB_FAILED, appreciate any help on how to do that.
  • What_Lies_Bene1's avatar
    What_Lies_Bene1
    Icon for Cirrostratus rankCirrostratus
    Dec 28, 2012
    Gopal, please start a new topic rather than posting in unrelated existing (and unresolved) ones.