Forum Discussion

Roman_80473's avatar
Roman_80473
Icon for Nimbostratus rankNimbostratus
Nov 17, 2011

Detect if node is half-dead with an iRule?

Hi folks,

 

 

I was tasked to monitor app servers in the pool with an iRule (LTM 10.2). I wrote a simple rule which does the following:

 

if I get into LB_FAILED, I take the node out and resend request

 

if I get into HTTP_RESPONSE, and http status >= 500, I take the node out and resend request

 

 

It only seems to be working when the nodes are either fine or completely dead. Otherwise, (server ran out of memory), request gets into LB_SELECTED, and sits there forever. I get "The connection to the server was reset while the page was loading" error in the browser after about a minute or two, but my iRule never kicks in.

 

 

Is there a way to detect that the node is "half-dead" with iRule? Or, there're some external configuration in the VIP, pool, etc?

 

 

Any help is greatly appreciated

 

Thanks, Roman
application delivery
dev
devops
iRules
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Nov 17, 2011
    Hi Roman,

     

     

    It sounds like the server is completing a three way handshake to accept the TCP connection, but not responding to the HTTP request. In that case, you could use the 'after' command to set a timer for how long to wait before taking some action. See the second example on the 'after' wki page for details:

     

     

    http://devcentral.f5.com/wiki/iRules.after.ashx

     

     

    Aaron
  • Roman_80473's avatar
    Roman_80473
    Icon for Nimbostratus rankNimbostratus
    Nov 17, 2011
    Hi Aaron,

     

     

    Where would I put the "after" command? The example talks about http_request, but since I never "technically" get there, do I need to put it in the next sequential event after LB_SELECTED, i.e. SERVER_CONNECTED or HTTP_REQUEST_SEND?

     

     

    Thank you, Roman
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Nov 17, 2011
    You set the timer in HTTP_REQUEST and then cancel it in HTTP_RESPONSE if a response is received within the timeout. You should be able to just modify the TCP::respond command to do whatever action you want (disable the pool member, select a new one, etc), but leave the events as they are.

     

     

    Aaron
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Nov 18, 2011
    pool foo accepts 3-ways handshake but not responding any http request. i tried to re-select pool foo2 instead but i got - Address in use tcl error message (and of course, bigip sent reset to both sides). 

    [root@ve1023:Active] config  b virtual bar list
virtual bar {
   snat automap
   pool foo
   destination 172.28.19.79:80
   ip protocol 6
   rules myrule
   profiles {
      http {}
      tcp {}
   }
}
[root@ve1023:Active] config  b pool foo list
pool foo {
   members 200.200.200.101:88 {}
}
[root@ve1023:Active] config  b pool foo2 list
pool foo2 {
   members 200.200.200.101:80 {}
}
[root@ve1023:Active] config  b rule myrule list
rule myrule {
   when RULE_INIT {
        set static::response_timeout  5
}

when HTTP_REQUEST {
        log local0. "Received request, beginning response monitor interval. [clock seconds]"
}

when LB_SELECTED {
        set monitor_id [\
                after $static::response_timeout {
                        LB::reselect pool foo2
                        log local0. "Timeout $static::response_timeout milliseconds elapsed without server response. [clock seconds]"
                }\
        ]
}

when HTTP_RESPONSE {
        log local0. "Received server response."
        if {[info exists monitor_id]} {
                log local0. "Canceling after script with id $monitor_id"
                after cancel $monitor_id
        }
}
}


 curl -i http://172.28.19.79
curl: (52) Empty reply from server


Nov 18 00:16:02 local/tmm info tmm[4766]: Rule myrule : Received request, beginning response monitor interval. 1321604162
Nov 18 00:16:02 local/tmm err tmm[4766]: 01220001:3: TCL error: myrule  - Address in use (line 1)     invoked from within "LB::reselect pool foo2"


00:16:02.240879 IP 172.28.19.253.34199 > 172.28.19.79.80: S 3035055119:3035055119(0) win 5840 
00:16:02.240917 IP 172.28.19.79.80 > 172.28.19.253.34199: S 2804861295:2804861295(0) ack 3035055120 win 4380 
00:16:02.244080 IP 172.28.19.253.34199 > 172.28.19.79.80: . ack 1 win 46 
00:16:02.244104 IP 172.28.19.253.34199 > 172.28.19.79.80: P 1:155(154) ack 1 win 46 
00:16:02.244203 IP 200.200.200.10.34199 > 200.200.200.101.88: S 3752179749:3752179749(0) win 4380 
00:16:02.244864 IP 200.200.200.101.88 > 200.200.200.10.34199: S 195117275:195117275(0) ack 3752179750 win 5792 
00:16:02.244875 IP 200.200.200.10.34199 > 200.200.200.101.88: . ack 1 win 4380 
00:16:02.244886 IP 200.200.200.10.34199 > 200.200.200.101.88: P 1:155(154) ack 1 win 4380 
00:16:02.245899 IP 200.200.200.101.88 > 200.200.200.10.34199: . ack 155 win 54 
00:16:02.249309 IP 200.200.200.10.34199 > 200.200.200.101.88: R 155:155(0) ack 1 win 4380
00:16:02.249333 IP 172.28.19.79.80 > 172.28.19.253.34199: R 1:1(0) ack 155 win 4534
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Nov 18, 2011
    if i put LB::detach before LB::reselect, tcl was not error but after sending fin, bigip did not establish connection to pool foo2. i think because bigip already established connection to pool foo. 

    [root@ve1023:Active] config  b rule myrule list
rule myrule {
   when RULE_INIT {
        set static::response_timeout  5
}

when HTTP_REQUEST {
        log local0. "Received request, beginning response monitor interval. [clock seconds]"
}

when LB_SELECTED {
        set monitor_id [\
                after $static::response_timeout {
                        LB::detach
                        LB::reselect pool foo2
                        log local0. "Timeout $static::response_timeout milliseconds elapsed without server response. [clock seconds]"
                }\
        ]
}

when HTTP_RESPONSE {
        log local0. "Received server response."
        if {[info exists monitor_id]} {
                log local0. "Canceling after script with id $monitor_id"
                after cancel $monitor_id
        }
}
}

 curl -i http://172.28.19.79
...no response...

Nov 18 00:27:41 local/tmm info tmm[4766]: Rule myrule : Received request, beginning response monitor interval. 1321604861
Nov 18 00:27:41 local/tmm info tmm[4766]: Rule myrule : Timeout 5 milliseconds elapsed without server response. 1321604861

00:27:41.513841 IP 172.28.19.253.47126 > 172.28.19.79.80: S 2431649126:2431649126(0) win 5840 
00:27:41.513884 IP 172.28.19.79.80 > 172.28.19.253.47126: S 3818007430:3818007430(0) ack 2431649127 win 4380 
00:27:41.517039 IP 172.28.19.253.47126 > 172.28.19.79.80: . ack 1 win 46 
00:27:41.517091 IP 172.28.19.253.47126 > 172.28.19.79.80: P 1:155(154) ack 1 win 46 
00:27:41.517268 IP 200.200.200.10.47126 > 200.200.200.101.88: S 1242901181:1242901181(0) win 4380 
00:27:41.517934 IP 200.200.200.101.88 > 200.200.200.10.47126: S 3205360631:3205360631(0) ack 1242901182 win 5792 
00:27:41.517946 IP 200.200.200.10.47126 > 200.200.200.101.88: . ack 1 win 4380 
00:27:41.517957 IP 200.200.200.10.47126 > 200.200.200.101.88: P 1:155(154) ack 1 win 4380 
00:27:41.518888 IP 200.200.200.101.88 > 200.200.200.10.47126: . ack 155 win 54 
00:27:41.522163 IP 200.200.200.10.47126 > 200.200.200.101.88: F 155:155(0) ack 1 win 4380 
00:27:41.522884 IP 200.200.200.101.88 > 200.200.200.10.47126: F 1:1(0) ack 156 win 54 
00:27:41.522899 IP 200.200.200.10.47126 > 200.200.200.101.88: . ack 2 win 4380 
00:27:41.617423 IP 172.28.19.79.80 > 172.28.19.253.47126: . ack 155 win 4534
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Nov 18, 2011
    Maybe we could use HTTP::retry instead of LB::reselect?

     

     

    Aaron
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Nov 18, 2011
    HTTP::retry is not valid in LB_SELECTED event.

     

     

    Please feel free to let me know if you would like to try any other command.
  • Roman_80473's avatar
    Roman_80473
    Icon for Nimbostratus rankNimbostratus
    Nov 18, 2011
    Aaron and nitass,

     

     

    Thank you for your replies. I still think I don't quite get it. I set the monitor_id with "after" inside LB_SELECTED, then cancel it in both http_request and http_response. That works and I don't get stuck in lb_selected anymore. But, the same story now happens in http_request, where I get in, but never exit. I set the monitor at the end of http_request the same way, and I do trap the timeout when server is not responding. However, I also trap timeout even if nothing's wrong. Am I missing smth?

     

     

    Thanks, Roman
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Nov 18, 2011
    I set the monitor_id with "after" inside LB_SELECTED, then cancel it in both http_request and http_response.i do not think we should cancel monitor_id in HTTP_REQUEST event since the monitor_id will start scheduling in LB_SELECTED event.

     

     

    the situation Aaron and I am discussing is when bigip has established 3-ways handshake to pool member and that is it (the pool member does not respond to GET request). what we want to do is to re-select another pool or pool member, establish a new 3-ways handshake and send GET request over. however, it appears bigip does not pick up a new pool or pool member.
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Nov 18, 2011
    Nitass, can you try putting the after statement in HTTP_REQUEST and use (LB::detach + LB::reselect) or HTTP::retry? Else, I can try testing this later today.

     

     

    Aaron