Forum Discussion
Alan_Sha
Nimbostratus
NimbostratusHow to keep the client connection when LB_FAILED?
Hi,
I need some help on the following scenario.
I have been maintaining a persistent TCP connection between a client application and Big IP. The client application is sending requests continously to Big IP while Big IP re-routes each request via iRule accordingly. Typically, the client will send one request and wait for the reply before it sends another one. However, in the event of the destination server being unreachable, I want to have Big IP drop the current package after a number of attempts to reconnect, then wait for a new one coming in and process it. I have tried to write some error handling codes in LB_FAILED event, but it appears that once I stopped calling LB::reselect in the LB_FAILED event, the client to Big IP connection will be immediately closed. That's something that I don't want to happen. I want to keep the connection alive all the time and the traffic keeps flowing.
Below is my codes for LB_FAILED event. What can I do after I stop calling LB::reselect to keep the connection and continue to process any subsequent request? Any suggestion?
when LB_FAILED {
set connRetry [expr {$connRetry + 1}]
if { $connRetry < $maxAttempts } {
log local0.error "Conn. to [IP::server_addr] failed. Retrying..."
LB::reselect
}
}
Thanks,
Alan
12 Replies
- You could either do an HTTP::respond or an HTTP::redirect.
Sounds, like you might want to do the HTTP::respond. You could use a 5xx status code to indicate there was a server error or you could just respond 200 with no content too (make sure to set Cache-Control so the client won't cache the result and will retry when they hit refresh). 
Alan_ShaThanks for the hints! Since I am dealing with a TCP socket connection instead of HTTP, I use TCP::respond instead and rerun my test.
When I just used the TCP::respond alone to send a blank message back to client, the client was able to receive the response message and start to send another one. However, the client connection was still closed immediately after the response was sent from Big IP. So, eventually, the second request never got to Big IP and the client failed.
Then, I added the TCP::notify response to trigger the USER_RESPONSE event to do some clean-up and detach the server connection (even it is never connected. See below). This time, the client connection stayed alive! However, the subsequent send from the client always got stuck, even it was going to a different pool that is up and running. I put some debug statement in the LB_SELECTED event, it never shows up in the log, which seems to prove that it never got fired after the previous connection got stuck. It appears to be me that somehow the LB was not cleaned up after the LB::detach, which is blocking the next LB select.
How could I clean up the LB when a connection attempt fails?when USER_RESPONSE { if {$debug} { log local0.debug "Receive USER response ... " } LB::detach if {$debug} { log local0.debug "Detach server connection." } reset some of the flags set responsePending 0 set initResponse 1 set connRetry 0 } when LB_FAILED { set connRetry [expr {$connRetry + 1}] if { $connRetry < $maxAttempts } { log local0.error "Conn. to [IP::server_addr] failed. Retrying..." LB::reselect } else { log local0.error "Max. re-conn. attempts exhausted. Transaction dropped." TCP::respond "$::STX$::ETX" serverside {TCP::notify response} } }
Thx,
Alan- What happens if you just do the LB::detach instead of issuing the "TCP::notify response"? I suspect you may be right that the original LB code isn't getting properly cleaned up. I'm thinking that you don't actually need the USER_RESPONSE event, so let's try to eliminate that first since it just adds complexity.
I still have some tricks up my sleeve, so I think we can get this to work. Let me know if the behavior is the same with just doing the LB::detach in the LB_FAILED event. - Alan_ShaI agree that we don't really need the USER_RESPONSE event. We can just perform the LB::detach in the LB_FAILED event.
But the behavior is still the same after I replace the "TCP::notify response" statement with "LB::detach". - I think part of the problem might be that you need to "drop" the current package. You should be doing this with the following command:
You could probably put this right before your TCP::respond. I'm not totally sure this will get us all the way there, but you do need to remove the queue'd transaction since you are responding to it.TCP::payload replace 0 [TCP::payload length] "" - Alan_ShaThat doesn't work either. It appears that the TCP:payload is already empty at the point the connection fails. I put some print statements to the iRule to display the [TCP::payload length] value, it shows 0. I evaluated this for both the clientside and serverside context, and both show 0. I am suspecting that once a pool statement is issued, the TCP::payload will be transferred to another internal buffer, waiting to be sent out. If this buffer is locked up or not freed for whatever reason, then any subsequent incoming request will not be able to get through because they won't be able to go into this buffer before they are forwarded to the destination.
I am just guessing. Big IP might work differently. But, to me, definitely something is blocking the traffic to flow. - I'm still trying to figure out the best way to handle this. One crazy approach would be to attach to the localhost discard port and let the transaction be dumped into that. So, something like this:
when LB_FAILED { set connRetry [expr {$connRetry + 1}] if { $connRetry < $maxAttempts } { log local0.error "Conn. to [IP::server_addr] failed. Retrying..." LB::reselect } else { log local0.error "Max. re-conn. attempts exhausted. Transaction dropped." node 127.0.0.1 9 } } when SERVER_CONNECTED { if { $connRetry >= $maxAttempts } { clientside {TCP::respond "$::STX$::ETX"} LB::detach } } - Hmm, just realized a potential problem with the above approach. We don't install inetd on the localhost. So, there's nothing listening on that particular port. Perhaps you can use the discard service on another system.
- Alan_ShaWell, it does work. I re-routed the failed transaction to another system and let the system discard it. After that, the sub-sequent requests were able to flow without any problem. That pretty much solved the problem.
Another thing I am trying right now is to re-route the traffic to another virtual server and let the iRule drop the data. However, that doesn't seem to work. I couldn't see the connection being established between the two virtual servers at all. Is it by design that we won't be able to forward traffic from one virtual server to another? - I actually left that option out because we didn't support it - yet.
It's not by design that it doesn't work, but merely due to an implementation detail that is being worked on. Support for that will hopefully be complete by the next major release.
