Error "Address in use" when doing pool

Normal 0 false false false EN-US ZH-CN X-NONE MicrosoftInternetExplorer4

We are implementing a high availability solution for a messaging based product (The communication is message based).

 

 

One persistent client connection will be handled by multiple backend servers. The message format we are using is private (2 bytes binary header + binary payload). We will do load balancing based on one field in the message. We created iRules script and use message based load balancing (mblb) profile and TCP profile to support it.

 

 

The iRules script works fine under low volume (less than 20 TPS), but we got error and client connection reset when increase the volume (e.g. 30 TPS), the following are the error messages:

 

 

 

 

Apr 20 16:43:21 local/tmm1 info tmm1[5549]: Rule Score-MBLB : Key: 0000000000000000003

 

 

Apr 20 16:43:21 local/tmm1 err tmm1[5549]: 01220001:3: TCL error: Score-MBLB - Address in use (line 1) invoked from within "pool [LB::server pool] member [lindex $W 0] [lindex $W 1]"

 

 

Apr 20 16:43:21 local/tmm1 err tmm1[5549]: 01230140:3: RST sent from 10.104.77.17:7778 to 10.104.78.142:53083, Unknown reason

 

 

Apr 20 16:43:21 local/tmm1 err tmm1[5549]: 01230140:3: RST sent from 10.104.78.188:53083 to 10.104.78.142:8778, [0x11aa220:2564] conn

 

 

Apr 20 16:43:23 local/tmm err tmm[5548]: 01230140:3: RST sent from 10.104.78.142:7778 to 10.104.78.189:3690, [0x11a7431:5112] RST from BIG-IP internal Linux host

 

 

What causes the “Address in use” error? Could we resolve this issue through configuration?

 

 

The following are code that cause the error (bold line):

 

 

when CLIENT_DATA {

 

while { [TCP::payload length] >= 2 } {

 

binary scan [TCP::payload] S a

 

 

set mlen [expr { [expr { $a & 0xffff }] + 2 }]

 

 

if { [TCP::payload length] < $mlen } {

 

break

 

}

 

 

binary scan [TCP::payload] x2S b

 

set hlen [expr { $b & 0xffff }]

 

log local0. "Header len: $hlen"

 

 

Get key

 

set keyoffset [ expr {8 + $hlen + 117} ]

 

binary scan [TCP::payload] x[expr {8 + $hlen + 117}]a19 key

 

log local0. "Key: $key"

 

 

Simple election load balancing based on key

 

set S ""

 

foreach N [active_members -list [LB::server pool]] {

 

if { [md5 $N$key] > $S } {

 

set S [md5 $N$key]

 

set W $N

 

}

 

}

 

 

log local0. "Sending request to $W"

 

pool [LB::server pool] member [lindex $W 0] [lindex $W 1]

 

 

 

TCP::release $mlen

 

TCP::notify request

 

}

 

TCP::collect

 

}