Forum Discussion

Shu_97237's avatar
Shu_97237
Icon for Nimbostratus rankNimbostratus
Jul 12, 2012

Dispatch messages from a single connection to pool members

Hi,

 

 

I'm trying to write irules to do message-based load balancing. I am not using SIP/radius/diameter, so cannot use an existing profile. The request messages are all sent over a single TCP connection. Each message has a 8-byte header, of which the last 4 bytes store the length of the message body. I want the load balancer to forward messages to pool members in a round-robin manner. Currently, the problem is after "TCP::notify request" is executed, the USER_REQUEST won't fire. The irules are pasted below:

 

 

 

when CLIENT_ACCEPTED {

 

log local0.debug "Get a connection"

 

TCP::collect

 

}

 

 

 

when CLIENT_DATA {

 

log local0.debug "in CLIENT_DATA"

 

binary scan [TCP::payload] II head rlen

 

if {($head & 0x3) == 1} {

 

if {[TCP::payload length] < $rlen} {

 

TCP::collect $rlen

 

log local0.debug "a message is received"

 

TCP::release $rlen

 

TCP::notify request

 

log local0.debug "requested"

 

}

 

}

 

}

 

 

 

when USER_REQUEST {

 

Dispatch to the pool

 

log local0.debug "in USER_REQUEST"

 

pool my_pool

 

}

 

 

 

 

Thank you for your help!

 

Shu

 

 

 

  • this is my testing. client sends 2 http requests in 1 tcp connection. each request is load balanced to each pool member.

    in irule, i determine end of each request by looking for \r\n\r\n.

    [root@ve10:Active] config  b virtual bar list
    virtual bar {
       snat automap
       pool foo
       destination 172.28.19.79:80
       ip protocol 6
       rules myrule
       profiles {
          mblb {}
          tcp {}
       }
    }
    [root@ve10:Active] config  b pool foo list
    pool foo {
       members {
          200.200.200.101:80 {}
          200.200.200.102:80 {}
       }
    }
    [root@ve10:Active] config  b rule myrule list
    rule myrule {
       when CLIENT_ACCEPTED {
       log local0. "client: [IP::client_addr]:[TCP::client_port]"
       log local0. "virtual: [IP::local_addr]:[TCP::local_port]"
       TCP::collect
    }
    when CLIENT_DATA {
       log local0. "request: [TCP::payload]"
       if {[TCP::payload] ends_with "\r\n\r\n"} {
          TCP::release
          TCP::notify request
       }
       TCP::collect
    }
    when SERVER_CONNECTED {
       log local0. "node: [IP::remote_addr]:[TCP::remote_port]"
       TCP::collect
    }
    when SERVER_DATA {
       log local0. "response: [TCP::payload]"
       TCP::release
       TCP::notify response
       TCP::collect
    }
    when SERVER_CLOSED {
       log local0. ""
    }
    when CLIENT_CLOSED {
       log local0. ""
    }
    }
    
    /var/log/ltm
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule : client: 172.28.19.251:37382
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule : virtual: 172.28.19.79:80
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule : request: GET / HTTP/1.0  User-Agent: ApacheBench/2.0.40-dev  Connection: Keep-Alive  Host: 172.28.19.79  Accept: */*
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule : node: 200.200.200.102:80
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule : response: HTTP/1.1 200 OK  Date: Sat, 14 Jul 2012 08:35:19 GMT  Server: Apache/2.2.3 (CentOS)  Last-Modified: Tue, 08 Nov 2011 12:26:29 GMT  ETag: "4183f1-30-47e02740"  Accept-Ranges: bytes  Content-Length: 48  Keep-Alive: timeout=15, max=100  Connection: Keep-Alive  Content-Type: text/html; charset=UTF-8      This is 102 host.  
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule : request: GET / HTTP/1.0  User-Agent: ApacheBench/2.0.40-dev  Connection: Keep-Alive  Host: 172.28.19.79  Accept: */*
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule :
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule : node: 200.200.200.101:80
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule : response: HTTP/1.1 200 OK  Date: Sat, 14 Jul 2012 08:47:25 GMT  Server: Apache/2.2.3 (CentOS)  Last-Modified: Fri, 11 Nov 2011 14:48:14 GMT  ETag: "4183e4-3e-9c564780"  Accept-Ranges: bytes  Content-Length: 62  Keep-Alive: timeout=15, max=100  Connection: Keep-Alive  Content-Type: text/html; charset=UTF-8       This is 101 host.  
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule :
    Jul 14 16:35:00 local/tmm info tmm[5111]: Rule myrule :
    
  • Thank you nitass!

     

     

    I tried to follow your example, but have not succeeded yet. Using "\r\n\r\n" to find the end of message may not work in my application, which does not use HTTP. Is it standard for HTTP messages? In your example, you don't handle USER_REQUEST event. Is there a default action for USER_REQUEST event?

     

     

     

    My understanding is whether I use length field found in message header or use "\r\n\r\n" to find the end of a message, as long as I call TCP::notify request, USER_REQUEST should be triggered. In my case, it has never been triggered. What prevents it from happening?

     

     

     

    Below is the latest configuration I used:

     

     

     

    datastor {

     

    low water mark 80

     

    high water mark 92

     

    }

     

    deduplication {}

     

    shell write partition Common

     

    configsync {

     

    password crypt "****"

     

    }

     

    route default inet {

     

    vlan external

     

    }

     

    monitor MYMBLB {

     

    defaults from inband

     

    }

     

    node 10.124.66.35 {}

     

    node 10.124.66.48 {}

     

    pool MYMBLB_pool {

     

    monitor all MYMBLB

     

    members {

     

    10.124.66.35:cslistener {}

     

    10.124.66.48:cslistener {}

     

    }

     

    }

     

    snat snat_one_ip_test {

     

    translation 10.124.6.35

     

    origins {

     

    10.124.66.35

     

    10.124.66.48

     

    }

     

    vlans external enable

     

    }

     

    rule MYRULES {

     

    when CLIENT_ACCEPTED {

     

    log local0.debug "Get a connection"

     

    set client_closed 0

     

    TCP::collect 1000

     

    }

     

     

     

    when CLIENT_CLOSED {

     

    set client_closed 1

     

    }

     

     

     

     

     

    when SERVER_CONNECTED {

     

    log local0.debug "Server_connected"

     

    after 1000 -periodic if {$client_closed} {TCP::close}

     

    TCP::collect

     

    }

     

     

     

    when CLIENT_DATA {

     

    log local0.debug "in CLIENT_DATA"

     

     

    binary scan [TCP::payload] II head rlen

     

    if {($head & 0x3) == 1} {

     

    if {[TCP::payload length] < $rlen} {

     

    TCP::collect $rlen

     

    log local0.debug "a message is received"

     

    TCP::release $rlen

     

    TCP::notify request

     

    log local0.debug "requested"

     

    }

     

    }

     

    TCP::collect

     

    }

     

     

     

    }

     

    virtual MYMBLBLB {

     

    pool MYMBLB_pool

     

    destination 10.124.6.35:cslistener

     

    ip protocol tcp

     

    rules MYRULES

     

    profiles {

     

    mblb {}

     

    tcp {}

     

    }

     

    }

     

     

  • I read the wiki of TCP::notify again trying to understand the meaning of "the USER_REQUEST event will not be raised until the server-side TCP connection is complete". Maybe the problem is virtual server cannot establish a connection with the pool member. How can I test if the virtual server can talk to pool members correctly?
  • Using "\r\n\r\n" to find the end of message may not work in my application, which does not use HTTP. Is it standard for HTTP messages?i used http for testing since i do not have your application. in my case, \r\n\r\n is used to find an end of each message. in your case, you have to parse header to get payload length.

     

     

    In your example, you don't handle USER_REQUEST event. Is there a default action for USER_REQUEST event?you do not need to have USER_REQUEST event. TCP::notify is used to tell bigip it is an end of each message.

     

     

    additionally, TCP::notify is not working in 11.1.0. it is tracked as ID 383853. if you are using 11.1.0, please open a support case for verification.

     

     

    ID 383853 - Need a synchronous event to signal end of message from TCP rule event
  • not sure if code is correct. anyway, would you mind trying it? i assume only one message per packet.

     

     

    [root@ve10:Active] config  b rule myrule list
    rule myrule {
       when CLIENT_ACCEPTED {
       TCP::collect
    }
    when CLIENT_DATA {
       binary scan [TCP::payload] @4I len
       if {[TCP::payload length] < [expr {8 + $len}]} {
          TCP::collect [expr {8 + $len - [TCP::payload length]}]
          return
       }
       TCP::release [expr {8 + $len}]
       TCP::notify request
       TCP::collect
    }
    }
    
  • Thank you very much nitass!

     

     

    The irules you wrote works. My F5 still doesn't work well, it should be caused by network configuration issue but not irule. The irules to do simple MBLB turned out pretty straight forward.

     

  • Posted By nitass on 07/22/2012 05:46 AM

     

    not sure if code is correct. anyway, would you mind trying it? i assume only one message per packet.

     

    nitass, I don't understand why you need to assume only one message per packet? Our message is bigger than a TCP packet, e.g. 2830 bytes. Is there anything to pay attention?