Forum Discussion

Steve_M__153836's avatar
Steve_M__153836
Icon for Nimbostratus rankNimbostratus
Jan 19, 2015

Load Balancing to Only One Pool Member

I have an issue where all traffic in a pool is going to the fifth of 5 pool members. We are using cookie persistence and I know there can be issues with that. We are also using both an http and OneConnect profile. From the sols and DC articles I've read it seems like the problem will arise if you do NOT have a oneconnect profile assigned. I have tried pulling http/oneconnect/tcp profiles on and off, using default profiles. The only thing I haven't tried is falling back to source address persistence. That is a last resort I do not want to use. Due to the application architecture source address persistence will definitely result in uneven load. All the "custom" profiles are built from an F5 guide for the application (Epic HyperSpace Web; Link to Guide). That being said this config seems pretty straight forward, but any help would be very much appreciated. We have captured traffic and seen the cookies present in the sessions.

Below is the config.

ltm virtual /PARTITION/v_80 {
    destination /PARTITION/10.10.1.1:80
    ip-protocol tcp
    mask 255.255.255.255
    persist {
        /PARTITION/Custom-cookie {
            default yes
        }
    }
    pool /PARTITION/pool_80
    profiles {
        /PARTITION/Custom-OneConnect { }
        /PARTITION/Custom-http { }
        /PARTITION/Custom-lan-optimized { }
    }
    source 0.0.0.0/0
    source-address-translation {
        type automap
    }
    translate-address enabled
    translate-port enabled
}

ltm pool /PARTITION/Pool_80 {
    description "HTTP Pool"
    load-balancing-mode least-connections-member
    members {
        /PARTITION/001:80 {
            address 10.1.1.1
        }
        /PARTITION/002:80 {
            address 10.1.1.2
        }
        /PARTITION/003:80 {
            address 10.1.1.3
        }
        /PARTITION/004:80 {
            address 10.1.1.4
        }
        /PARTITION/005:80 {
            address 10.1.1.5
        }
    }
    monitor /Common/http_head_f5 
    service-down-action reselect
}

Only change to custom oneconnect profile is mask is 255.255.255.255.

Only change to tcp-lan-optimized profile is the idle timeout is set to 1200s.

Only change to http profile is that 'Redirect Rewrite' is set to Matching.

Custom cookie persistence uses default settings.

  • Cleared all cookies & cache on the browser by any chance?

     

    used a different browser?

     

    What model is the Big-ip? CMP enabled? (this can have an affect on LB's decisions)

     

    disable or force offline pool member 5 and what happens?

     

    • Steve_M__153836's avatar
      Steve_M__153836
      Icon for Nimbostratus rankNimbostratus
      Nikhil here are the answers to your questions. Cleared all cookies & cache on the browser by any chance? Yes. used a different browser? Yes. What model is the Big-ip? CMP enabled? (this can have an affect on LB's decisions) This is a virtual edition running 11.4.1HF7. CMP is enabled on this virtual disable or force offline pool member 5 and what happens? I have not been given the OK to do this yet.
    • NikhilB's avatar
      NikhilB
      Icon for Employee rankEmployee
      "show ltm persistence persist-records" - consistently shows 5 in the table? Tried using a different persistence profile? (hash?) Any connection limits at the node or pool member level? what LB algo are you using? (least sessions works best with persistence)
    • Steve_M__153836's avatar
      Steve_M__153836
      Icon for Nimbostratus rankNimbostratus
      "show ltm persistence persist-records" - consistently shows 5 in the table? Yes. Tried using a different persistence profile? (hash?) TBD Any connection limits at the node or pool member level? Not that I am aware of. None configured in any profiles. what LB algo are you using? (least sessions works best with persistence) Least Connections (member). There are few, if any, connections left as the activity has all but ceased for the day. I will attempt using cookie hash persistence tomorrow as well as changing the lb algo.
  • NikhilB_149913's avatar
    NikhilB_149913
    Historic F5 Account

    Cleared all cookies & cache on the browser by any chance?

     

    used a different browser?

     

    What model is the Big-ip? CMP enabled? (this can have an affect on LB's decisions)

     

    disable or force offline pool member 5 and what happens?

     

    • Steve_M__153836's avatar
      Steve_M__153836
      Icon for Nimbostratus rankNimbostratus
      Nikhil here are the answers to your questions. Cleared all cookies & cache on the browser by any chance? Yes. used a different browser? Yes. What model is the Big-ip? CMP enabled? (this can have an affect on LB's decisions) This is a virtual edition running 11.4.1HF7. CMP is enabled on this virtual disable or force offline pool member 5 and what happens? I have not been given the OK to do this yet.
    • NikhilB_149913's avatar
      NikhilB_149913
      Historic F5 Account
      "show ltm persistence persist-records" - consistently shows 5 in the table? Tried using a different persistence profile? (hash?) Any connection limits at the node or pool member level? what LB algo are you using? (least sessions works best with persistence)
    • Steve_M__153836's avatar
      Steve_M__153836
      Icon for Nimbostratus rankNimbostratus
      "show ltm persistence persist-records" - consistently shows 5 in the table? Yes. Tried using a different persistence profile? (hash?) TBD Any connection limits at the node or pool member level? Not that I am aware of. None configured in any profiles. what LB algo are you using? (least sessions works best with persistence) Least Connections (member). There are few, if any, connections left as the activity has all but ceased for the day. I will attempt using cookie hash persistence tomorrow as well as changing the lb algo.
  • You going to need to replicate the environment in a non-production setting to perform functional testing. If you have one already does this problem occur there?

     

    From your F5 can you do curl 10.1.1.1 and get a response? Try the other pool membes as well.

     

    Are you sending lots of HTTP traffic down single connections? If so how many requests (rough estimate) per connection.

     

    How many clients are using the service? The clients users or an application calling the service.

     

    • Steve_M__153836's avatar
      Steve_M__153836
      Icon for Nimbostratus rankNimbostratus
      This is in a non-prod environment. This is in preparation for deploying a production environment. Curl results in a 200 OK HTTP response from each pool member. No to lots of traffic from single connections. Right now we should have roughly 5-10 per, but when this goes to production that will go to at least 150 per. These users are running a browser from citrix servers. We have verified the users do have unique BigIP issued cookies in their sessions.
    • Brad_Parker's avatar
      Brad_Parker
      Icon for Cirrus rankCirrus
      The browsers are getting unique cookies in their sessions would mean they are getting cookies for different pool members. Are they keeping the same cookie value throughout their session or is it changing with each request?
    • Steve_M__153836's avatar
      Steve_M__153836
      Icon for Nimbostratus rankNimbostratus
      If you are referring to each request within a single session they were retaining the same cookie value.
  • So it looks like in this environment (load balancing traffic from user sessions on Citrix servers) there is some issue with the combination of an http profile, onceconnect profile, cookie session persistence, and the least connections (member) load balancing method. When changing to Observed (member; retaining cookie session persistence and other profiles) our load balancing issue was resolved. According to F5 this is related to CMP in a manner I do not understand.

     

    • NikhilB's avatar
      NikhilB
      Icon for Employee rankEmployee
      Yes, thx for the update. Could be related to just Citrix servers.
  • Remco's avatar
    Remco
    Icon for Nimbostratus rankNimbostratus

    Hi, we hit the same issue when we upgraded from 10.2.4 to 11.4.1HF8.

     

    This is the answer we recieved from F5 when we opened a case:

     

    This BIGIP is hitting a known issuse ID504538: Bug alias 504538-1 OneConnect + Least connections lb mode (member) is broken.

     

    So if you are happy to remove OneConnect profile or use other LB method, this should be resolved. We also should be able to provide you with EngHF on top of the latest HF.

     

    • Steve_M__153836's avatar
      Steve_M__153836
      Icon for Nimbostratus rankNimbostratus
      Remco thank you for posting. I think my case prompted the known issue/bug. We are happy with the Observed (member) LB method so we'll stick with that. I actually think I may utilize that more in the future.
  • Hi, we hit the same issue when we upgraded from 10.2.4 to 11.4.1HF8.

     

    This is the answer we recieved from F5 when we opened a case:

     

    This BIGIP is hitting a known issuse ID504538: Bug alias 504538-1 OneConnect + Least connections lb mode (member) is broken.

     

    So if you are happy to remove OneConnect profile or use other LB method, this should be resolved. We also should be able to provide you with EngHF on top of the latest HF.

     

    • Steve_M__153836's avatar
      Steve_M__153836
      Icon for Nimbostratus rankNimbostratus
      Remco thank you for posting. I think my case prompted the known issue/bug. We are happy with the Observed (member) LB method so we'll stick with that. I actually think I may utilize that more in the future.