I have an issue where all traffic in a pool is going to the fifth of 5 pool members. We are using cookie persistence and I know there can be issues with that. We are also using both an http and OneConnect profile. From the sols and DC articles I’ve read it seems like the problem will arise if you do NOT have a oneconnect profile assigned. I have tried pulling http/oneconnect/tcp profiles on and off, using default profiles. The only thing I haven’t tried is falling back to source address persistence. That is a last resort I do not want to use. Due to the application architecture source address persistence will definitely result in uneven load. All the “custom” profiles are built from an F5 guide for the application (Epic HyperSpace Web; Link to Guide). That being said this config seems pretty straight forward, but any help would be very much appreciated. We have captured traffic and seen the cookies present in the sessions.
Below is the config.
ltm virtual /PARTITION/v_80 {
destination /PARTITION/10.10.1.1:80
ip-protocol tcp
mask 255.255.255.255
persist {
/PARTITION/Custom-cookie {
default yes
}
}
pool /PARTITION/pool_80
profiles {
/PARTITION/Custom-OneConnect { }
/PARTITION/Custom-http { }
/PARTITION/Custom-lan-optimized { }
}
source 0.0.0.0/0
source-address-translation {
type automap
}
translate-address enabled
translate-port enabled
}
ltm pool /PARTITION/Pool_80 {
description "HTTP Pool"
load-balancing-mode least-connections-member
members {
/PARTITION/001:80 {
address 10.1.1.1
}
/PARTITION/002:80 {
address 10.1.1.2
}
/PARTITION/003:80 {
address 10.1.1.3
}
/PARTITION/004:80 {
address 10.1.1.4
}
/PARTITION/005:80 {
address 10.1.1.5
}
}
monitor /Common/http_head_f5
service-down-action reselect
}
Only change to custom oneconnect profile is mask is 255.255.255.255.
Only change to tcp-lan-optimized profile is the idle timeout is set to 1200s.
Only change to http profile is that ‘Redirect Rewrite’ is set to Matching.
Custom cookie persistence uses default settings.
Cleared all cookies & cache on the browser by any chance?
used a different browser?
What model is the Big-ip? CMP enabled? (this can have an affect on LB’s decisions)
disable or force offline pool member 5 and what happens?
Cleared all cookies & cache on the browser by any chance?
used a different browser?
What model is the Big-ip? CMP enabled? (this can have an affect on LB’s decisions)
disable or force offline pool member 5 and what happens?
Nikhil here are the answers to your questions. Cleared all cookies & cache on the browser by any chance? Yes. used a different browser? Yes. What model is the Big-ip? CMP enabled? (this can have an affect on LB’s decisions) This is a virtual edition running 11.4.1HF7. CMP is enabled on this virtual disable or force offline pool member 5 and what happens? I have not been given the OK to do this yet.
Nikhil here are the answers to your questions. Cleared all cookies & cache on the browser by any chance? Yes. used a different browser? Yes. What model is the Big-ip? CMP enabled? (this can have an affect on LB’s decisions) This is a virtual edition running 11.4.1HF7. CMP is enabled on this virtual disable or force offline pool member 5 and what happens? I have not been given the OK to do this yet.
You going to need to replicate the environment in a non-production setting to perform functional testing. If you have one already does this problem occur there?
From your F5 can you do curl 10.1.1.1 and get a response? Try the other pool membes as well.
Are you sending lots of HTTP traffic down single connections? If so how many requests (rough estimate) per connection.
How many clients are using the service? The clients users or an application calling the service.
“show ltm persistence persist-records” - consistently shows 5 in the table? Tried using a different persistence profile? (hash?) Any connection limits at the node or pool member level? what LB algo are you using? (least sessions works best with persistence)
“show ltm persistence persist-records” - consistently shows 5 in the table? Tried using a different persistence profile? (hash?) Any connection limits at the node or pool member level? what LB algo are you using? (least sessions works best with persistence)
“show ltm persistence persist-records” - consistently shows 5 in the table? Yes. Tried using a different persistence profile? (hash?) TBD Any connection limits at the node or pool member level? Not that I am aware of. None configured in any profiles. what LB algo are you using? (least sessions works best with persistence) Least Connections (member). There are few, if any, connections left as the activity has all but ceased for the day. I will attempt using cookie hash persistence tomorrow as well as changing the lb algo.
“show ltm persistence persist-records” - consistently shows 5 in the table? Yes. Tried using a different persistence profile? (hash?) TBD Any connection limits at the node or pool member level? Not that I am aware of. None configured in any profiles. what LB algo are you using? (least sessions works best with persistence) Least Connections (member). There are few, if any, connections left as the activity has all but ceased for the day. I will attempt using cookie hash persistence tomorrow as well as changing the lb algo.
This is in a non-prod environment. This is in preparation for deploying a production environment. Curl results in a 200 OK HTTP response from each pool member. No to lots of traffic from single connections. Right now we should have roughly 5-10 per, but when this goes to production that will go to at least 150 per. These users are running a browser from citrix servers. We have verified the users do have unique BigIP issued cookies in their sessions.
Using cookie hash persistence and least sessions did not result in any change. We were also able to create a duplicate vip/pool combo to do some additional side-by-side testing since this is not production. This is set with source-address persistence and least-connections (member) LBing. Here are the persistence records: source-address 10.12.2.1 10.10.1.2:80 10.1.1.4:80 (tmm: 0) source-address 10.12.2.3 10.10.1.2:80 10.1.1.4:80 (tmm: 0)
Using cookie hash persistence and least sessions did not result in any change. We were also able to create a duplicate vip/pool combo to do some additional side-by-side testing since this is not production. This is set with source-address persistence and least-connections (member) LBing. Here are the persistence records: source-address 10.12.2.1 10.10.1.2:80 10.1.1.4:80 (tmm: 0) source-address 10.12.2.3 10.10.1.2:80 10.1.1.4:80 (tmm: 0)
The browsers are getting unique cookies in their sessions would mean they are getting cookies for different pool members. Are they keeping the same cookie value throughout their session or is it changing with each request?
If you are referring to each request within a single session they were retaining the same cookie value.
I have forcefully removed the persistence table records for this virtual. When they are recreated they are exactly the same. Does this indicate something is not working or do I have something configured incorrectly?
I have forcefully removed the persistence table records for this virtual. When they are recreated they are exactly the same. Does this indicate something is not working or do I have something configured incorrectly?
Since this is not production, can you remove node 10.1.1.4 and test again?