Forum Discussion
voxbox
Nimbostratus
NimbostratusLTM - Connection limit not respected
Hi, I setup a very basic LTM. Type Standard. TCP + HTTP + OneConnect default profiles. Going to 2 different pools in the backend. I set a connection limit of 10. When doing some tests, I ...
Hi,
F5 BIG-IP is based on Clustered Multiprocessing where traffic needs to be distributed across the CPUs, this is referred to as disaggregation. The connection limit is divided across the number of TMMs running in the system. If you set a connection limit of 16 and your system has 16 TMMs you can hit the connection limit if all your traffic lands on the same TMM. Traffic is distributed to the different TMMs based on the disaggregation hash. If your testbed is using a limited number of servers to generate traffic you will also only leverage a limited number of TMMs. Other behavior that can impact how traffic flows is many load generators are based on the linux kernel, many kernels have a source port stride of 2 (IE if the first connection is on an odd port the rest of the outgoing connections will be on an odd port, same pattern if you start on an even port). The behavior you are seeing is expected and you will need to increase the distribution of entropy (IP address and source ports) to create an accurate test as the system is designed to handle a large number of clients and a large number of connections creating a large array of entropy in how the connections present on the client side. For example my own test bed consisted of a a locust web traffic array that runs on 5 servers with 360 CPUs to generate traffic entropy. For a non-production system you can disable the CMP processing on a virtual server with the following commands : tmsh modify ltm virtual <virtual server name> cmp-enabled no then tmsh save /sys config
Heath_Parrott
Employee
EmployeeHi,
F5 BIG-IP is based on Clustered Multiprocessing where traffic needs to be distributed across the CPUs, this is referred to as disaggregation. The connection limit is divided across the number of TMMs running in the system. If you set a connection limit of 16 and your system has 16 TMMs you can hit the connection limit if all your traffic lands on the same TMM. Traffic is distributed to the different TMMs based on the disaggregation hash. If your testbed is using a limited number of servers to generate traffic you will also only leverage a limited number of TMMs. Other behavior that can impact how traffic flows is many load generators are based on the linux kernel, many kernels have a source port stride of 2 (IE if the first connection is on an odd port the rest of the outgoing connections will be on an odd port, same pattern if you start on an even port). The behavior you are seeing is expected and you will need to increase the distribution of entropy (IP address and source ports) to create an accurate test as the system is designed to handle a large number of clients and a large number of connections creating a large array of entropy in how the connections present on the client side. For example my own test bed consisted of a a locust web traffic array that runs on 5 servers with 360 CPUs to generate traffic entropy. For a non-production system you can disable the CMP processing on a virtual server with the following commands : tmsh modify ltm virtual <virtual server name> cmp-enabled no then tmsh save /sys config
voxboxHi Heath,
Thanks for the answer. I can confirm when disabling CMP its accepting all 10 connections now.
Quick follow-up, does the same thing apply to backend connections (with OneConnect)? Does each TMM have its own connections to the backend pool, or are all the oneconnect connections shared between the TMMs?- Heath_Parrott
Hi,
Connection limits can be set at different objects: virtual servers, pools and nodes. The connection from BIG-IP to the backend pool member is unique for each TMM (conceptually reverse the incoming CMP/DAG hash from a client side function to a server side function) this ensures that the requests and response flow through the same TMM. One connect does not create a singular connection between a TMM and a backend pool member, it creates the ability to reuse a TCP connection that has already been established to a backend pool member by manipulating the HTTP connection close function on the server side. A one connect profile will allow a single TMM to open multiple connections to a single node and those connections can be reused for new client HTTP requests based on the one connect profile setting - https://my.f5.com/manage/s/article/K7208 .