Forum Discussion

voxbox's avatar
voxbox
Icon for Nimbostratus rankNimbostratus
Oct 08, 2024

LTM - Connection limit not respected

Hi,

 

I setup a very basic LTM. Type Standard. TCP + HTTP + OneConnect default profiles. Going to 2 different pools in the backend.

 

I set a connection limit of 10.

 

When doing some tests, I noticed that it only seems to accept 7-8 connections but never 10.

 

TCP Goon output:

* You are going to run a TCP stress check with these arguments:
*       - Host: x.x.x.x
*       - TCP Port: 80
*       - # of concurrent connections: 10
*********************************************************************
Do you want to continue? (y/N): y
Total: 10, Dialing: 0, Established: 0, Closed: 0, Error: 0, NotInitiated: 10
Total: 10, Dialing: 0, Established: 6, Closed: 0, Error: 4, NotInitiated: 0

----------------------------------------------------------------------------------------------------------------

Looking in the logs, I'm seeing 

Warning, connections equals limit x.x.x.x:55960 -> x.x.x.x:80, proto TCP, VS /Common/VirtualServer01: Connection limit reached. 

----------------------------------------------------------------------------------------------------------------

When looking at the show rst-cause:

------------------------------------------
TCP/IP Reset Cause
RST Cause:                           Count
------------------------------------------
Connection limit exceeded                4

 

 

Can anyone help me understand this?

 

Thanks

  • Hi,

    F5 BIG-IP is based on Clustered Multiprocessing where traffic needs to be distributed across the CPUs, this is referred to as disaggregation. The connection limit is divided across the number of TMMs running in the system.   If you set a connection limit of 16 and your system has 16 TMMs you can hit the connection limit if all your traffic lands on the same TMM.  Traffic is distributed to the different TMMs based on the disaggregation hash.   If your testbed is using a limited number of servers to generate traffic you will also only leverage a limited number of TMMs.  Other behavior that can impact how traffic flows is many load generators are based on the linux kernel, many kernels have a source port stride of 2 (IE if the first connection is on an odd port the rest of the outgoing connections will be on an odd port, same pattern if you start on an even port).    The behavior you are seeing is expected and you will need to increase the distribution of entropy (IP address and source ports) to create an accurate test as the system is designed to handle a large number of clients and a large number of connections creating a large array of entropy in how the connections present on the client side.  For example my own test bed consisted of a a locust web traffic array that runs on 5 servers with 360 CPUs to generate traffic entropy.  For a non-production system you can disable the CMP processing on a virtual server with the following commands : tmsh modify ltm virtual <virtual server name> cmp-enabled no  then tmsh save /sys config 

  • Hi,

    F5 BIG-IP is based on Clustered Multiprocessing where traffic needs to be distributed across the CPUs, this is referred to as disaggregation. The connection limit is divided across the number of TMMs running in the system.   If you set a connection limit of 16 and your system has 16 TMMs you can hit the connection limit if all your traffic lands on the same TMM.  Traffic is distributed to the different TMMs based on the disaggregation hash.   If your testbed is using a limited number of servers to generate traffic you will also only leverage a limited number of TMMs.  Other behavior that can impact how traffic flows is many load generators are based on the linux kernel, many kernels have a source port stride of 2 (IE if the first connection is on an odd port the rest of the outgoing connections will be on an odd port, same pattern if you start on an even port).    The behavior you are seeing is expected and you will need to increase the distribution of entropy (IP address and source ports) to create an accurate test as the system is designed to handle a large number of clients and a large number of connections creating a large array of entropy in how the connections present on the client side.  For example my own test bed consisted of a a locust web traffic array that runs on 5 servers with 360 CPUs to generate traffic entropy.  For a non-production system you can disable the CMP processing on a virtual server with the following commands : tmsh modify ltm virtual <virtual server name> cmp-enabled no  then tmsh save /sys config 

    • voxbox's avatar
      voxbox
      Icon for Nimbostratus rankNimbostratus

      Hi Heath,

       

      Thanks for the answer. I can confirm when disabling CMP its accepting all 10 connections now.

      Quick follow-up, does the same thing apply to backend connections (with OneConnect)? Does each TMM have its own connections to the backend pool, or are all the oneconnect connections shared between the TMMs?

      • Heath_Parrott's avatar
        Heath_Parrott
        Icon for Employee rankEmployee

        Hi,

        Connection limits can be set at different objects: virtual servers, pools and nodes.  The connection from BIG-IP to the backend pool member is unique for each TMM (conceptually reverse the incoming CMP/DAG hash from a client side function to a server side function) this ensures that the requests and response flow through the same TMM.  One connect does not create a singular connection between a TMM and a backend pool member, it creates the ability to reuse a TCP connection that has already been established to a backend pool member by manipulating the HTTP connection close function on the server side.  A one connect profile will allow a single TMM to open multiple connections to a single node and those connections can be reused for new client HTTP requests based on the one connect profile setting - https://my.f5.com/manage/s/article/K7208 .