Forum Discussion

sand87ch's avatar
sand87ch
Icon for Cirrus rankCirrus
Dec 05, 2022

BigIP F5 LTM fails during Load testing

My Set up has a BIGIP single CPU hosted on AWS.This sits in front of a node API i use for load testing. Now when i send a 1K_1krpm request..i.e., 5000 requests in 5 mins to my node api via the BIG I...
cloud
sand87ch's avatar
sand87ch
Icon for Cirrus rankCirrus
Dec 07, 2022

I am using the F5 Big IP VE BYOL(14.xxx)livcense available in AWS marketplace.And yes the instance is T2 medium.Sorry that my posting confused you.By 1CPu i meant 1 boot location as shown in the attachment below.

The instance i am using is m5.xlarge having 4 cpu and 16GB ram.I hope this is ok.

 

Heath_Parrott's avatar
Heath_Parrott
Icon for Employee rankEmployee
Dec 07, 2022

Yes with BIG-IP running on an m5.xl 4CPU:16GB ram you should be good to load test as that configuration from a BIG-IP standpoint.  The M5xl still has a connection table on the small side (WRT to the AWS SG - sorry AWS does not publicly publish these numbers) that you may be hitting and you should disable connection tracking to ensure that is not the issue. (see AWS documentation on how).  I would recomend doing this if you have not since it is an external limit on the system that will not show up in the BIG-IP logs. 

 

My expereince on hitting SG table limits plays out like:

Generate Traffic load --> Latency --> error.  Let the test bed sit for 15 minutes and you can repeat at will.  Repeating more frequently you hit the error sooner.    

 

 

  • sand87ch's avatar
    sand87ch
    Icon for Cirrus rankCirrus
    Dec 07, 2022

    Heath_Parrott  Reading the document it seems that if the security group allows all inbound and outbound connections then this tracking and blocking of connections wont occur.Correct me if i am wrong.I enabled all traffic on our big ip VE on AWS but still the jmeter tests shows the same errors.Am i missing something here?

    • Heath_Parrott's avatar
      Heath_Parrott
      Icon for Employee rankEmployee
      Dec 07, 2022

      What is the SG config on both BIG-IP and the backend application (also how many application servers are in the pool)?

      What is the instance type on the backend?

      You will have connection tracking limits in all scenarios unless the SG reads permit 0.0.0.0/0 ANY in both directions.  

      Here are the areas that you can exhaust network resources based on how you described the situation and possible mitigation:

      AWS Security Group Flow table full Set all instances and all SGs to 0.0.0.0/0 ANY permit or change to a much larger instance
      SNAT port exhaustion (you have not indicated this by the log messages) Add pool members, add more secondary IPs to the instance and create a snat pool
      Burstable instances on backend (T2/T3)  change to a non-burstable instance type or add more. 
      Backend sending TCP RST to new connections from BIG-IP Open security group, add instances, increase instance size 
      Few clients generating traffic increase number of clients.  Modern linux OS have a port stride of 2 for outbound connections. This can lead to all traffic being pinned to a subset of TMMs (you need to look at individual CPU graphs). 
      Using a rate limited license For rate limited licenses you can have a scneario where you are not hitting the limit but you are hitting the limit/# of TMMs and your flows are pinned to that number of TMMs (example 25Mb/S license, 4 TMMs, all flows pinned to one TMM and you can hit a rater limiter about 6.x Mb/S).  Increase client diversity for actual load test to spread flows over TMMs. Increase license used etc. Open support case for tunning options. 


      The easiest things to narrow down the issue 

      1. SG changes

      2. Increase all instances to larger types and / or add more instances to the pool

      3. Run a load test with client side entropy (many source IPs) if you are not.