Slow application performance when using BIG-IP LTM VE for load balancing.
The Question
Is there some limitation on the BIG-IP F5 LTM that prevents it from being able to support a large user load and/or are there any configurations that could specifically interfere with performance of an application?
Supporting Information
I am load testing our application, and wanting to utilize our virtual F5 as the load balancer for the various clusters. Currently I am seeing slow response times when using a Virtual F5 as the load balancer. The slow performance is most apparent when the system is under load. While I can see slow performance in single user tests, it is not as extreme as when the system is under a large user load. When using other load balancing options (in our specific case, NLB) I am not seeing the same slow performance.
I have tried a variaty of debugging steps, but none have really helped me to put a finger on a solution to the problem.
1. Checking VM resource usage
All VMs show acceptable resource consumption and availability (no VMs appear to be strained... this includes the F5 VM.)
2. Checking ESX host usage
All ESX hosts show acceptable resource consumption and availability (no ESX hosts appear to be strained.)
3. Adjustments to F5 configuration
Disabled oneConnect for the F5 virtual server configuration. There was an issue discovered earlier with using oneConnect with our application and the 11.x versions of F5. This change did not have any obvious affect on test results.
4. Test validation:
We have ran the same test with another load balancer set-up. This set-up shows acceptable response times with that configuration. Is there perhaps a limitation with the virtual server as to the max number of users we can expect to support? If so what might this number be?
5. Comparison against other environments
I have compared configurations between an environment using a physical F5 to the the virtual F5 setup I am having issues with. I am not seeing any noticeable differences that would potentially cause issues with performance. It should be noted that using the physical F5 is yielding expected response time performance.
6. Analytics from the F5 console
Monitoring Latency of pool members (while under load) is showing an average latency of 1,000+ ms. This seems high... particularly for a virtualized environment. Is this perhaps an F5 VM limitation, or is there perhaps something at an F5 configuration level that we are overlooking, that could cause this?
7. Changing vNIC type
It had been suggested that the particular vNIC being used may be a possible source of bad behavior. As such, the vNIC type was changed from Intel to vmxnet3. Current test executions thus far have not shown any noticeable change, but there are additional tests to be executed in this avenue.
8. DynaTrace Analysis
This is the latest testing being done, and as such the results are still under analysis. However, initial test runs suggest that the majority of the extra time is being spent in two points:
- Requests between load test agents and
- Requests between Java Tier and IIS host
This seems to suggest that the F5 is somehow bottle-necking the request.
Version Information
- F5 Version: BIG-IP v11.3.0 (Build 2806.0)
- Node OS version (Windows 2008 r2)
- VMWare Tools Version: Version 9.4.0, build-1280544
Thank you in advance for any help that can be provided.
The overall answer to this question is, yes there is a limitation... but it depends on the license type purchased. The license I had available to me had a 10 Mbps limitation on it, rendering it useless for load testing scenarios. This information in the assorted comment threads.
The following references on F5 were helpful, but ultimately, I would have liked to find a SOL that explicitly calls out this limitation, as it feels somewhat masked (it may not be, when talking to account teams from F5, but unfortunately I came into this post-purchase.) It is not listed in the licensing information when running tmsh show /sys license.
- https://devcentral.f5.com/questions/limited-bandwidth-on-virtual-edition
- https://devcentral.f5.com/questions/f5-big-ve-lab-license
- https://devcentral.f5.com/questions/bigip-ltm-ve-lab-limitations
Thank you again to WLB for all of the help. It was WLB's guidance that helped me find the answer to this question. I am just posting the direct answer so that the thread can be closed.