Slow application performance when using BIG-IP LTM VE for load balancing.

Not sure what you mean by: "Requests between load test agents and" It's hard to say not knowing the application architecture, and how the application works.

OK, standard set of questions here;

 

• Please double-check all vNICs (apart from mgmt) are VMXNET3
• What TCP profile settings are being used client and server side?
• What vNICs are assigned to the servers?
• Are you using any FastL4 virtuals?
• Have you enabled TCP Segmentation Offload (TSO) in VE?
• Have you enabled Large Receive Offload (LRO) in the hypervisor?
• If your physical host NICs support it, have you enabled SR-IOV support?

There's lots of factors that may influence things and that can be tweaked including;

 

• TCP Slow Start
• Nagle (disabled by default) and Delayed ACKs (enabled by default)
• Server/host receive window, initial congestion window etc.

Anyway, if you can answer/investigate my initial questions we can take it from there.

 

Apologies for the typo. It is not letting me save edits to the original question. That should have been Requests between load test agents and IIS host.

Alright. Sorry for the delay. Interestingly enough I can across a similar issue in my own environment today where file transfers takes approx. 10x long via 'traffic' interfaces compared to the management interface.

 

We can probably leave the TSO/LSO options alone, especially if FastL4 isn't being used. SRV-IO would be nice but shouldn't make a massive difference.

 

• Create custom client and server side TCP profiles

   

• I'd suggest bumping the TCP initial congestion window up to 10 if you can - server and client side, this should be done on the servers/hosts: http://andydavies.me/blog/2011/11/21/increasing-the-tcp-initial-congestion-window-on-windows-2008-server-r2/ and within the TCP profile changing the Initial Congestion Window Size from the default of 0 (which actually means 4).

   

• Enable Proxy MSS client side

   

• Up the Initial Receive Window Size server side

   

• If client side is lossy/wireless, consider enabling D-SACK*

   

• Ensure Congestion Control is set to High Speed. If clients are on a lossy network, consider New Reno instead.

   

Let me know how you get on.

 

Thanks for the extensive feedback. VE should be able to handle such a load with ease.

This command may provide some insight;

more /var/log/tmm | grep UNIC

My v11.4.1 VE (Openstack on KVM on RHEL) returns this;

<13> Dec  3 11:54:18 host-192-xxx-xxx-xxx notice UNIC: set MTU for eth1 to 9000
<13> Dec  3 11:54:18 host-192-xxx-xxx-xxx notice UNIC: eth1 supports tx csum
<13> Dec  3 11:54:18 host-192-xxx-xxx-xxx notice UNIC: eth1 supports TSO
<13> Dec  3 11:54:18 host-192-xxx-xxx-xxx notice UNIC [un1] unic_attach(450): Hardware checksum offload is enabled
<13> Dec  3 11:54:18 host-192-xxx-xxx-xxx notice UNIC [un1] unic_attach(462): TCP segmentation offload is enabled
<13> Dec  3 11:54:18 host-192-xxx-xxx-xxx notice UNIC [un1] unic_attach(466): VLAN hardware checksum is disabled
<13> Dec  3 11:54:18 host-192-xxx-xxx-xxx notice UNIC [un1] unic_attach(470): VLAN TCP segmentation offload is disabled

I'm not sure what the last two lines might indicate and unfortunately can't test under load. Would be interesting to see what you get back on v11.3.

I think, until you get to v11.5 at least the only next logical step is to do a packet capture client and server side and try to identify where the delay actually is.

You may also want to consider these from the v11.4 and 11.5 release notes respectively;

 

ID 409234

 

"FastL4 Virtual Servers can experience very low throughput on Virtual Edition with TCP Segmentation Offload disabled. The customer will notice a large amount of Transmit Datagram Errors for the fastl4 profile (tmsh show ltm profile fastl4)" "The customer must be running Big-IP version 11.4 Virtual Edition with at least one fastL4 virtual server configured. The customer must additionally have TCP Segmentation Offload (TSO) disabled in the TMM (sys db tm.tcpsegmentationoffload). The customer may see low throughput numbers in this configuration if their hypervisor has Large Receive Offload (LRO) enabled. This is a hypervisor configuration and is beyond our control. The customer may also see these low throughput numbers when their Virtual Edition is passing traffic to other virtual machines running on the same physical hypervisor." FastL4 virtual servers affected will have very low throughput. "The customer should enable TCP Segmentation Offload by modifying 'sys db tm.tcpsegmentationoffload'. The customer may also resolve this issue by disabling large Receive Offload (LRO) on any hypervisor they plan on running Virtual Edition."

 

ID 455361

 

Fixed improper handling of ICMP (Internet Control Message Protocol) 'Fragmentation Required' messages from routers. Bug resulted in extremely inefficient behavior by BIG-IP TCP segmentation offload if path MTU (Maximum Transmission Unit) was smaller than what TCP endpoints negotiated.