Forum Discussion
brad_11440
Nimbostratus
NimbostratusFastL4 vs Optimized TCP Profiles
We run active/standby 6400 (platform D63a) LTM's for our production applications. They are accessed via clients on a private network with an average latency around 80ms. Nearly all of our virtual servers are using the default standard TCP profile (nagle's, bandwidth delay, low send/receive windows and proxy buffers). We've recently implemented an end-to-end response time analysis solution and have observed very high server response times for those applications that utilize the LTM.
We want to improve performance and obviously switching to either FastL4 or TCP LAN Optimized profile would do just that. But which one would be better ? I think taking advantage of the two TCP stacks would be a benefit but I am concerned with the increased CPU/memory utilization this would cause. FastL4 would put more burden on the server itself and I am not sure if that is a good thing or not. For the most part, we do not use any extensive iRules (only a simple SNAT rule), so both would be an option. Has anyone done a comparison between the two ?
Any tips are appreciated. Thanks in advance!
- Maybe it's about the right time to have you post your VIP config from the bigip.conf. It's possible you have other settings applied to the VIP which conflict with a FastL4 profile??? Also it might be helpful to paste the values of the TCP profile you have applied.
Regarding the window size...the behavior you see was bothering me for a long time until it finally clicked in my brain. Yes, the TCP window size always starts at 4380. The trick is to understand whether the LTM is sending or receiving data. When the LTM is sending data, as in this case, there is no need for the LTM to increase its receive window - which is the parameter you refer to in the trace.
When the LTM is receiving large amounts of data and you do a trace, you'll see that the LTM receive window starts out at 4380, and quickly increases to the value of the Receive Window as specified in the TCP profile that is applied.
The LTM is smart - it starts out with a small window size so it does not need to allocate more memory than necessary for the TCP connection. If it needs to adjust the receive window (when it is receiving data), you'll see it go up.
21 Replies
- JRahm
Admin
I agree with smp, fastL4 will be better unless you need the features above tcp. You probably won't see much impact on the servers, though if the tcp overhead becomes a problem on the server side, you could implement oneconnect with the optimized tcp profiles, and that will reuse connections to your servers so that the high cost of setup and teardown is eliminated. There are a lot of ways to skin the cat and plenty of nuances in every environment that there isn't a one size fits all approach, and the functionality is there exactly for this reason. 
brad_11440Well, we made the change this morning. For reasons we are still investigating, after moving to FastL4, the VIP went down and the application stopped working. It did not come back up until we switched over to TCP-LAN-Optimized on both sides (did not try reverting back to the default standard).
After moving to this optimized profile, we've seen roughly a 100ms decrease in Data Transfer Time (the time between the first reply sent by the server and the last packet sent to fufill the request). The server response time has dropped by about 30ms. I was expecting this to drop further but it's probably due to the application being poorly written.
One question I do have is in regards to the Server TCP window size. I ran a packet trace after the change and when the server responds, the initial TCP window size is always 4380. The client's initial request is always at 65535. By using the TCP Lan Optimized profile, the F5 is acting as a proxy and negotiating the TCP parameters, correct ? I was under the impression that when others here have made this same change, they noticed the TCP window size opened up on the server side. Any ideas ?
smp_86112Cirrostratus
Maybe it's about the right time to have you post your VIP config from the bigip.conf. It's possible you have other settings applied to the VIP which conflict with a FastL4 profile??? Also it might be helpful to paste the values of the TCP profile you have applied.
Regarding the window size...the behavior you see was bothering me for a long time until it finally clicked in my brain. Yes, the TCP window size always starts at 4380. The trick is to understand whether the LTM is sending or receiving data. When the LTM is sending data, as in this case, there is no need for the LTM to increase its receive window - which is the parameter you refer to in the trace.
When the LTM is receiving large amounts of data and you do a trace, you'll see that the LTM receive window starts out at 4380, and quickly increases to the value of the Receive Window as specified in the TCP profile that is applied.
The LTM is smart - it starts out with a small window size so it does not need to allocate more memory than necessary for the TCP connection. If it needs to adjust the receive window (when it is receiving data), you'll see it go up.- brad_11440Here is the config. Thanks for clarifying the window size issue. That makes perfect sense.
profile fastL4 fastL4 { reset on timeout enable reassemble fragments disable idle timeout 300 max segment override 0 pva acceleration full } pool wmtis03ssl { snat disable nat disable monitor all tcp member 10.212.xxx.xxx:5553 session disable member 10.212.xxx.xxx:40202 member 10.212.yyy.yyy:5553 session disable member 10.212.yyy.yyy:40202 } pool wmtis03ssl_SNAT { monitor all tcp member 10.212.xxx.xxx:5553 session disable member 10.212.xxx.xxx:40202 member 10.212.yyy.yyy:5553 session disable member 10.212.yyy.yyy:40202 monitor tcp } rule wmtis03ssl_Irule { when CLIENT_ACCEPTED { if { [IP::addr [IP::client_addr] equals 10.212.xxx.0/255.255.254.0] } { pool wmtis03ssl_SNAT } else { pool wmtis03ssl } } } virtual wmtis03ssl_VIP { destination 10.212.aaa.aaa:5553 ip protocol tcp pool wmtis03ssl rule wmtis03ssl_Irule } profile tcp tcp-lan-optimized { defaults from tcp slow start disable bandwidth delay disable nagle disable proxy buffer low 98304 proxy buffer high 131072 send buffer 65535 recv window 65535 } 
Josh_41258Brad, a bit off topic, but would you mind sharing how you are conducting these response time tests? Thanks!
Josh- brad_11440
Posted By jbaird on 01/20/2009 8:03 PM
Brad, a bit off topic, but would you mind sharing how you are conducting these response time tests? Thanks!
Josh
I am using a product by NetQos called Superagent. It measures end to end application response time by passively monitoring TCP traffic and calculating metrics based off of the header information. You specify servers, application ports and networks. Right now since we are monitoring in front of where our LTM sits on the network, many of the servers are the VIPs themselves. It's nice being able to determine the exact performance gains when making a change like this.
I've attached screenshots off the application performance before and after the profile change was made. An hour worth of data today and then an hour last Wednesday. - Josh_41258Looks nice. Do you know of any other (free) products similar to this?
- JRahmThe upper bound for TCP's initial window is represented by this formula:
min (4*MSS, max (2*MSS, 4380 bytes))
4380, by the way, is 3*MSS (1500 MTU - 20 IP - 20 TCP = 1460, which *3 = 4380. - JRahmAnd smp is absolutely spot-on on the clientside of LTM regarding window size, client is not sending much except a request, so the LTM stack has no reason to increase the window. If you look at the flow on the serverside, in this perspective, the server is the client and will need to increase the window to maximize efficiency in receiving the data from the server.
- smp_86112Thanks for the details on the initial window.
So...back to the original question...given the config info provided, can anyone see why flipping the VS to FastL4 broke his app? I don't have much experience with these types of Virtual Servers, so I don't see what broke. I wonder if the LTM can't process the iRule, but the VS looks like it has a default pool so not sure if that's what the problem was.
