Forum Discussion

DB's avatar
DB
Icon for Nimbostratus rankNimbostratus
Aug 30, 2010

Ideas on how to measure performance hit of SSL on a VIP?

Running LTM 9.3.1, users complain of response time problems for websites through it, as compared to hitting the website bypassing the F5. Lots of places to look, I know, but we use a tool called HTTPWatch and it points out that on a web page load that might have 50 objects, some of them (always gifs and jpgs), the "time chart" or breakdown of where the time was spent to download that item shows an unusually long time in the TCP Connect portion of the Get. HTTPWatch help says "Connect is the time required to create a TCP connection to the web server (or proxy). If a secure HTTPS connection is being used this time includes the SSL handshake process. Keep-Alive connections are often used to avoid the overhead of repeatedly connecting to the web server."

 

SSL Overhead is one part of this slice of time, and when my testers go around the F5 they're not hitting the server using SSL (and time to load a page is cut in half or better). I set up a test VIP to try and prevent SSL Offload, however some of the redirects on the page still go to HTTPS for a subset of GIFs (I haven't gotten Redirect Rewrite to rewrite all the Redirects correctly). The thing I notice, which drives this question, is that it's pretty consistent when there's a "[1 to 3 second] delay" in grabbing an object on the webpage, it's always when that object is at HTTPS://.... and never when the object is just HTTP://...

 

 

So I'm wondering if you have any thoughts on how to measure just how much overhead/delay SSL processing adds. It's not consistently on the same objects for each repeat of the same page loading, but there's always at least one or two objects SSL protected on the page that throw out one of these relatively long delays. Changing our website to not use SSL is not an option, but if our LTM is the bottleneck in it's ability to handle the SSL TPS (which according to the onboard Performance graph is <=20), I'd like to know that. If you look at the screenshot attached to this post, the picture really is worth a thousand words.

 

application delivery
design
performance
testing
  • Ray_Dodier_1102's avatar
    Ray_Dodier_1102
    Icon for Nimbostratus rankNimbostratus
    Sep 03, 2010
    Check out AOL_PageTest. It only works on IE browsers but it is free and breaks out the SSL time
  • Hamish's avatar
    Hamish
    Icon for Cirrocumulus rankCirrocumulus
    Sep 13, 2010
    I suspect you're not doing HTTP keepalives on your website.

     

     

    From your description, you're opening a new TCP connection (And thus a new SSL negotiation) for every object being hit on the web page, making one request and then closing it.

     

     

    To fix this, either enable HTTP keepalives in the webservers behind the F5, or if that's not possible, enable one connect and set a netmask of 255.255.255.255 on the oneconnect profile used (Make a custom one).

     

     

    Also make sure that you haven't limited the 'Maximum Requests' in the HTTP profile to 1... (Default is 0 == don't limit). And that your HTTP profile has one connect enabled (If you have to use it).

     

     

    Also, I'd suggest doing a manual GET against the web server with either CURL or openssl (That's really manual) and taking a tcpdump of the traffic at the time to determine where there's any lag in responses to the SSL negotiation (SSL negotiation is a sequence of several packets. Any delays in here would have an adverse affect on your performance too). Wireshark is a good tool for taking the tcpdump output and presenting it in a nice format for you to view. When you do take the tcpdump make sure you use a suitable filter so you don't have to try & find your connection in a sea of millions...

     

     

    H
  • JRahm's avatar
    JRahm
    Icon for Admin rankAdmin
    Sep 13, 2010
    You can use timers in curl to do this for you:

     

     

     

    jrahm@ubuntu:~$ curl -kso /dev/null -w "tcp handshake: %{time_connect}, ssl handshake: %{time_appconnect}\n" https://mail.google.com

     

    tcp handshake: 0.073, ssl handshake: 0.300

     

  • DB's avatar
    DB
    Icon for Nimbostratus rankNimbostratus
    Sep 13, 2010
    Thanks for the suggestions. I ran the AOL Test Page app from a service on the Web and it gave my site, VHA.COM an "A" for using Keep Alives. That was the only "A". I don't have access to a new enough version of cURL right now to get that time_appconnect variable but I'll work on that and let you know. That looks interesting. And I checked my HTTP profile: we're using the default which has Max Requests=0. Thanks again.
  • Hamish's avatar
    Hamish
    Icon for Cirrocumulus rankCirrocumulus
    Sep 13, 2010
    Can you get a tcpdump with curl?

     

     

    Then you can check the packet timings with wireshark.

     

     

    H
  • DB's avatar
    DB
    Icon for Nimbostratus rankNimbostratus
    Sep 13, 2010
    I did that (traces on inside and outside F5 interfaces, and just in front of a desktop client) when I first posted this. I found it extremely difficult to match up the traces and timings because the traffic is all encrypted (outside the F5 and desktop), but was able to determine that out of 5 instances I could find the "pattern" of packets and packet-sizes and line them up, the data was being delivered to the desktop sub-second, leaving the only place for the performance bottleneck to be in the PC and/or the Browser, or it's ability to render the page. I've been looking for a re-usable and bullet proof way of confirming that for more data samples: something I can take to my web developers to prove it's not in the network components which they're absolutely convinced it is. Yes, the classic Application <--> Network finger pointing match.
  • Ray_Dodier_1102's avatar
    Ray_Dodier_1102
    Icon for Nimbostratus rankNimbostratus
    Sep 27, 2010
    Not sure if this helps but I've been playing around with WireShark and found a few things that were helpful to me. The first was to get the TCP Stream Index numbers for streams of interest. Once I did the WireShark capture I'd enter the following WireShark filter to try to get the beginning of each sequence using -

     

     

    tcp.flags == 0x02 and ip.addr == 192.168.11.21 or ip.addr == 192.168.11.27

     

     

    This filters for the SYN flag and a given set of IPs

     

     

    After applying the filter I expanded the TCP section in the middle pane of WireShark and then could click on each line in the top pane to get a list of Stream Indexes to look at. Once I had that I used the following WireShark filter to pull out just the SSL handshake packets for a given stream (i.e. say stream 36) -

     

     

    tcp.stream eq 36 and ssl.handshake

     

     

    If you Google "WireShark tips Delta Time" there was one on adding Delta Time to the columns that I found helpful.

     

     

     

     

  • DB's avatar
    DB
    Icon for Nimbostratus rankNimbostratus
    Oct 14, 2010

    I was able to confirm that SSL was my problem. By setting up a seperate VIP that uses the same pool, but the VIP doesn't use SSL, I was able to close the gap between the time it takes to run through our F5 and Firewall as compared to going directly to a back-end server whereby the client and server are both inside our firewall. But of course, getting rid of SSL is not an option for our websites (It just explains the almost 100% performance degradation). This was all tested using a client machine that is internal to our network. Once I took that client to an offiste location and tested using a slow speed wireless connection at a resturaunt, SSL or no SSL makes no noticeable difference, because the propagation delay and bandwidth constraints cause page load times to increase 5-fold anyway, and the SSL delays become overshadowed by the network delays.

     

     

    Which brings me to a new twist to this tale I need your help with now. The websites in question offer content both to employees of my company (internal clients, who get to it via the same F5/firewall path as external clients, but at much higher "LAN" speeds), and it serves content to customers and member across the Internet. I'd like to experiment with LAN-Optimization and WAN-Optimization techniques, which are available on the [tcp] protocol profiles. Typically we use the default tcp profile. If I wanted to begin using a tuned profile for LAN or WAN users, can it be done with a single VIP and somehow select the profile based on the IP address of the client? It seems to me this is impossible because the TCP connection to the LTM is already established by the time I can run an iRule against it to select a certain protocol profile. If there is a way to do this and anyone can point me to an example, I'd appreciate it.

     

    Another question is on the "client" vs. "server" protocol profiles. By default the server profile in LTM uses whatever is specified as the client profile. Does it make sense, if trying to optimize for slow WAN connections, to use wan optimization profiles on the [LTM VIP Configuration Setting] "Protocol Profile (Client)" side which is facing the end user, and use a LAN optimized profile on the "Protocol Profile (Server)" side which is on the same subnet as an interface of the LTM, just one hop away at LAN speeds?