Very Slow Application performance behind F5

I know the issue's been solved, but I thought I'd share some things I've learned about Windows web server loadbalancing, NetBIOS, and Windows Integrated Authentication.

The behavior described above is absolutely the behavior of a Windows box that has its NetBIOS node-type set incorrectly and is set to use Integrated auth. What it's doing here is attempting to locate domain resources based on the incoming interface IP address. The system probably doesn't have a WINS server set (making it a b-node), or may have a WINS server set to which it can't talk. In both cases, it's either sending out a broadcast or a WINS query which isn't getting a response, and times out after 9 seconds (3x3). I run into this all the time will "multi-tier" Windows apps that have their Web front-ends in a firewalled DMZ.

It gets even worse when the system has Windows Integrated Auth enabled and can communicate with a WINS server but not the AD domain for which the WINS host is primary. The delay will be equal to 3 times the number of domain controllers that WINS reports are in the domain. In a big environment, this could take up to two minutes and stall the connection the whole time.

This can occur in two places: when the user connects and authenticates to the front-end, or when the front-end tries to authenticate to a WCF web service on the back-end. In both cases, the solution is the same: you must modify either the authentication order (Negotiate,NTLM becomes NTLM,Negotiate - http://support.microsoft.com/kb/215383) or disable Windows Integrated auth.

There's another little gotcha that can cause a connection to stall. If you've got WIA enabled and Negotiate mode comes up first as an authentication offer, most modern Windows systems will attempt to locate Kerberos services first, then fall back to NTLM. This works fine on an internal network, but not if you leave this setting in place when you move the app to be Internet-facing but choose (smartly) to leave your Kerberos servers unexposed. In a strange little twist, the Kerberos lookups usually last longer than the browser's TCP timeout, leading to a "Page cannot be displayed" error. The browser will try to find a domain Kerberos server, never failing over to NTLM as it should.

This iRule works to knock down the Negotiate header from being send to the browser, keeping this from occuring. It should be installed on the F5 that's handling incoming connections from the Internet.

when RULE_INIT {  
      set ::negotiate_rule_debug 0  
  }  
    
  when HTTP_REQUEST {  
       
      set negotiate_disable 0  
      if { [HTTP::header exists "Authorization"] } {  
          set negotiate_disable 0  
      } else {  
          set auth_host [string tolower [HTTP::host]]  
          set negotiate_disable 1  
      }  
  }  
    
  when HTTP_RESPONSE {  
    
       On 401 requests, if we've got the Negotiate holddown cookie, remove the WWW-Authenticate  
       headers for Negotiate and keep only the NTLM and basic auth ones headed to the client. The realm  
       header on the basic auth is set to the present hostname captured in HTTP_REQUEST.  
    
      if { ($negotiate_disable) && ([HTTP::status] == "401") } {  
          HTTP::header remove WWW-Authenticate  
          HTTP::header insert "WWW-Authenticate" "NTLM"  
          HTTP::header insert "WWW-Authenticate" "Basic realm=\"$auth_host\""  
          unset auth_host  
          unset negotiate_disable  
          if { ($::negotiate_rule_debug) } { log local0. "Replacing Negotiate for initial authentication." }  
      }  
  }