Troubleshooting Lync/Skype for Business Reverse Proxy

     At F5’s Agility conference, I spoke briefly about troubleshooting Lync/Skype for Business Reverse Proxy at the request of our support people.  Regardless if you deploy Lync/Skype via iApps or manually define your BIG-IP elements, there’s rumbling suspicion of witchcraft around how this works.  Well settle down there Salem.  Looking at support call metrics, we bundled a rather high percent of support cases into two specific issues; certificate deployment problems and misconfigurations with port redirection.  Deploying reverse proxy starting with simple configurations and working up to your more complicated requirements will significantly reduce headaches down the road.

Reverse Proxy’s Port Redirection

  At it’s core, the reverse proxy functionality for Lync/Skype is a simple public endpoint listening on port 80 and 443.  External connections are port redirected to internal front end pools, listening on 4443 and 8080 respectivly (if you don't redirect 80 to 443 as we do).  Services managed through reverse proxy are required for mobile, autodiscover, and extended collaboration services; presented via the following public DNS entries:

• Meet.yourdomain.com
• Dialin.yourdomain.com 
• Lyncwebservice.yourdomain.com (this name is up to you and added to topology builder during deployment) 
• Lyncautodiscover.yourdomain.com (required for mobile)

      When deploying reverse proxy services at F5 (and certifying our functionality for Microsoft), we simply CNAME’d the meet, dialin, and lyncautodiscover namespaces to the FQDN of the web service IP.  Fig. 1 shows the mobile client initiating two separate connection points into our Lync/Skype environment.  Primary client login is achieved via SIP registration through edge services (access:443).  The client also initiates a secondary connection to the web services namespace (lyncwebservices.fqdn listening on 443/80 in Fig. 1).  You could add authentication mechanisms and security features with ASM/APM, but when we certified BIG-IP for Lync Reverse Proxy, we achieved validation using a FastL4 virtual translating 443 to 4443 only.  Microsoft’s certification process ensures the device(s) responsible for reverse proxy functionality allow their services to work, any extra features added are secondary and up to the admin for ownership and support.  Remember, we’re starting simple to make sure we get it working first.  From here let’s build certs to get this listening on 443.

Building/Installing your Certificate

     We now have published public DNS records, all used by the reverse proxy, only one name requires a host record binding, the FQDN defined in topology builder for the external web services (in our example lyncwebservice.domain.com).  You’ll still add all other CNAME’s to the SAN certificate and Lync/Skype’s cert builder will configure this for your public and private domains.  I purchased my Entrust certificate and it included two bonuses, the CA root and intermediary chain certs; they're super important later.  Seeing that I use a fastL4 for the reverse proxy functionality, there’s no SSL termination/bridge required at the external LTM in Fig. 1 and instead I’ll bridge SSL at the internal front end pool virtual listening on 4443; again for simplicity and testing.  

     We also install this certificate chain on the external web services defined within the Lync/Skype installation wizard, allowing us to separate certs for internal and external features.  Microsoft makes this fairly easy so don’t try to deviate TOO far from recommended configurations.  If you want to do this on two separate certificates (one for web services and one for edge services) that’s fine, if you want to create one mega SAN cert, go for it.  Microsoft’s cert creation wizard will potentially add more names than you need, so just pay attention.

Places of Potential Failure (read: where everything blows up)

     The previously mentioned secondary connection our client makes to the Lync/Skype web services in Fig. 1 houses a potential hidden evil; A SECONDARY AUTHENTICATION!  Breath deep!  It’s ok.  The secondary authentication creates a client session token from a front end server via the web service's Certificate Provisioning Services published URI (a great write up on this is found here).  This authentication is critical to all reverse proxy web services and will cause very odd behavior in a failure scenario, I’ll give examples later.  To validate this URI, initiate a browser connection to https://lyncwebservices.yourdomain.com/CertProv/CertProvisioningService.svc and if it’s listening properly, you’ll get a login dialog.  It will accept your domain credentials and give you a basic landing page stating “You have created a service”.  If you can authenticate from a public connection, you’re most of the way there.  If this doesn’t work, check the following:

1. Are you SNATTING the reverse proxy (or is outbound traffic for web services is going back out via the same device; MSFT isn’t going to help with this)
2. Is the node for the reverse proxy pool is listening on 4443 (configure a monitor for this port because the same IP will listen for services on 443 and could cause false positive/negative results)
3. The perimeter LTM cannot reach the internal front end pool; time to check/blame the firewall if you have one separating internal and DMZ networks; Lync/Skype’s planning tool will show you all of the ports required between various entry points.
4. 443 isn’t listening on the public IP or your DNS is pointing to a black hole.
5. …. you get the idea.... basic pathway troubleshooting

     Start with networking basics;  NMAP, Dig/Host/NSLookup, and curl are your friends here.  It takes 5 minutes to run NMAP to validate listening ports, Dig public records to validate DNS is publishing proper namespaces, and curl to validate URL’s responses (even if auth prevents full HTTP connections, for that you could OpenSSL s_client).  Once you get the web service public URL responding, the next step is testing with a client, and here's where certificates have the potential to cause major headaches.  Lync/Skype is one of the few applications who's client logs are actually useful and will prevent hours of packet capturing at your LTM and front end servers.  Enable your client logs from within the app and locate them in the following locations:

• OSX: /Users//Library/Logs/Microsoft-Lync-0.log (OSX provides a generic log location for applications so there’ll be quite a few)
• Windows: C:\Users\\AppData\Local\Microsoft\Office\15.0 (or whatever)\Lync\Tracing\

Login to Lync/Skype and your newly generated logs will display a plethora of piñatas… a.k.a. client communication data.  Search for one of two things:  the FQDN of the published web services, OR the phrase WWW-Authenticate.  Both will get you to the web service authentication token creation as shown below:

WWW-Authenticate: NTLM realm="SIP Communications Service", targetname="feserver.fqdn", version=4
WWW-Authenticate: TLS-DSK realm="SIP Communications Service", targetname=“feserver.fqdn", version=4, sts-uri="https://lyncwebservices.yourdomain:443/CertProv/CertProvisioningService.svc"
WWW-Authenticate: TLS-DSK opaque="3AFFA71F", gssapi-data="FgMBDKsCAA ......AAA4AAAA=", targetname="feserver.fqdn", realm="SIP Communications Service", version=4
WWW-Authenticate: TLS-DSK opaque="3AFFA71F", gssapi-data=”XXXXXXX", targetname="feserver.fqdn", realm="SIP Communications Service", version=4