LTM 11.3 with APM: smart card authentication not working

If you are not receiving a prompt for certificates then it might be a ca certificate issue on the BIG-IP. One of the questions in the iApp is "Which CA certificate bundle do you want to use for your trusted and advertised certificate authorities?". The certificate selected for this question needs to match the trusted CA that issued the certificates contained in your smartcard, only the client certificates issued by the selected CA will be prompted for a pin.

To verify, make sure BIG-IP apm logging is set to debug (note this is pretty verbose, so if you are in production use caution) and run tail -f /var/log/apm during a connection attempt. Look for "Session variable 'session.ssl.cert.exist' set to '1'" which will confirm a cert was never received (as you would suspect since a pin prompt to open smartcard was seen) and a look for a note similar to: Following rule 'fallback' from item 'Start' to item 'On-Demand Cert Auth'

If I may add, the smartcard PIN prompt is a function of the client middleware that provides access to the card's cryptographic functions. This should happen before the client sends its certificate to the server, so if you're getting an error and never getting prompted, the likely culprit is the initial parts of the handshake, or perhaps before that at layer 4.

1. Post-TCP handshake, the client initiates the SSL session with a CLIENTHELLO
2. The server responds with SERVERHELLO and CERTIFICATE messages. The client must validate the server's certificate against a local trust (the browser's CA root store).
3. If mutual authentication is required, the server will also send a CERTIFICATEREQUEST message.
4. In order for the client to send its certificate back to the server, the middleware must request cryptographic services from the smartcard. The public certificate should already be available in the browser, but the message must be digitally signed by the private key on the card, which is a function of the smartcard chip itself.
5. The server must then validate the client's certificate against a local trust (the Trusted Certificate Authorities bundle in the client SSL profile).
6. If all of that works, the client and server will negotiate a session encryption key and begin the actual encrypted session.

If you run a WireShark or SSLDUMP capture you'll see this negotiation. If you apply the private key to either of these, you should also be able to see inside the encrypted session. In either, if the SSL handshake is failing, you'll see that in the capture.

ssldump -k [path to private key] -AdNn -i 0.0 port 443 [and any additional filters]

Now, what are you trying to deploy with APM? XenApp or XenDesktop? The configurations for each are actually very different. Do you want to remove the Web Interface and use the APM webtop, or do you need to keep WI?

You can continue to use the Web Interface here, but in order for CAC authentication to work through an SSL-terminating proxy (like the Netscaler or APM), you have to configure WI in "At Access Gateway" authentication mode, which is described in CTX124603. The APM configuration isn't much different, you point WI's gateway URL at an APM VIP instead of a Netscaler. There's more to it than that of course, but before we get into those details, I'd highly recommend the APM webtop approach. Not only does it simplify the architecture (removing the WI/gateway/STA components), but it's also a pretty simple config. The idea here is that you'll perform client side PKI, extract the usable identity information, and then do Kerberos SSO to the XML broker directly. In 11.4 and above, the SSO profile is assigned directly inside the Citrix Remote Desktop Resource agent. The native/direct authentication to the XML broker with Kerberos produces a LogonTicket in the ICA file that allows single sign-on to the app server.

Admittedly there's some confusion about the various authentication methods inside the Citrix Remote Desktop Resource agent, and how they affect the XenApp/XenDesktop environment, and really it boils down to smartcard PIN prompts. To achieve a single prompt experience, the XML broker in the XenApp environment must be IIS-integrated. It's a feature that you enable when you install the XML broker and a pain in the rear to enable after the fact. The Netscaler "At Access Gateway" method also relies on this because WI will actually perform Kerberos Protocol Transition and Constrained Delegation, on behalf of the user, to the XML broker to get that LogonTicket. Since you've made this work with the Netscaler, I'm assuming your XML brokers are indeed IIS-integrated. XenDesktop's DDC (XML broker) is not, and cannot be IIS-integrated, so you cannot pass it Kerberos tickets. For that reason, you'll never get a single PIN prompt to a XenDesktop environment - one prompt at the WI/APM, and one prompt at the VDI through an ICA secure channel. I say never, but depending on your smartcard middleware you could technically experience a single PIN prompt based on caching properties, but it isn't consistent. The smartcard auth option in the Citrix Remote Desktop Resource agent is intended for XenDesktop, where you'll do PKI authentication separately to the APM and to the VDI.

So in this case, for a XenApp environment, you'll want to use the Kerberos auth option in the agent and point it at your IIS-integrated XML brokers. The guide that comes with the latest iApp should clarify how all of this works, and give you some additional information on settings/policies/registry keys/delegation settings that have to be modified in Windows and in XenApp.

As for troubleshooting, there are certainly a few places that things can go wrong. If it's a client side SSL issue, the WireShark or SSLDUMP capture should illuminate that. Look for any handshake failure messages. If SSL is succeeding, then make sure 1) that traffic is making it to the XML broker, and 2) that Kerberos isn't failing. Here's another place that WireShark comes in handy. There's simply no better tool for troubleshooting Kerberos issues. Finally, because you mentioned CAC, the userPrincipalName in the DoD CAC card will not match the UPN of the user in your AD. APM Kerberos SSO is still missing a canonical referral extension, so you can't pass this value directly. You must first perform an LDAP/AD query (LDAP is faster) and retrieve the user's sAMAccountName from the directory, based on the UPN, and then supply that value to the Kerberos SSO as the username source.

Wow. I am incredibly impressed with the insightful F5 DevCentral community. I wasn't expecting such an in-depth treatment of the processes involved in smartcard authentication integration with Citrix XenApp here. Very well done, Kevin (and Greg). I'm new to F5 DevCentral and I'll be on the lookout for your future contributions. And, hopefully my pain in this instance can be someone else's gain.

Now, I follow everything except the very last sentence. We've got an APM AAA Server object for our first AD domain, but I don't know how that will "retrieve the user's sAMAccountName" from AD and supply it to the Kerberos SSO. With NetScalers, we create an "Authentication Server" for every domain where we perform authentication (smart card or otherwise). So, if there are 30 different AD domains, we would need at least 30 unique "Authentication Server" objects. We associate those "Authentication Servers" with "Authentication Policy" objects, which we bind to our Access Gateway Virtual Server. For this account, there are 7 different domains that have been identified so far. I wanted to get the first one completed and then add the others once the first has been tested successfully. Now, I think we've already configured what you've recommended with our current LTM/APM configuration. And that wasn't entirely my doing: it was the latest iApp template and the granular instructions in the Deployment Guide for Citrix, but we'll see. I'll let everyone know how it goes on Monday.

Very insightful response. Thanks, much-

First thing to understand is that there's a significant departure from how you configure Netscaler here and how you use APM with a webtop to remove those components. The Kerberos pieces in WI are implicit and not something that you have to mess with. With the APM webtop approach though, you actually have to configure the Kerberos. The linchpin in this process, when dealing with DoD CACS, is that the realm in the SAN UPN on the card (@mil) is never going to match the realm of your AD domain name. Canonical referrals is an extension to Kerberos that is often seen in Exchange environments. You have multiple domains that a user can belong to (ex. dom1.example.com, dom2.example.com, dom3.example.com), but all users use an alias email address (bob.user@example.com). For email that's usually just the mail value in the user's attributes. For desktop smartcard logon, you have to apply an alternate UPN suffix to the domain to allow the @mil UPN realm. When a Kerberos client/service needs to authenticate itself, or a delegated user to the forest, the alias name doesn't map to anything, so the service will make a canonical enterprise referral request to its local KDC with the principal name. The local KDC will then query, either locally or via the global catalog, where that user really exists, and then return a referral to the correct realm for the client to follow. Sort of like a DNS CNAME for Kerberos. APM, as of 11.5, still doesn't support this extension, so you have to perform the query yourself and specify the real realm in the ticket request. And because the UPN realm doesn't match the real realm, the request would confuse the KDC. Luckily, a Windows DC can consume a few different forms of the user identity, so we'll use the cert UPN to make an LDAP query into the cd and fetch the user's sAMAccountName. We'll then use that returned SAM as the username source input to the Kerberos ticket request.

1. Set up an LDAP AAA.
2. Prompt the user for their certificate (On-Demand cert auth agent in the APM VPE or client SSL profile).
3. Use an iRule or branch rule in the VPE to extract the SAN UPN from the x509 extensions section of the cert and assign it to a variable (ex. session.ssl.cert.upn).

4. Prepare your LDAP query to search the AD for the userPrincipalName:

   userPrincipalName=%{session.ssl.cert.upn}        
   

5. Set the branch condition of that LDAP query to something reasonable. Could be as simple as an LDAP query passed result.

6. If the query succeeds, there should be a value returned for the SAM:

   session.ldap.last.attr.sAMAccountName        
   

7. You can use that session variable directly inside the username source value of the Kerberos SSO profile.

The UPN in th AD must of course match this value, so you'll need to set the @mil alternate UPN suffix, or potentially query another value in the AD to return the correct SAM.

Now, if you have multiple domains, you can perform cross-domain authentication as long as:

1. APM can directly resolve and communicate with the KDCs of each
2. There is a two-way trust between the domains. Kerberos protocol transition requires it.
3. The AD delegation account that you define in the Kerberos SSO is in the SAME domain as the resource (the XenApp servers). Kerberos constrained delegation requires this.
4. You can figure out and specify which domain the user actually belongs to. If this a forest trust, then you should be able to LDAP query the global catalog directly. Otherwise you have to query each domain successive to find the user. I'm hoping canonical referral support will come along soon to alleviate this process. Once you know which domain the user is from, assign that to the realm source variable in the Kerberos SSO (ie. session.logon.last.domain), and of course also assign the sAMAccountName to the username source variable.

I should also mention the following. In 11.4 and higher, the Kerberos SSO is defined directly inside the Citrix Remote Desktop resource agent. In 11.3 you would need to point the resource agent at a VIP that load balances the XML broker(s) and applies an APM Kerberos SSO. Because the APM for the client side PKI and APM for the server side Kerberos are in the same memory space, you can assign the SSO source session variables in the first and consume them in the second.

But I don't know how to use the results from the "userPrincipalName=%{session.ssl.cert.upn}" query to put that variable directly inside the username source value of the Kerberos SSO profile

Simply assign this session variable (session.ssl.cert.upn) as the Kerberos SSO username source value. But wait, you're not going to be able to use the userPrincipalName (SAN UPN) from the CAC directly. You must query the AD (LDAP or AD query) and retrieve the user's sAMAccountName and use THAT value in the username source field of the Kerberos SSO. You can test Kerberos SSO directly simply by adding a variable assignment at the very end of the policy, just before the Allow block, and statically assign a value to the username source variable:

session.ssl.cert.upn = expr { "bob.user" }

Now we are getting a little further and it's allowing us to "add item to workQueue". But it is giving: Kerberos: Failed to resolve IP address ::ffff:122.22.22.22

When we look at our Virtual servers and Pools and members, these show "Up". Why would Kerberos send us to an IP address that we can't resolve (even though we should be able to resolve it). And what is the "ffff" doing in the IP address?

Well, the value you're seeing is an IPv4-translated IPv6 address, so it's left-padded with "ffff". But aside from that, Kerberos uses names, service principal names (SPNs) to be exact. So when APM needs a SPN, it takes the IP address that's been given to it, usually a pool member, and performs a reverse DNS lookup into the domain to get the host name of the server. It then takes that name, let's say for example:

xmlbroker1.mydomain.local    

It then adds the service type and domain realm to make a valid SPN:

http/xmlbroker1.mydomain.local@MYDOMAIN.LOCAL    

I think what you're experiencing here is that the address you're giving to APM doesn't resolve to a name, specifically the host name of the XML broker.