LTM 11.3 with APM: smart card authentication not working

You can continue to use the Web Interface here, but in order for CAC authentication to work through an SSL-terminating proxy (like the Netscaler or APM), you have to configure WI in "At Access Gateway" authentication mode, which is described in CTX124603. The APM configuration isn't much different, you point WI's gateway URL at an APM VIP instead of a Netscaler. There's more to it than that of course, but before we get into those details, I'd highly recommend the APM webtop approach. Not only does it simplify the architecture (removing the WI/gateway/STA components), but it's also a pretty simple config. The idea here is that you'll perform client side PKI, extract the usable identity information, and then do Kerberos SSO to the XML broker directly. In 11.4 and above, the SSO profile is assigned directly inside the Citrix Remote Desktop Resource agent. The native/direct authentication to the XML broker with Kerberos produces a LogonTicket in the ICA file that allows single sign-on to the app server.

Admittedly there's some confusion about the various authentication methods inside the Citrix Remote Desktop Resource agent, and how they affect the XenApp/XenDesktop environment, and really it boils down to smartcard PIN prompts. To achieve a single prompt experience, the XML broker in the XenApp environment must be IIS-integrated. It's a feature that you enable when you install the XML broker and a pain in the rear to enable after the fact. The Netscaler "At Access Gateway" method also relies on this because WI will actually perform Kerberos Protocol Transition and Constrained Delegation, on behalf of the user, to the XML broker to get that LogonTicket. Since you've made this work with the Netscaler, I'm assuming your XML brokers are indeed IIS-integrated. XenDesktop's DDC (XML broker) is not, and cannot be IIS-integrated, so you cannot pass it Kerberos tickets. For that reason, you'll never get a single PIN prompt to a XenDesktop environment - one prompt at the WI/APM, and one prompt at the VDI through an ICA secure channel. I say never, but depending on your smartcard middleware you could technically experience a single PIN prompt based on caching properties, but it isn't consistent. The smartcard auth option in the Citrix Remote Desktop Resource agent is intended for XenDesktop, where you'll do PKI authentication separately to the APM and to the VDI.

So in this case, for a XenApp environment, you'll want to use the Kerberos auth option in the agent and point it at your IIS-integrated XML brokers. The guide that comes with the latest iApp should clarify how all of this works, and give you some additional information on settings/policies/registry keys/delegation settings that have to be modified in Windows and in XenApp.

As for troubleshooting, there are certainly a few places that things can go wrong. If it's a client side SSL issue, the WireShark or SSLDUMP capture should illuminate that. Look for any handshake failure messages. If SSL is succeeding, then make sure 1) that traffic is making it to the XML broker, and 2) that Kerberos isn't failing. Here's another place that WireShark comes in handy. There's simply no better tool for troubleshooting Kerberos issues. Finally, because you mentioned CAC, the userPrincipalName in the DoD CAC card will not match the UPN of the user in your AD. APM Kerberos SSO is still missing a canonical referral extension, so you can't pass this value directly. You must first perform an LDAP/AD query (LDAP is faster) and retrieve the user's sAMAccountName from the directory, based on the UPN, and then supply that value to the Kerberos SSO as the username source.