LTM 11.3 with APM: smart card authentication not working

If I may add, the smartcard PIN prompt is a function of the client middleware that provides access to the card's cryptographic functions. This should happen before the client sends its certificate to the server, so if you're getting an error and never getting prompted, the likely culprit is the initial parts of the handshake, or perhaps before that at layer 4.

1. Post-TCP handshake, the client initiates the SSL session with a CLIENTHELLO
2. The server responds with SERVERHELLO and CERTIFICATE messages. The client must validate the server's certificate against a local trust (the browser's CA root store).
3. If mutual authentication is required, the server will also send a CERTIFICATEREQUEST message.
4. In order for the client to send its certificate back to the server, the middleware must request cryptographic services from the smartcard. The public certificate should already be available in the browser, but the message must be digitally signed by the private key on the card, which is a function of the smartcard chip itself.
5. The server must then validate the client's certificate against a local trust (the Trusted Certificate Authorities bundle in the client SSL profile).
6. If all of that works, the client and server will negotiate a session encryption key and begin the actual encrypted session.

If you run a WireShark or SSLDUMP capture you'll see this negotiation. If you apply the private key to either of these, you should also be able to see inside the encrypted session. In either, if the SSL handshake is failing, you'll see that in the capture.

ssldump -k [path to private key] -AdNn -i 0.0 port 443 [and any additional filters]

Now, what are you trying to deploy with APM? XenApp or XenDesktop? The configurations for each are actually very different. Do you want to remove the Web Interface and use the APM webtop, or do you need to keep WI?