Citrix Federated Authentication Service Integration with APM

Double check that you completed every step in the section "StoreFront Config for SAML NetScaler Gateway" on Carl Stalhood's FAS setup guide that I linked above. Usually this is caused by missing a step. An alternative that I have seen is where the VIP is not accessible (or internal DNS doesn't point to the VIP) so the callback fails. Storefront logs will tell you this.

If you disable APM for the calls from Storefront as it sounds like you've done, then your callback will definitely fail because the BIG-IP doesn't know how to process the callback. This has been deployed widely and no disabling of APM is necessary for the callbacks (and would actually break it as noted).

If for some reason you can't allow Storefront to talk to the VS on an external interface, you can stand up another VS on the internal interface on a new IP and either use internal DNS or HOSTS file on the Storefront server to point the DNS name at that internal side VS. The VS should be setup exactly the same as the regular one, all the same profiles.

I have also seen this occur due to cipher selections, where custom high security cipher selections were used on the BIG-IP and the Storefront server could not negotiate them.