APM KCD SSO - Requesting ticket can't get forwardable tickets (-1765328163) but works eventually

Can you by chance watch the Kerberos traffic between APM and the DC? Capture with tcpdump and load the pcap into Wireshark. I'm wondering if APM is contacting the same KDC every time, or maybe the KDC isn't returning good information.

Kevin - we will be wiresharking the traffic this afternoon - hopefully it sheds some light on the issue.

Kevin, What we are seeing in wireshark on the DC is the TGT request and TGS request completing without issue. The F5 is requesting a forwardable ticket per the option fields. We see no KRB errors.

On the APM WEBSSO set to debug - we see the first

Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: ssoMethod: kerberos usernameSource: session.ldap.last.attr.sAMAccountName userRealmSource: session.logon.last.domain Realm: DOMAIN.NAME KDC: 10.10.227.54 AccountName: HOST/serviceacct spnPatterh: HTTP/somesite.com@DOMAIN.NAME TicketLifetime: 600 UseClientcert: 0 SendAuthorization: 0
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: ctx: 0x5a905598, CLIENT: TMEVT_REQUEST
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: ctx: 0x5a905598, CLIENT: TMEVT_REQUEST_DONE
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: ctx: 0x5a905598, CLIENT: TMEVT_SESSION_RESULT
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: ctx: 0x5a905598, CLIENT: TMEVT_SESSION_RESULT
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: ctx: 0x5a905598, CLIENT: TMEVT_SESSION_RESULT
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: ctx: 0x5a907250, SERVER: TMEVT_REQUEST
Oct  4 14:46:25 F5-APM info websso.1[14917]: 014d0011:6: 4e45c27d: Websso Kerberos authentication for user 'eric.haupt1' using config '/Common/Kerberos_SSO'
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0046:7: 4e45c27d: adding item to WorkQueue
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0021:7: sid:4e45c27d ctx:0x5a905598 SPN = HTTP/somesite.com@DOMAIN.NAME
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0023:7: S4U ======> ctx: 4e45c27d, sid: 0x5a905598, user: eric.haupt1@DOMAIN.NAME, SPN: HTTP/somesite.com@DOMAIN.NAME
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: Getting UCC:eric.haupt1@DOMAIN.NAME@DOMAIN.NAME, lifetime:36000
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: fetched new TGT, total active TGTs:1
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: TGT: client=HOST/serviceacct@DOMAIN.NAME server=krbtgt/DOMAIN.NAME@DOMAIN.NAME expiration=Fri Oct  5 00:46:25 2018  flags=40610000
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: TGT expires:1538714785 CC count:0
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: Initialized UCC:eric.haupt1@DOMAIN.NAME@DOMAIN.NAME, lifetime:36000 kcc:0x5a907aa0
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: UCCmap.size = 1, UCClist.size = 1
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: S4U ======> - NO cached S4U2Proxy ticket for user: eric.haupt1@DOMAIN.NAME server: HTTP/somesite.com@DOMAIN.NAME - trying to fetch
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: S4U ======> - NO cached S4U2Self ticket for user: eric.haupt1@DOMAIN.NAME - trying to fetch
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: S4U ======> - fetched S4U2Self ticket for user: eric.haupt1@DOMAIN.NAME
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: S4U ======> trying to fetch S4U2Proxy ticket for user: eric.haupt1@DOMAIN.NAME server: HTTP/somesite.com@DOMAIN.NAME
Oct  4 14:46:25 F5-APM err websso.1[14917]: 014d0005:3: Kerberos: can't get S4U2Proxy ticket for server HTTP/somesite.com@DOMAIN.NAME - Requesting ticket can't get forwardable tickets (-1765328163)
Oct  4 14:46:25 F5-APM err websso.1[14917]: 014d0024:3: 4e45c27d: Kerberos: Failed to get ticket for user eric.haupt1@DOMAIN.NAME
Oct  4 14:46:25 F5-APM debug websso.1[14917]: 014d0001:7: ctx: 0x5a907250, SERVER: TMEVT_NOTIFY
Oct  4 14:46:25 F5-APM err websso.1[14917]: 014d0048:3: 4e45c27d: failure occurred when processing the work item

Then right after that - it seems the the F5 then finds the newly cached ticket and processes properly. I won't post debug, but we can trace quite a few GETs with some of them showing S4U===>OK and others seeming like they have no ticket and must request one. Our SSO config is set to be pretty vanilla- send auth always 600 timeout. We are on 11.6.1 HF2 - getting ready to go to 13.1.1.

Within the SSO profile - I've defined the KDC by IP to rule out any DNS issues. This is not our primary Domain/Realm. This is a tenant Domain with the F5 fronting a single webservice. The realm and KDC options have also been defined as a separate realm within /etc/krb5.conf and of course the SPNs and the service accounts are specific to this domain logic.

I should add that the FQDN for the service SPN is not part of the back-end domain... but this shouldn't matter I don't think, as long as the service account in the application pool on IIS is the SPN holder and part of the Realm I'm operating in with my service account with delegation to that SPN.

Eric,

I've been staring at this for a while and this just caught my eye, "I should add that the FQDN for the service SPN is not part of the back-end domain... but this shouldn't matter I don't think". It actually does matter. The APM SSO service account MUST be in the same realm as the target service. Users don't have to be in the same realm, but these account must be.

Kevin, So: if the realm is "internal.com" but the user hit the FQDN app1.external.com and the SPN in the @INTERNAL.COM" REALM is HTTP/app1.external.com and the F5 service account is delegated as a host/ for the service SPN

Then the APM SSO service account should be in @INTERNAL.COM? Because it is. The SPN defined for the external service is the only object of the KCD transaction that doesn't define the internal realm in any way.

Yes, the APM SSO service account and the target resource must always be in the same realm.

The "Requesting ticket can't get forwardable tickets" error will generally happen if either delegation settings are misconfigured, or there's a duplicate SPN on an endpoint service. Have you checked for duplicate SPNs?

setspn -X

Getting different error now... the user variables are valid... hrmm. the ticket being requested looks valid. I have an SSO credential object at the end of my policy assigning sAMA... My SSO profile is currently set for explicit IP to KDC with service SPN explicitly defined. The service SPN is not duplicated... this is a small tenant domain... with just one app running.

    Oct  9 10:47:19 hostpxnapm13 info websso.1[20055]: 014d0011:6: 27b40f75: Websso Kerberos authentication for user 'eric.haupt1' using config '/Common/ACCS_Kerberos-app1.domain.com'
Oct  9 10:47:19 hostpxnapm13 debug websso.1[20055]: 014d0046:7: 27b40f75: adding item to WorkQueue
Oct  9 10:47:19 hostpxnapm13 debug websso.1[20055]: 014d0021:7: sid:27b40f75 ctx:0x5ab013f0 SPN = HTTP/app1.domain.com@INT.REALM
Oct  9 10:47:19 hostpxnapm13 info websso.1[20055]: 014d0022:6: 27b40f75: Kerberos: realm for user eric.haupt1 is not set, using server's realm INT.REALM
Oct  9 10:47:19 hostpxnapm13 debug websso.1[20055]: 014d0023:7: S4U ======> ctx: 27b40f75, sid: 0x5ab013f0, user: eric.haupt1@INT.REALM, SPN: HTTP/app1.domain.com@INT.REALM
Oct  9 10:47:19 hostpxnapm13 debug websso.1[20055]: 014d0001:7: Getting UCC:eric.haupt1@INT.REALM@INT.REALM, lifetime:36000
Oct  9 10:47:19 hostpxnapm13 debug websso.1[20055]: 014d0001:7: Found UCC:eric.haupt1@INT.REALM@INT.REALM, lifetime:36000 left:32140
Oct  9 10:47:19 hostpxnapm13 debug websso.1[20055]: 014d0001:7: UCCmap.size = 1, UCClist.size = 1
Oct  9 10:47:19 hostpxnapm13 debug websso.1[20055]: 014d0001:7: S4U ======> - NO cached S4U2Proxy ticket for user: eric.haupt1@INT.REALM server: HTTP/app1.domain.com@INT.REALM - trying to fetch
Oct  9 10:47:19 hostpxnapm13 debug websso.1[20055]: 014d0001:7: S4U ======> - NO cached S4U2Self ticket for user: eric.haupt1@INT.REALM - trying to fetch
Oct  9 10:47:20 hostpxnapm13 err websso.1[20055]: 014d0005:3: Kerberos: can't get S4U2Self ticket for user eric.haupt1@INT.REALM - Generic error (see e-text) (-1765328324)
Oct  9 10:47:20 hostpxnapm13 err websso.1[20055]: 014d0024:3: 27b40f75: Kerberos: Failed to get ticket for user eric.haupt1@INT.REALM
Oct  9 10:47:20 hostpxnapm13 err websso.1[20055]: 014d0048:3: 27b40f75: failure occurred when processing the work item

Don't use an SSO Credential Mapping agent for Kerberos SSO. You don't need it. The SSO profile has two session variable inputs, session.sso.token.last.username, and session.logon.last.domain. You simply need to make sure these session variables are populated before the end of the policy, and the domain variable is usually statically set.

session.logon.last.domain = expr { "INTERNAL.COM" }

And your username variable can either be the sAMAccountName (preferred) or UPN.

session.sso.token.last.username = expr { "bob" }

In fact you can isolate SSO for testing by simply assigning these values statically in the VPE.

Okay, so some additional thoughts.

• In your APM Kerberos SSO, ensure that the Send Authorization setting is set to "Always" for Microsoft services. For anything else it depends on the version of Kerberos they use (MITv5 or SPNEGO).

• Make sure time is good between APM and the KDC and target server.

• Set a static SPN in the APM SSO and disable all but one pool member at a time to make sure it's just not one of the servers having an issue.

• Expand your Wireshark inspection to include DNS traffic between APM and the DC.

  kerberos or dns or http
  

• The KRB_ERR_GENERIC message can happen if,

  • If UDP is fragmented and/or TCP is selected and the PAC data is overloading the ticket. This usually happens if the user is a member of many groups.
  • The SPN is too long or has too many parts. Try using your SPN to something simpler, like "HOST/krbsso.internal.com"

• If you're using a domain user account as the IIS application pool owner, you need to disable Windows Integrated Authentication kernel mode.

• To clear the Kerberos SSO cache between tests:

  bigstart restart websso
  

• To add debug logging (remember to turn this off when you're done):

  tmsh modify sys db log.sso.level value debug
  

• For even more debug logging:

  export KRB5_TRACE=/tmp/krb5trace
  tail -f /tmp/krb5trace