kerberos and ntlm authentication using APM

One of the beauties of APM's authentication FULL proxy is its native distinction between client side and server side authentication functions. While you can do client side certificate to server side Kerberos, the only things that Kerberos really needs is a valid domain username and realm, and it doesn't matter how it gets those. That said, you present a few (doable) challenges in your request that should be addressed separately.

 

 

Configuring server side Kerberos SSO

 

As I mentioned before, Kerberos SSO only needs a valid username and domain realm to authenticate a user to a service. There are, however, a few other things that you have to configure to get Kerberos SSO working:

 

 

1. Kerberos SSO is a protocol transition and constrained delegation mechanism, and as it isn't a proper member of the domain, it needs an account in the domain to do its bidding. Create a standard user account in the AD. In the User Logon Name field, create an arbitrary servicePrincipalName. This account will be used to delegate to other services and get tickets on behalf of users, so it needs to have an SPN assigned (user accounts don't normally have SPNs). An SPN is a string that represents a service and a name, so for example "host/krbsvc.example.com", where "host/" is the service and "krbsvc.example.com" is the object's name. This SPN must be unique in the domain, so it's best to create one from scratch. The User Logon (pre-windows 2000) name isn't important. Once the account is created, open up ADSIEDIT.msc or the advanced view in Active Directory Users and Computers. In ADSIEDIT, find the account and option its properties. In ADUC, open the account and go to the Attribute Editor tab. In both, find the (blank) entry for servicePrincipalName and enter the SAME SPN that you entered for the User Logon name (ex. host/krbsvc.example.com). It's not required, but probably best practice to give the SPN a fully qualified name in your domain. You can also create the SPN with the SETSPN command. Close and re-open the account properties and you'll now see a Delegation tab. Go to that tab, select "Trust this user for delegation to specified services only" and "Use any authentication protocol". This last option enabled protocol transition. Now click the Add button, find the servers that you'll be providing SSO to, and then select their http/ SPNs. So if you have 10 web servers in a BIG-IP pool with Kerberos SSO applied, you'll need to configure this account to be able to delegate to the SPNs of those web servers.

 

 

2. Create your Kerberos SSO profile. Enter the Kerberos Realm, your domain name in all upper case. Then enter the SPN that you created for the domain service account in the Account Name field. In this case however, enter the SPN with the realm (ex. host/krbsvc.example.com@EXAMPLE.COM). This will allow cross-domain authentication to work if you ever need that. Set the account's password. Now here's where it gets a little tricky. Depending on your environment, your web servers will be "owned" by different entities. I'm not going to dive into the complexities of the Kerberos protocol here, but to say that Kerberos is governed by encryption keys that are tied to SPNs. If you're using IIS, the application pool for a given site will be owned by someone or something. If you've not touched it, then by default the application pool will be owned by the host server and the SPN will be "http/". If however you've configured the application pool to be owned by another entity, a user account perhaps, then that account needs an SPN assigned to it for Kerberos authentication to work. This is all very important because of the way APM load balances Kerberos-enabled servers. When the LTM chooses a server from a pool, APM performs a reverse DNS lookup to get the name of the server based on that IP. It takes that name, adds "http/" to the front and "@" to the back to derive an SPN. So if the IIS application pool isn't owned by the server, the SPN Pattern field in the Kerberos SSO profile allows you to "short circuit" this process. One other interesting caveat (in 11.3) is that you can't put an explicit string in this field (ex. http/mywebservers.example.com@EXAMPLE.COM). You need to create a hosts entry on the BIG-IP (under System) that points every server IP in the pool to this static name, then use the %s string replacement in your SPN pattern (ex. htt/%s@EXAMPLE.COM). The reverse DNS lookup still happens, but the Hosts entry will trump any real data and return the desired name (ex. mywebservers.example.com). One final important point when configuring the Kerberos SSO profile. In the Credentials Source section there are two values, Username Source and User Realm Source. These are the APM session values that the Kerberos SSO will use to perform authentication. Somewhere in your access policy you must ensure that these values are set to a valid username and realm name for the SSO to consume. That can come from a client certificate, logon page, LDAP/AD query, or anywhere at all.

 

 

3. Ensure that the BIG-IP's time is synchronized with the domain. Kerberos is very picky about time skew. Ensure that BIG-IP is configured with the domain's DNS. The BIG-IP should be able to perform forward and reverse DNS entries for objects in the domain.

 

 

This is really all there is to server side Kerberos SSO. If you run into problems, check your SPNs and install Wireshark on the DC (if possible) to capture the Kerberos traffic for troubleshooting.

 

 

 

Failing over between server side Kerberos and NTLM SSOs

 

This one is interesting, especially if you consider that the SSO methods are generally preemptive (they don't wait for a 401 to pass credentials). You can certainly switch SSO methods via iRule, but the condition by which you decide that is the tough bit. You could, for instance, switch to NTLM if the user entered a valid username and password, and do Kerberos if the user only entered a username (and some other auth vector like RSA token or smartcard), but it would be less trivial (though not impossible) to switch from Kerberos to NTLM in the event of a 401 response. I'd recommend a reconsideration of your requirements and use cases before attempting this.

 

 

 

Passing client side Kerberos through if it exists

 

Again, without digging into the depths of the Kerberos protocol, authentication is based upon encryption keys and associated SPNs (names). A unique SPN is associated with a unique encryption key. So when a client requests a resource within a domain, a resource that requires Kerberos authentication, the client will go to the domain controller (KDC) and request a ticket to that service BY NAME. That ticket will have information about both the client and service and be wrapped in encryption keys owned by user and services. If you ask for a service by the wrong name, the KDC will encrypt it with the wrong key and the service won't be able to decrypt it. Now when you're accessing a resource through a proxy, you're typically requesting it by some name that is associated with the IP address on the proxy. So if the web server's SPN is, for example, http/server1.example.com, and the client is accessing a proxy using the name http://www.example.com, the client would ask the KDC for a ticket to http/www.example.com. If the proxy just passed that traffic through to the server, then the server wouldn't be able to do anything with it (wrong name = wrong encryption key). So you have two options:

 

 

1. Make the name that the client requests, as associated with the IP on the proxy server, the same SPN that owns the web server's application pool. You'll necessarily need to assign the application pool to a domain (user) account a) because you'll need to be able to load balance and you're only making a ticket request for ONE name, and b) the client may know how to get to server1.example.com directly and completely bypass the proxy. To your request, the implication here is that you'd have to sniff the initial request for an Authorization header and bypass the logon page in APM.

 

 

2. Proxy the client side Kerberos. This is a completely different Kerberos configuration that I won't dive into, but no more complicated than the first. To your request, the implication here is roughly the same in that you have to sniff for the Authorization header and bypass the 401 and Kerberos Auth agents and go to a logon page instead.

Kevin,

 

Hi, thanks for the response, it will take me a wee while to absorb all that you are saying, however, one of the areas where I am really struggling is with the "access policy" editor. Here there are many choices, and truthfully, I am not sure that I have enough information to make an informed decision. Here is my current access policy...

 

 

Start---->fallback---->[HTTP 401 Response]---->basic---->[Deny]

 

[HTTP 401 Response]---->negotiate---->[Allow]

 

[HTTP 401 Response]---->fallback---->[Logon Page]---->fallback---->[AD Auth]---->successful---->[SSO Credential Mapping]---->fallback---->[Allow]

 

[AD Auth]---->fallback---->[Deny]

 

 

From the Logon page, going forward, I have a working setup with NTLMv1 authentication. Chosen at the SSO credential mapping stage, but I am really unsure, if I want to put an HTTP 401 response object before this to handle Kerberos(just seems wrong). I have put this there, but I am pretty sure it is not handling kerberos at all, as my policy is still using NTLMv1.

 

 

It is really all the predefined actions, that are availble when you access the plus sign in the access policy editor, that to me have very little or poor documentation(it is very probably they do have good documentation, and I don't know where to find it) - I have copied the object list below, but I am really struggling to know what each one does, and what a typical scenario might be for kerberos(for example to I need to define a Kerberos AAA server object or not?). I appreciate that all this falls between two stools, F5 and Microsoft, but I assume I am not the first person to do this, and am looking for as much help and documentation as possible.

 

 

thanks to any and all for bearing with me on this one.

 

 

Sc0tt...

 

 

 

object options for access policy objects

 

General Purpose

 

Date TimeCreate branch rules based on timeLogon PageWeb form-based logon page for collecting end user credentials

 

HTTP 401 ResponseHTTP 401 Response for Basic or SPNEGO/Kerberos authenticationExternal Logon PageRedirect user to externally hosted web form-based logon page

 

Full Resource AssignAdvanced expression-based assignment of Connectivity Resources, Webtop, and ACLsResource AssignSimple assignment of Connectivity Resources

 

ACL AssignSimple assignment of on-box created Access Control Lists (ACLs)Webtop and Links AssignSimple assignment of Webtop and Webtop Links

 

Pool AssignAssign Local Traffic PoolVariable AssignAdvanced assignment of custom variables, configuration variables, or predefined session variables

 

Virtual KeyboardEnables a virtual keyboard on the web form-based logon page for entering credentialsSSO Credential MappingEnables Single Sign-On (SSO) credentials caching and assigns SSO variables

 

Citrix Smart AccessEnables Citrix SmartAccess filters when deploying with XenApp or XenDesktopRoute Domain and SNAT SelectionDynamic Route Domain and SNAT settings selection

 

LoggingLog custom messages and session variables for reporting and troubleshootingEmailConfigure Email messages for reporting

 

Message BoxCreate a custom message to display to the end user with prompt to continueDecision BoxCreate a custom decision page to display to the end user with two choices

 

Dynamic ACLAssignment of Access Control Lists (ACLs) retrieved from an external directory such as RADIUS or LDAPiRule EventRaises an iRule ACCESS_POLICY_AGENT_EVENT event for use with custom iRules scripts

 

EmptyCreates an Empty Action for constructing custom Branch Rules

 

Authentication

 

AD AuthActive Directory authentication of end user credentialsAD QueryActive Directory query to pull user attributes for use with resource assignment or other functions / Group Mapping

 

Client Cert InspectionCheck the result of client certificate authentication by the Local Traffic Client SSL profileCRLDP AuthCertificate Revocation List Distribution Point (CRLDP) client certificate authentication

 

HTTP AuthHTTP authentication of end user credentialsLDAP AuthLDAP authentication of end user credentials

 

LDAP QueryLDAP query to pull user attributes for use with resource assignment or other functions / Group MappingNTLM Auth Result CheckCheck the result of NTLM authentication of end user credentials

 

OCSP AuthOnline Certificate Status Protocol (OCSP) client certificate authenticationOn-Demand Cert AuthDynamically initiate an SSL re-handshake and validate the received client certificate

 

RADIUS AuthRADIUS authentication of end user credentialsRADIUS AcctSend accounting messages to a RADIUS server when users log on and off

 

RSA SecurIDRSA SecurID two-factor authentication of end user credentialsTACACS+ AuthTACACS+ Authentication of end user credentials

 

TACACS+ AcctSend accounting messages to a TACACS+ server when users log on and offKerberos AuthKerberos authentication, typically following an HTTP 401 Response action

 

OAMOracle Access Manager (OAM) authentication of end user credentialsSAML AuthSAML Auth using SAML Service Provider Interface

 

OTP GenerateGenerate One Time Passcode (OTP)OTP VerifyVerify One Time Passcode (OTP)

 

Client Side Checks

 

Antivirus CheckAntivirus Check for Windows, Mac and LinuxFirewall CheckFirewall Check for Windows, Mac and Linux

 

Windows File CheckWindows File CheckMachine Cert AuthWindows Machine Cert Auth

 

Windows InfoWindows OS InfoMachine InfoWindows Machine Info

 

Windows Process CheckWindows Process CheckRegistry CheckWindows Registry Check

 

Mac File CheckMac File CheckMac Process CheckMac Process Check

 

Linux File CheckLinux File CheckLinux Process CheckLinux Process Check

 

Client Side Actions

 

Cache and Session ControlWindows Browser Cache and Session ControlProtected WorkspaceWindows Protected Workspace

 

Windows Group PolicyWindows Group Policy

 

Server Side Checks

 

Client-Side Check CapabilityClient-Side Check CapabilityClient OSType of Client OS

 

Client TypeType of Client ApplicationLanding URICheck Landing URI

 

IP Subnet MatchCheck Client's IP SubnetIP Reputation CheckCheck Client's IP Reputation

 

Client for MS ExchangeCheck for client for MS Exchange Server, such as MS Outlook, etc. This action requires _sys_APM_ExchangeSupport_main or _sys_APM_ExchangeSupport_OA_BasicAuth iRule or _sys_APM_ExchangeSupport_OA_NtlmAuth iRule.IP Geolocation MatchMatch IP Geolocation

 

License CheckCreate branch rules based on license usage

 

HI,

 

I believe I have found the correct document for what I am trying to do. This seems to use a 401 response to collect credentials and allows me to use either basic or kerberos authentication. Pretty much what I think I a trying to do.

 

 

http://support.f5.com/kb/en-us/products/big-ip_apm/manuals/product/apm-aaa-auth-config-11-3-0/3.htmlconceptid

That's a decent document, though I would add a few thoughts:

 

 

1. You can add and remove branches from the 401 agent, so you could strip out the Basic auth and just do Negotiate. To do that, go to the Branch Rules tab and remove the Basic condition, then go back to the Properties tab and select negotiate as the HTTP Auth Level.

 

 

2. Negotiate applies to both NTLM and Kerberos on the client side.

 

 

3. I would recommend using the KRB5_NT_PRINCIPAL PTYPE when exporting the keytab. It shouldn't break it to use another type, but this is, in my opinion, more semantically correct. You also shouldn't have to specify the -crypto version if using Win2008, though it may come in handy with different OS versions. Your setspn would then look something like this:

 

setspn -princ HTTP/www.example.com@EXAMPLE.COM -mapuser example\joe.user -ptype KRB5_NT_PRINCIPAL -pass password -out c:\temp\www.example.com.keytab

 

 

So from an APM traffic flow perspective:

 

1. The user accesses the VIP for the first time and does not send a session cookie - APM redirects the user to /my.policy with a new session cookie

 

2. The user accesses the /my.policy URI, sends the new session cookie, signaling the beginning of the access policy evaluation

 

3. The 401 agent sees that the user is not sending an Authorization header and returns a 401 response with the configured options (Basic or Negotiate)

 

4. The user, in the case of Negotiate, communicates with the KDC (domain controller) to get a ticket for the service (as specified in the URL - so for example "http/www.example.com@EXAMPLE.COM")

 

5. The KDC validates the user's TGT (a long-life ticket they received when they first authenticated to the domain), generates a ticket (nonce, time stamp, client and service information, PAC data, one copy of short-lived session encryption key, other stuff), encrypts that value in the service's Kerberos encryption key (that it shares with the service), adds the other copy of the short-lived session encryption key and wraps it again in the user's encryption key (that is shares with the user), and then passes it back to the user.

 

6. The user decrypts the outer shell and extracts a session encryption key and an encrypted blob that it cannot decrypt, and passes that blob to the service. This binary data is base64-encoded and placed into an Authorization: Negotiate HTTP header.

 

7. The APM 401 agent sees the Authorization header, that it is of the type Negotiate, and then sends it down its Negotiate branch, to the Kerberos Auth agent.

 

8. The Kerberos auth agent should posses a keytab file, as defined in the Kerberos AAA, that contains the encryption of the service (ie. http/www.example.com@EXAMPLE.COM). If it does, then it should be able to decrypt the remaining layer and expose the Kerberos ticket data and the second session encryption key. If that succeeds then APM client side Kerberos authentication is COMPLETE. The Kerberos auth fills the session.logon.last.username session variable with the UPN of the user (ex. joe.user@example.com). You can then use this value however you like to do SSO on the server side.

 

 

And now a few last thoughts.

 

1. The 401 agent will pass the traffic through a specified branch if it sees the Authorization header.

 

2. The SSO Credential Mapping agent is only needed with certain types of SSO profiles, and has nothing to do with client side authentication. It is there to format user/pass/domain values as required for the SSO. For client side Kerberos you only need the 401 and Kerberos Auth agents.

I am new to this forum and to F5 prodcuts in general. I was troubleshooting an issue with APM Kerberos Auth AAA and stumbled to this forum. Very good discussion indeed. I have done most of the things pointed out on this article

 

 

1. VIP IP registered in DNS (host A and PTR records) - Testserver.demo.com 10.1.1.5

 

2. F5 Configured to point to AD domain DNS server

 

3. Created a service account in AD - demo\SvcAcct

 

4. Create an SPN -- setspn -U -A HTTP/Testserver.demo.com SvcAcct

 

5. Created a keytab file = ktpass -princ HTTP/Testserver.demo.com@DEMO.COM -mapuser svcAcct@DEMO.COM -crypto rc4-hmac-nt -ptype KRB5_NT_SRV_HST -pass password -out c:\temp\svcacct.keytab

 

6. Uploaded the keytab file to the Access Policy AAA Server configuration (verified keytab file using Klist command tool on the F5 command shell)

 

7. Created a constrained delegation on the SvcAcct to specific servicess (HOST and HTTP services of the web servers) any protocol

 

8. verified delegation by using the Kinit tool

 

9. created a acces policy HTTP 401 Response --> Nego -> Kerberos Auth --> success -> Allow

 

 

After all this, I still cannot get Kerberos auth to be successfull. Can someone tell me what I may be missing out or doing wrong.

 

 

Thank you for you input in advance

 

 

 

 

Okay, to start with let's clarify that the Kerberos AAA is a client side config. If you have to use Kerberos on the server side, that's a completely different set of options. Here are some things to look at as you troubleshoot APM Kerberos AAA:

 

 

1. The first thing you should do is set up a capture. If you can get Wireshark on the DC, that tends to be the best vantage point. You'll be looking specifically for Kerberos (and potentially DNS) traffic.

 

 

2. There are a few places client side Kerberos can go wrong, so to rule out issues on the client itself, add some message boxes inline with the 401 and Kerberos auth agents. You want to see a message before the 401 (initial request), after the 401 (browser returning with a Kerberos ticket), and then after the Kerberos auth agent (good auth). If it never makes it past the 401 agent, then your browser is likely not making a Kerberos request. If an IE browser, you need to add the VIP's host name to your Trusted Intranet Sites list. If you've done that and now failing at the Kerberos auth agent, continue the steps below.

 

 

3. I've have very limited success with AD service accounts that are not all lowercase. I know it doesn't make sense, but that is a pretty consistent observation.

 

 

4. Once you've created the service account, you don't have to manually create the SPN with SETSPN as the KTPASS command will do that for you. And although there are a few references to using a ptype of KRB5_NT_SRV_HOST, you're tying this SPN to a user account, so it's more semantically correct to use KRB5_NT_PRINCIPAL. When you're done you can verify the SPN creation in the service account.

 

 

5. Delegation settings are absolutely NOT required for client side Kerberos.

 

 

6. Verify of course that you have good A and PTR records for the VIP's host name and that the clocks of all involved parties are fairly close if not in sync. Also check for duplicate SPNs. There's nothing more frustrating than a week of troubleshooting to find out someone else created the same SPN somewhere else.

 

 

7. Between testing it's a good idea to clear caches. On the APM side, issue the following command in the shell:

 

 

bigstart restart rba

 

 

On the client side (Windows), issue the following command in the shell:

 

 

klist purge

 

 

8. You can increase logging with the following TMSH command:

 

 

tmsh modify sys db log.rba.level value debug

 

 

Make sure to set it back to "notice" when you're done troubleshooting.

 

 

9. If you're still not getting past the Kerberos auth agent, take a closer look at DNS traffic in the captures. On some systems I've seen APM attempt to resolve AAAA records in AD and then fail if they don't exist. Just make sure whatever APM is asking for exists and is resolvable.

 

 

10. Finally, and this is a stretch, but if you've been at this troubleshooting for awhile, there's a good possibility that you could have mismatched KVNO values. On your DC, issue the following command (modify as required):

 

 

ldifde -f c:\spn_out.txt -d "DC=mydomain,DC=com" -l *,msDS-KeyVersionNumber -r "(serviceprincipalname=HTTP/webtest*)" -p subtree

 

 

The ldifde command comes in a resource pack I believe. Look for the "msDS-KeyVersionNumber" string in the output file. On the BIG-IP:

 

 

klist -ekt

 

 

If the KVNO values don't match, delete and recreate the AD service account, rerun KTPASS, and reimport the keytab to the AAA config.

 

 

The above steps usually resolve most of my client side Kerberos issues. If you're still not getting it to work, please reply back and submit a sample of the captures if you can.

 

The above klist command should be:

 

 

klist -ekt [path to keytab file]

Thank you Kevin,

 

 

1. I created a new user account in AD -- all lower case

 

2. Generated new keytab file with KRB5_NT_PRINCIPAL and uploaded to the AAA server config in APM

 

3. Verified Host and PTR records can be resolved by all units involved - Client machine, Domain Controller, F5 appliance, Webserver

 

4. Domain Controller (PDC) is configured as the NTP server of the F5 appliance to avoid time skew

 

5. I have checked the KVNO values in AD and the keytab file - it is a match

 

6. Tested keytab file using KList command - good

 

 

Still No good

 

 

Took a wireshark capture on the domain controller

 

 

I could see TGS kerberos request from the client machine to DC and DC sending back granted TGS ticket to client

 

Checked DNS traffic, I saw the F5 was looking for AAAA record which it does not find, then it asks for host A record which it resolves but I don't know what it does with it after that

 

On further analysis, we noticed traffic from F5 (e.g DNS lookup, TCP) are coming from the mgmt port of the appliance rather than the Self-IP configured on the LTM (which I believe APM is listening on)

 

Futhermore, the IP addres of the MGMT port is on the same VLAN as the Domain Controller. We are suspecting this could be the problem but we are yet to confirm. I will update you once we make the change of the MGMT Ip address to a diff VLAN.

 

 

Thanks you again for you response.

 

 

 

 

 

 

 

 

Try this:

 

 

1. Remove the DNS Lookup Server List value from the BIG-IP (System - Configuration - Device - DNS) and test again.

 

 

2. If the above works, add the DNS server IP back and create a bogus AAAA value for this host in your DNS so that the AAAA request succeeds.

OK, I finally got this to work. on creating the keytab file, I notieced I was using the "domainname\UserName" parameter while mapping the user to the SPN as show below

 

ktpass -princ HTTP/www.example.com@EXAMPLE.COM -mapuser example\joe.user -ptype KRB5_NT_PRINCIPAL -pass password -out c:\temp\www.example.com.keytab

 

I created a new keytab file but changed the -mapuser to "username@EXAMPLE.COM.