Kerberos is Easy - Part 2

Yes my friends, this post has been long overdue. Life, work and all the other good excuses got in the way. However, there is nothing like a friend calling you out with a “WTF I need part 2” to get the motivation and kerberos mana flowing again. So where did we leave off? In part 1 we discussed some of the most common issues with Kerberos authentication and the necessity to break the problem down to client-side vs server-side authentication.

In part 2 we will look at some of the first troubleshooting steps I take to determine why nothing is working - yes, this happens to me too.

ADTest is Your New BF4L

You know those people that open a web browser to see if their Internet works… yea, don’t be that person. Open a terminal, check if you can ping your gateway, public DNS server, etc. and then, only then, open a browser.

ADTest is your equivalent of ping. Don’t assume because you configured an Active Directory AAA object that authentication is just going to work. Please open a console and verify that Kerberos authentication against the AD server is working with ADTest. Check out my APM Troubleshooting with ADTest for more information.

Time is not on Your Side

If you have worked with Kerberos before you know it is supper picky about time drift. If ADTest just won’t work and you can’t figure out why ensure the BIG-IP’s time matches the KDC. If you need to adjust the BIG-IP follow the F5 SOL3381.

These Are Not the KDCs You’re Looking For

Ever been in a multi-domain environment and the AD admin swears the KDC you’re talking to is the correct one; never mind the “Kerberos Principal Unknown” error you keep getting. So if ADTest doesn’t work then we need to ensure the AD server we’re talking to is a KDC for the expected realm. Now, if someone can RDP into the server this can be ruled out pretty quickly, but when in life is anything that easy.

So it’s helpful to use nslookup to find all KDCs for the intended domain and ensure the IP you were give is in this list - example below:

C:\Users\user>nslookup -type=SRV _kerberos._tcp.dc._msdcs.f5demo.com

Server: ad1.f5demo.com

Address: 10.1.10.2

_kerberos._tcp.dc._msdcs.f5demo.com SRV service location:

priority = 0

weight = 100

port = 88

svr hostname = ad1.f5demo.com

ad1.f5demo.com internet address = 10.1.10.2

ad1.f5demo.com internet address = 10.1.1.3

Playing Go Fish with SPNs

You have to admire how Microsoft took something as complicated as Kerberos and made it trivial to deploy and manage inside of Active Directory. Having managed MIT and MS version of Kerberos myself I felt a little guilty after setting up my first AD server without hours of troubleshooting issues. However, with that ease of use Microsoft also made it easy to shoot your own foot off if you have no basic understanding of Kerberos.

Think of the KDC as a key, value pair database. The KDC will let you store multiple keys of the same value even though you shouldn’t do that. So if you have multiple SPN entries in AD you are not guarantied that a request for a ticket will return the value you’re looking for. This typically presents itself in APM as authentication works onetime and not the other.

An easy way to check this is to log into a domain machine and issue:

setspn -X 

This will print out any duplicate SPNs in your KDC. If the SPN you are working with appears in this list then you need to correct this issue. The easiest way I find is to delete the service account you have created for APM and just use the service account the web server application pool is using.

Cached Tickets

APM caches Kerberos tickets for both client side Kerberos authentication and server side Kerberos SSO. If you’re troubleshooting Kerberos be sure to clear these caches after you’ve made modifications.

For Kerberos AAA:

bigstart restart apd rba

For Kerberos SSO:

bigstart restart websso

I’ll Just Do It Myself

If you are working with Kerberos SSO then you have to sometimes determine if the issue is with APM obtaining the token or the web server not accepting the token. If you have your WebSSO logging set to debug and Kerberos SSO is working then you should see

S4U=====>OK

If not, then there are a few CLI commands you can use to simulate the request APM makes to the KDC.

First, remove all kerberos tickets.

kdestroy

Second, obtain a kerberos ticket as the AD delegation account (if this works you won’t get a response)

kinit -f <SPN of AD delegation account>

Finally, test if the ticket you obtain has delegation capabilities

knvo -C -U <username> <SPN of AD delegation account>

If you receive a key version number on the kvno command then everything is working and it proves that ASREQ and RSREQ work. So the issue more than likely is on the web server side and not the F5 - more on this in part 3.