Configuring Smart Card Authentication to BIG-IP Management Interface
Developed on BIG-IP Version 13.1 It's been quite a while since my last article, so I wanted to come up with something that I know would benefit all current, future and past customers. Over the past few years of deploying and managing BIG-IP's, I always got the same question from my federal customers. How do we smart card enable our BIG-IP management interface? Well, I'm here to not only tell you but show how it's done. I will also share some of the troubleshooting steps, logs and tools I used to overcome my own issues while attempting this. So, with that, let's get started. Configuring Remote Role Groups So, how many of you today are still using local credentials or defining administrative users one by one within the BIG-IP TMUI? Did you know you could use active directory security groups to make managing administrative access a whole lot easier? Thank goodness the answer is yes, and it's called Remote Role Groups. Before we begin defining the authentication method, you must configure Remote Role Groups since this will be referenced immediately after changing authentication to Remote - ClientCert LDAP. Navigate to System > Users > Select Remote Role Groups Click Create Group Name: BIGIPAdmins Line Order: 1 Attribute String:memberOF=CN=BIGIPadmins,OU=Groups,DC=demo,DC=lab Note: Use the full DN of the active directory security group you are defining with a preceeding 'memberOF='. Assigned Role: Administrator Partition Access: All Terminal Access: tmsh Validating Certificate Revocation Using OCSP You might be wondering; wouldn't this be a part of the troubleshooting steps after we configure TMUI to support Smart Card? You would be right, though doing this in a lab I do not consider myself an expert when deploying and configuring a PKI infrastructure within a Windows 2012 environment so if this is helpful great, if not continue to the next step. Using a copy of my user certificate, I am going to run a command to obtain the AIA information and perform a revocation check against my local OCSP responder to validate I am able to successfully verify my certificate. certutil -URL path\user.cer You will then be prompted with a URL Retrieval Tool Select OCSP (from AIA) and click Retrieve If valid, you will receive a status of Verified as shown above. Obtain a CA or CA Bundle in PEM Encoded Format For DoD Customers, navigate to https://iase.disa.mil/pki-pke/Pages/index.aspx Select For Administrators, Integrators and Developers Select Tools and continue to browse until you locate PKI CA Certificate Bundles: PEM Self-Extracting ZIP Select the .exe that is appropriate for your organization though as an example I have selected For DoD PKI Only. Run the executable to extract all CA certs into new empty directory Launch a command prompt from Start > Run > cmd Change directories until you are at the location where you extracted all CA certificate files. Run the command copy /B *.cer DoDCABundle.cer Import CA Bundle into BIG-IP Log into the BIG-IP TMUI > System > Certificate Management > Device Certificate Management > Device CA Certificate List > Import Browse to the directory that you stored the CA Bundle in Provide a Name and select Import Configure User Authentication Navigate to System > Users > Authentication > Change From the User Directory drop down select Remote - ClientCert LDAP Host: IP address of your directory services server Port: 389 Remote Directory Tree: DC=demo,DC=lab Scope: Sub Bind DN: CN=admin,CN=Users,DC=demo,DC=lab Provide Password and Confirm Check Member Attribute in Group: Enabled SSL: Disabled CA Certificate: Select the CA certificate bundle created in the previous steps. Login Name: Can leave empty Login LDAP Attribute: userPrincipalName (Case Sensitive) Login Filter: [a-zA-Z0-9]\\w*(\?=@) Depth: 10 Client Certificate Name Field: Other Name... OID: 1.3.6.1.4.1.311.20.2.3 OCSP Override: On OCSP Responder: http://IP/ocsp OCSP Response Max Age: -1 OCSP Response Time Skew: 300 OCSP Response Timeout: 300 External Users: Leave Defaults Select Finished If successful, you will be prompted for a client certificate and re authenticated with no issues. But let’s just say I wasn't that lucky the first few times I attempted this config. The big scary error I got and continued to get is below.... Therefor I will provide some of the troubleshooting tips and tricks that assisted me in determining why authentication was failing. Troubleshooting So, let’s start with the one issue that really scared me the most even though it was just a development environment, httpd. During my first attempt at ClientCert authentication I followed the instructions as they were laid out in the deployment guide though for some reason not only could I not authenticate using a certificate, when turning SSO off I still couldn't log in. So, to save you all the trouble of determining what configuration item was causing this issue, I can tell you it was the httpd service not starting due to a non PEM CA certificate. First, I ran a bigstart status httpd and noticed it was not running. When attempting a bigstart start httpd it would fail. Luckily httpd has its own log file though honestly it didn't help much. Launch a putty session and login using root or similar credentials that allow access to both the shell and tmsh. Navigate to the httpd directory by running cd /var/log/httpd Perform a tail on the httpd_errors log by running tail -f httpd_errors The error that I just could not figure out was "Unable to configure verify locations for client authentication." Believe it or not I didn't find much on devcentral or internal resources on this error, so I started rolling back configuration items back one by one until I ran into the ssl-ca-cert-file within the httpd config. After modifying this to none, I was able to log in. That is of course after disabling SSO by running tmsh modify auth cert-ldap system-auth sso off. So, before moving forward ensure your CA file is in PEM format before configuring client-cert LDAP auth. Now, moving on. I won't get into it too much but as mentioned at the beginning of the article, it is probably a good idea to ensure the client certificate can be validated by running certutil -URL path\user.cer. So, the other issue that I ran into because I am running Windows OCSP, I did not have Nonce extension support enabled on my responder. Therefor I was receiving the error messages below in the httpd_errors log. Jan 22 15:47:18 bigip1 err httpd[20075]: [error] OCSP response not successful: 0 Jan 22 15:47:18 bigip1 err httpd[20075]: [error] [client IP] Certificate Verification: Error (50): application verification failure After configuring Nonce extension support, I thought I was good...nope, more hurdles. So now that I have a CA bundle in PEM format, Nonce extension support enabled, a valid user certificate I still couldn't log on. This time after seeing successful responses from my OCSP responder, I went to my secure log due to getting prompted for cert and then username and password. From the shell, navigate to /var/log and run a tail -f secure. In the secure log I was continuously getting unknown user though I know the DN for my group was correct and I certainly have an AD account, so what now? So this is where I went old school and downloaded an archived version of Netmon! Thank you for teaching me this a long long time ago @Mike Melone! After downloading Netmon (you can of course use Wireshark) I started a packet capture with a filter of tcp.port == 389 so that I can see the ldap request and responses. Based on the help for the deployment of ClientCert - LDAP, my interpretation was that you MUST use sAMAccountName if authenticating against AD. However, clearly I misinterpreted because after looking at the capture there was no way the filter was my sAMAccountName but rather my UPN. With that information, I modified the cert-ldap login-attribute from sAMAccountName to userPrincipalName and boom, it worked! Now, by no means would this be the only attribute you could use but rather the attribute that I utilized for successful logon when the UPN on my cert references an alternate UPN suffix than my actual active directory domain name. That wraps another article that I hope the community finds helpful. Below you can find my actual config that you would find under tmsh list auth cert-ldap and tmsh list sys httpd. Until next time. root@(bigip1)(cfg-sync Standalone)(ModuleNotLicensed::Active)(/Common)(tmos)# list sys httpd sys httpd { auth-pam-idle-timeout 12000 ssl-ca-cert-file /Common/CABase64 ssl-ocsp-default-responder http://10.1.20.10/ocsp ssl-ocsp-enable on ssl-ocsp-override-responder on ssl-verify-client require } root@(bigip1)(cfg-sync Standalone)(ModuleNotLicensed::Active)(/Common)(tmos)# list auth cert-ldap auth cert-ldap system-auth { bind-dn CN=admin,CN=Users,DC=demo,DC=lab bind-pw $M$Sh$JrUPQrhEhMicK39ZostQJQ== check-roles-group enabled debug enabled login-attribute userPrincipalName login-filter [a-zA-Z0-9]\\\\w*(\\\?=@) search-base-dn DC=demo,DC=lab servers { 10.1.20.10 } ssl-ca-cert-file Base64CA.crt ssl-cname-field san-other ssl-cname-otheroid 1.3.6.1.4.1.311.20.2.3 sso on Reference Articles BIG-IP Remote User Account ManagementUS FEDERAL: Enabling Kerberos for Smartcard Authentication to Apache.
The following provides guidance on the configuration of BIG-IP Local Traffic Manager and Access Policy Manager in support of Apache Web Server Smartcard / Kerberos access using Active Directory as the Key Distribution Center. This content is part of a series developed to address the configuration of non IIS webservers to support Kerberos Single Sign On and therefore smartcard access, but should be relevant anywhere SSO utilizing Kerberos is needed. Several assumptions are made concerning the implementation of Active Directory, PKI, and the Linux Distro(s) used. Base Software Requirements The following base requirements are assumed for this configuration. Microsoft Windows Server 2008 R2 (Active Directory) BIG-IP LTM 11.4 or higher (the configuration items will probably work with most versions of 11 but only 11.4 and 11.5 were tested in the scenario) Ubuntu Server 13.10 (This is a fairly simple and user friendly distro based on Debian, this was also tested in RHEL/CentOS.) This config will work in other distro’s of Linux, but posting all the difference configurations would just be redundant. If you need help, reach out to the US Federal Team. How it Works The configuration of this scenario is fairly simple. The majority of the configuration and testing will most likely reside on the Linux side. The client access and authenticates to APM via a smartcard. Depending on the method of choice, an attribute identifying the user is extracted from the certificate and validated against an AD/LDAP. In Federal, this step has two purposes; to extract the UPN to query AD for the User (EDIPI@MIL), and to retrieve the sAMAccountName to use for the Kerberos Principal. Once the user has been validated and the sAMAccountName retrieved, the session variables are assigned and the user is granted access. Base Linux Configuration Configure Static IP & DNS You can use the text editor of your own preference, but I like nano so that is what I will document. sudo nano /etc/network/interfaces You will want to change iface eth0 inet dhcp to static, and change the network settings to match your environment. Since this scenario uses Windows AD as the KDC, you will want to make sure your DNS points to a domain controller. auto eth0 iface eth0 inet static address 192.168.1.2 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255 gateway 192.168.1.1 dns-nameservers 192.168.1.1 Note: Depending on your distro, you will use dns-nameservers or resolv.conf. I also removed the DHCP client entirely. (Not necessary, but I like to clean out things I wont ever use.) Restart networking sudo /etc/init.d/networking restart Or sudo service networking restart Install LAMP (Linux, Apache, MySQL, PHP) In Ubuntu, this is fairly simple, you can just do the following. sudo tasksel Then check the box for LAMP, and follow the on-screen instructions, set MySQL password, and then you are done. If you access the IP of your server from a browser, you will see the default Apache "It Works!" page. Install & Configure Kerberos sudo apt-get install krb5-user Some distros will ask for default REALM, KDC, and Admin server configs. In my case it is F5LAB.LOCAL, 192.168.1.5, 192.168.1.5. krb5.conf Depending on your distro, there will be a ton of extra settings in the krb5.conf file, some related to Heimdal and some for MIT Kerberos. The core settings that I needed for success are listed below. [libdefaults] Set your default realm, DNS lookups to true, and validate the encryption types. HMAC is good, Windows does not have DES enabled by default and you should not consider enabling it. default_realm = F5LAB.LOCAL dns_lookup_realm = true dns_lookup_kdc = true ticket_lifetime = 24h forwardable = true default_tgs_enctypes = arcfour-hmac-md5 des-cbc-crc des3-hmac-sha1 default_tkt_enctypes = arcfour-hmac-md5 des-cbc-crc des3-hmac-sha1 [realms] KDC: Domain Controller admin_server: Not required, but can also point o Domain Controller default_domain: Kerberos Realm F5LAB.LOCAL = { kdc = 192.168.1.5:88 admin_server = 192.168.1.5 default_domain = F5LAB.LOCAL } Install Mod_Auth_Kerb This is required to make Apache support Kerberos. Some distros include this when you load apache, but here is how you make sure. sudo apt-get install libapache2-mod-auth-kerb Testing Lets make sure that we configured networking and Kerberos properly. Use KINIT to test a known user account. This should reach out to the KDC to get a ticket for the user. REALMS are case sensitive, so make sure its all upper case. The following will request a password for the user, and if everything is set up properly, there will be no response. kinit mcoleman@F5LAB.LOCAL You can run KLIST to see your ticket. klist An example of what happens when the REALM is entered incorrectly: KDC reply did no match expectations while getting initial credentials. Windows Configurations Configuring SPNs Since Linux is not the KDC or Admin server, this is done on the Active Directory side. Create a user account for each application, with the appropriate Service Principal Names. Be aware, when we run keytab, all SPNs will be overwritten, with the exception of the SPN used in the command. Crypto Pay attention to the encryption types that are / were enabled in the krb5.conf file. It is important to remember that both DES cipher suites (DES-CBC-MD5 & DES-CBC-CRC) are disabled by default in Windows 7.The following cipher suites are enabled by default in Windows 7 and Windows Server 2008 R2: AES256-CTS-HMAC-SHA1-96 AES128-CTS-HMAC-SHA1-96 RC4-HMAC For the purposes of this guide and the available settings in Windows use RC4-HMAC. DO NOT enable DES on Windows. Create a Keytab Keytabs can be created in windows by using ktpass. A keytab is a file that contains a Kerberos Principal, and encrypted keys. The purpose is to allow authentication via Kerberos, without using a password. ktpass –princ HTTP/lamp.f5lab.local@F5LAB.LOCAL -mapuser F5LAB\apache.svc -crypto RC4-HMAC-NT -pass pass@word1 -ptype KRB5_NT_PRINCIPAL -kvno 0 -out LAMP.keytab Copy the keytab to your linux server(s). For my use case I put the keytab at /etc/apach2/auth/apache2.keytab Lock it down - Linux The security of a keytab is pretty important. Malicious users with access to keytabs can impersonate network services. To avoid this, we can secure the keytab’s permissions. sudo chown www-data:www-data /etc/apache2/auth/apache2.keytab sudo chmod 400 /etc/apache2/auth/apache2.keytab Testing Now, we want to make sure everything is looking alright so far. So lets make sure the keytab looks right, and we can authenticate properly against the KDC. List the contents of the Keytab klist –ke /etc/apache2/auth/apache2.keytab Test Authentication with the S4U SPN The following commands can be used to initialize the credential cache for the S4U proxy account and then to test authentication with a user account. kinit –f http/lamp.f5lab.local@F5LAB.LOCAL kvno http/lamp.f5lab.local@F5LAB.LOCAL sudo klist –e –k –t /etc/apache2/auth/apache2.keytab kvno –C –U mcoleman http/lamp.f5lab.local Apache Configurations I was able to get authentication working by adding the following to the default site. In Ubuntu its /etc/apache2/sites-enabled/000-default.conf. <VirtualHost *:80> … <Location /> Options Indexes AllowOverride None Order allow,deny allow from allAuthType Kerberos #KrbServiceName HTTP/lamp.f5lab.local@F5LAB.LOCAL AuthName "Kerberos Logon" KrbMethodNegotiate on KrbMethodK5Passwd on KrbVerifyKDC off KrbAuthRealm F5LAB.LOCAL Krb5KeyTab /etc/apache2/auth/apache2.keytab require valid-user </Location> </VirtualHost> BIG-IP Configurations This portion is actually pretty straightforward. Configure a standard Virtual Server with a Pool pointing at the Apache Servers. Configuration Items • Kerberos SSO Profile – This is used to authenticate to Apache. • Access Profile – The Access profile binds all of the APM resources. • iRule – an iRule is used to extract the smartcard certificate User Principal Name (UPN). • ClientSSL Profile - This is used to establish a secure connection between the user and the APM VIP. Apply the server certificate, key, and a trusted certificate authority’s bundle file. All other settings can be left at default. • HTTP profile – This is required for APM to function. A generic HTTP profile will do. • SNAT profile – Depending on other network factors, a SNAT profile may or may not be necessary in a routed environment. If the backend servers can route directly back to the clients, bypassing the BIG-IP, then a SNAT is required. • Virtual server –The virtual server must use an IP address accessible to client traffic. Assign a listener (destination) IP address and port, the HTTP profile, the client SSL profile, a SNAT profile (as required), the access profile, and the iRule. Modify the krb5.conf [libdefaults] default_realm = EXAMPLE.COM dns_lookup_realm = true dns_lookup_kdc = true ticket_lifetime = 24h forwardable = yes APM Kerberos SSO Profile Create an APM Kerberos SSO profile like the one shown below. Change the Username Source to “session.logon.last.username”, enter the Active Directory domain name (in all upper case), enter the full service principal name of the AD user service account previously created , and enter the account’s password. The only real change from IIS is the Send Authorization setting, which should be set to “On 401 Status Code.” Username Source: session.logon.last.username User REALM Source: session.logon.last.domain Kerberos REALM: F5LAB.COM KDC(optional): Account Name: HTTP/lamp.f5lab.com Account Password: password Confirm Account Password: password SPN Pattern (optional): Send Authorization: On 401 Status Note: The full service principal name includes the service type (ex. host/), the service name (ex. krbsrv.alpha.com), and the domain realm name (ex. @ALPHA.COM – in upper case). KDC can be specified, but is not needed unless you do not configure DNS lookup enabled in the krb5.conf on the F5. Basically, if you dont tell the F5 how to resolve the KDC, then you need to specify one. SPN Pattern can help resolve issues if you have issues with DNS/rDNS. You can specify which SPN you want to sent with either a designated, or dynamic option. VPE configuration The components of the VPE are as follows: • On-Demand Cert Auth – Set this to Require. • Rule event – Set the ID to “CERTPROC” to trigger the EDIPI extraction iRule code. • LDAP Query – Validates the UPN and retrieves sAMAccountName. Basic CAC iRule when ACCESS_ACL_ALLOWED { #Set Username to value of sAMAccountName extracted from LDAP Query. ACCESS::session data set session.logon.last.username [ACCESS::session data get "session.ldap.last.attr.sAMAccountName"] } when ACCESS_POLICY_AGENT_EVENT { switch [ACCESS::policy agent_id] { #Name of iRule event called from APM Policy "CERTPROC" { if { [ACCESS::session data get session.ssl.cert.x509extension] contains "othername:UPN<" } { #Set temporary session variable to value extracted from X.509 data. set tmpupn [findstr [ACCESS::session data get session.ssl.cert.x509extension] "othername:UPN<" 14 ">"] ACCESS::session data set session.custom.certupn $tmpupn #log local0. "Extracted OtherName Field: $tmpupn" } } } } Put it together. Now that all the functional parts are in place, you can test access to Apache. If you want to add some code to see what user is hitting your application, you can create a small PHP page containing the following code. $_SERVER['REMOTE_USER'] $_SERVER['KRB5CCNAME'] The server variables will echo the current authenticated user name. Troubleshooting Kerberos is fairly fault-tolerant, if the requisite services are in place. That being said, it can be a PITA to troubleshoot. If Kerberos authentication fails, check the following: The user has a valid ticket. Use klist, kinit, and kvno as explained previously. Validate basic network connectivity. DNS (Forward & Reverse), ensure no duplicate A or PTR records. This can be overwritten in the Keberos SSO profile SPN pattern settings. Verify the clocks of the KDC and local server are synced. Turn APM SSO logging up to debug and tail the APM logs (tail -f /var/log/apm). Questions? Contact the US Federal team, Federal [at] f5.com.
