Configuring Smart Card Authentication to BIG-IP Management Interface

Developed on BIG-IP Version 13.1

It's been quite a while since my last article, so I wanted to come up with something that I know would benefit all current, future and past customers. Over the past few years of deploying and managing BIG-IP's, I always got the same question from my federal customers. How do we smart card enable our BIG-IP management interface? Well, I'm here to not only tell you but show how it's done. I will also share some of the troubleshooting steps, logs and tools I used to overcome my own issues while attempting this. So, with that, let's get started.

Configuring Remote Role Groups

So, how many of you today are still using local credentials or defining administrative users one by one within the BIG-IP TMUI? Did you know you could use active directory security groups to make managing administrative access a whole lot easier? Thank goodness the answer is yes, and it's called Remote Role Groups.

Before we begin defining the authentication method, you must configure Remote Role Groups since this will be referenced immediately after changing authentication to Remote - ClientCert LDAP.

Navigate to System > Users > Select Remote Role Groups

Click Create
Group Name: BIGIPAdmins
Line Order: 1
Attribute String:memberOF=CN=BIGIPadmins,OU=Groups,DC=demo,DC=lab Note: Use the full DN of the active directory security group you are defining with a preceeding 'memberOF='.
Assigned Role: Administrator
Partition Access: All
Terminal Access: tmsh

Validating Certificate Revocation Using OCSP

You might be wondering; wouldn't this be a part of the troubleshooting steps after we configure TMUI to support Smart Card? You would be right, though doing this in a lab I do not consider myself an expert when deploying and configuring a PKI infrastructure within a Windows 2012 environment so if this is helpful great, if not continue to the next step.

Using a copy of my user certificate, I am going to run a command to obtain the AIA information and perform a revocation check against my local OCSP responder to validate I am able to successfully verify my certificate.

certutil -URL path\user.cer

You will then be prompted with a URL Retrieval Tool
Select OCSP (from AIA) and click Retrieve

If valid, you will receive a status of Verified as shown above.

Obtain a CA or CA Bundle in PEM Encoded Format

For DoD Customers, navigate to https://iase.disa.mil/pki-pke/Pages/index.aspx
Select For Administrators, Integrators and Developers
Select Tools and continue to browse until you locate PKI CA Certificate Bundles: PEM Self-Extracting ZIP
Select the .exe that is appropriate for your organization though as an example I have selected For DoD PKI Only.

Run the executable to extract all CA certs into new empty directory

Launch a command prompt from Start > Run > cmd
Change directories until you are at the location where you extracted all CA certificate files.
Run the command copy /B *.cer DoDCABundle.cer

Import CA Bundle into BIG-IP

Log into the BIG-IP TMUI > System > Certificate Management > Device Certificate Management > Device CA Certificate List > Import
Browse to the directory that you stored the CA Bundle in

Provide a Name and select Import

Configure User Authentication

Navigate to System > Users > Authentication > Change
From the User Directory drop down select Remote - ClientCert LDAP
Host: IP address of your directory services server
Port: 389
Remote Directory Tree: DC=demo,DC=lab
Scope: Sub
Bind DN: CN=admin,CN=Users,DC=demo,DC=lab

Provide Password and Confirm

Check Member Attribute in Group: Enabled
SSL: Disabled
CA Certificate: Select the CA certificate bundle created in the previous steps.
Login Name: Can leave empty
Login LDAP Attribute: userPrincipalName (Case Sensitive)
Login Filter: [a-zA-Z0-9]\\w*(\?=@)
Depth: 10
Client Certificate Name Field: Other Name... OID: 1.3.6.1.4.1.311.20.2.3
OCSP Override: On
OCSP Responder: http://IP/ocsp
OCSP Response Max Age: -1
OCSP Response Time Skew: 300
OCSP Response Timeout: 300
External Users: Leave Defaults
Select Finished

If successful, you will be prompted for a client certificate and re authenticated with no issues. But let’s just say I wasn't that lucky the first few times I attempted this config.

The big scary error I got and continued to get is below....

Therefor I will provide some of the troubleshooting tips and tricks that assisted me in determining why authentication was failing.

Troubleshooting

So, let’s start with the one issue that really scared me the most even though it was just a development environment, httpd. During my first attempt at ClientCert authentication I followed the instructions as they were laid out in the deployment guide though for some reason not only could I not authenticate using a certificate, when turning SSO off I still couldn't log in. So, to save you all the trouble of determining what configuration item was causing this issue, I can tell you it was the httpd service not starting due to a non PEM CA certificate.

First, I ran a bigstart status httpd and noticed it was not running. When attempting a bigstart start httpd it would fail. Luckily httpd has its own log file though honestly it didn't help much.

Launch a putty session and login using root or similar credentials that allow access to both the shell and tmsh.
Navigate to the httpd directory by running cd /var/log/httpd
Perform a tail on the httpd_errors log by running tail -f httpd_errors

The error that I just could not figure out was "Unable to configure verify locations for client authentication." Believe it or not I didn't find much on devcentral or internal resources on this error, so I started rolling back configuration items back one by one until I ran into the ssl-ca-cert-file within the httpd config. After modifying this to none, I was able to log in. That is of course after disabling SSO by running tmsh modify auth cert-ldap system-auth sso off. So, before moving forward ensure your CA file is in PEM format before configuring client-cert LDAP auth.

Now, moving on. I won't get into it too much but as mentioned at the beginning of the article, it is probably a good idea to ensure the client certificate can be validated by running certutil -URL path\user.cer. So, the other issue that I ran into because I am running Windows OCSP, I did not have Nonce extension support enabled on my responder. Therefor I was receiving the error messages below in the httpd_errors log.

Jan 22 15:47:18 bigip1 err httpd[20075]: [error] OCSP response not successful: 0

Jan 22 15:47:18 bigip1 err httpd[20075]: [error] [client IP] Certificate Verification: Error (50): application verification failure

After configuring Nonce extension support, I thought I was good...nope, more hurdles. So now that I have a CA bundle in PEM format, Nonce extension support enabled, a valid user certificate I still couldn't log on. This time after seeing successful responses from my OCSP responder, I went to my secure log due to getting prompted for cert and then username and password.

From the shell, navigate to /var/log and run a tail -f secure. In the secure log I was continuously getting unknown user though I know the DN for my group was correct and I certainly have an AD account, so what now? So this is where I went old school and downloaded an archived version of Netmon! Thank you for teaching me this a long long time ago @Mike Melone!

After downloading Netmon (you can of course use Wireshark) I started a packet capture with a filter of tcp.port == 389 so that I can see the ldap request and responses.

Based on the help for the deployment of ClientCert - LDAP, my interpretation was that you MUST use sAMAccountName if authenticating against AD. However, clearly I misinterpreted because after looking at the capture there was no way the filter was my sAMAccountName but rather my UPN. With that information, I modified the cert-ldap login-attribute from sAMAccountName to userPrincipalName and boom, it worked! Now, by no means would this be the only attribute you could use but rather the attribute that I utilized for successful logon when the UPN on my cert references an alternate UPN suffix than my actual active directory domain name.

That wraps another article that I hope the community finds helpful. Below you can find my actual config that you would find under tmsh list auth cert-ldap and tmsh list sys httpd. Until next time.

root@(bigip1)(cfg-sync Standalone)(ModuleNotLicensed::Active)(/Common)(tmos)# list sys httpd

sys httpd {
   auth-pam-idle-timeout 12000
   ssl-ca-cert-file /Common/CABase64
   ssl-ocsp-default-responder http://10.1.20.10/ocsp
   ssl-ocsp-enable on
   ssl-ocsp-override-responder on
   ssl-verify-client require

}

root@(bigip1)(cfg-sync Standalone)(ModuleNotLicensed::Active)(/Common)(tmos)# list auth cert-ldap

auth cert-ldap system-auth {
   bind-dn CN=admin,CN=Users,DC=demo,DC=lab
   bind-pw $M$Sh$JrUPQrhEhMicK39ZostQJQ==
   check-roles-group enabled
   debug enabled
   login-attribute userPrincipalName
   login-filter [a-zA-Z0-9]\\\\w*(\\\?=@)
   search-base-dn DC=demo,DC=lab
   servers { 10.1.20.10 }
   ssl-ca-cert-file Base64CA.crt
   ssl-cname-field san-other
   ssl-cname-otheroid 1.3.6.1.4.1.311.20.2.3
   sso on