Forum Discussion
Client authentication random failure - 11.6 HF4
We have a pair of BIG IP 6900 appliances that work as an active/passive HA pair. Recently we have upgraded the appliances to 11.6 HF4 (we were on 11.3 HF10) and have been having issues with our client certificate authentication. I have 2 APM policies configured that rely on the client certificate for authentication. The fallback in the event of a client certificate authentication failing is to prompt for alternative 2 factor authentication (using RSA) which works perfectly.
Since the upgrade, our clients connecting to VPN are failing with client certificate authentication and hence are constantly being prompted for the alternate authentication (some users don't have a token so you can see why when this fails it becomes a problem). The client certificate is valid, the bundle certificate is valid (so hence the trusted chain is valid).
When we initially performed the upgrade, we had a couple of machines that exhibited the problem but after 2 weeks we had a major influx of problems. The VPN configuration was set to "ignore" for the certificate but that has been changed to request. Doing this resolved the problem initially but then it came back about 5-6 hours later. We used to on 11.3 HF10 have a CRL process running every 10 minutes, this process doesn't work correctly on 11.6 (the script would run but not the cronjob to run every 10 minutes). I tried updating the CRL but that didn't seem to help.
Moving all the client connections back to 11.3 HF10 resolves the problem.
I have logged a case with F5 but so far no permanent solution has been found. Wonder if the forum might have come across this type of problem or knows what the potential problem is.
13 Replies
- Will_Adams_1995
Nimbostratus
The workstations connecting are Windows 7. The ca’s for the certificates are 2003. The workstations run ie11 (recently upgraded). The certificatr is an autoenrolled certificate. - Kevin_Stewart
Employee
What have you done to troubleshoot cert auth?
-
Removed the CRL check?
-
Enable debug logging for SSL?
-
Enable debug logging for APM?
-
Are you doing cert auth in the client SSL profile or APM on-demand cert auth?
-
Are you extracting any information from the cert before launching the VPN? If so, what and do you know if that's working?
-
- Will_Adams_1995
Nimbostratus
1 & 4) The CRL check I noted was no longer in the "certificate revocation list" under client authentication of the SSL client profile for the VPN connection. Originally (under 11.3) the client authentication options were set to use the parent profile "clientssl". On consultation with F5 support, we were advised to leave the clientssl profile as is and customise the client authentication for the VPN connection (which has been done). As it stands a loan device we have from F5 has this customisation and no CRL check but doesn't exhibit the problem.
2 & 3) Debug logs were sent to F5. I will likely do another debug check on SSL when I get a chance today. The APM logs previously had indicated that on the non-working machine, the certificate check returned a "1" and hence the certificate check process breaks.
5) Not sure where this would be configured or how to determine this.
- Kevin_Stewart
Employee
1 and 4: just want to confirm that on the broken platform, that there is no CRL checking done in the client SSL profile (or parent profile) applied to the VIP. Are you requesting the client certificate from the client authentication section of the client SSL profile, or are you doing an On-Demand Cert Auth in the APM visual policy?
2 and 3: Can you elaborate on "certificate check"?
5: I'm asking what you're doing with the certificate information. Do you do any sort of authentication in the access policy based on the client's certificate?
- Will_Adams_1995
Nimbostratus
Correct, there is no CRL checking occurring. We are doing on-demand cert auth in the APM Visual policy yes.
Regarding the "certificate check", I have enabled SSL debugging and this is what I am seeing (also on the loan device now). I extended handshake to 30 seconds in the interim.
*********** working example ***********
Session variable 'session.ssl.cert.valid' set to '1' Session variable 'session.ui.mode' set to '7' Executed agent '/Common/[apname]_act_ondemand_cert_auth_ag', return value 0 Following rule 'Successful' from item 'On-Demand Cert Auth' to item 'Assign Logon Variable for Logs' Username 'XXXX' Username 'xxxx' Executed agent '/Common/[apname]_act_variable_assign_ag', return value 0 Following rule 'fallback' from item 'Assign Logon Variable for Logs' to item 'Full VPN Access' Connectivity resource '/Common/[networkaccess]' assigned
*********** non working example ***********
Session variable 'session.clientcert.ssl_rehandshake_pending' set to '2' Session variable 'session.server.landinguri' set to '/' Session variable 'session.ssl.cert.exist' set to '0' Session variable 'session.ssl.cert.valid' set to '1' Session variable 'session.ui.mode' set to '7' Executed agent '/Common/[apname]_act_ondemand_cert_auth_ag', return value 0 Following rule 'fallback' from item 'On-Demand Cert Auth' to item 'Logon Page'
In the visual policy configuration, effectively the following happens:
On-Demand Cert Auth check is performed, this check is as follows:
Auth Mode is set to "Request" <-- so this seems to conflict with the AP profile setting. The branch rule has an expressions being "expr { [mcget {session.ssl.cert.valid}] == "0" }" If successful is moves to the assign logon variable for logs......and provides access to the VPN connection.
If not successful it moves to the fallback path where the Logon page is presented.
The logs above seem to indicate that while the same validation is being done and the same the end result is different.
- Kevin_Stewart
Employee
So it appears that the On-Demand Cert Auth agent is failing. Let's try a few things:
-
After the successful and fallback branches of the On-Demand Cert Auth agent, add some message boxes. After successful:
"Success: %{session.ssl.cert.subject}"After Fallback:
"Fallback: %{session.ssl.cert.subject}"If you see the certificate subject in the fallback branch message, then you know the client sent a cert, and that the SSL handshake succeeded.
-
You have "Request" set in the access policy On-Demand Cert Auth agent and the in the client SSL profile? If so, don't do that. The client SSL profile should be set to Ignore.
-
- Will_Adams_1995
Nimbostratus
On my loan device (which now no longer breaks!!!), I will get the certificate subject for the user account (FQDN, i.e. user, usergroups, domain). I noticed that if I set the CRL (as was being done on the 11.3 instance), this would cause a failure. The same certificate subject is returned.
- Kevin_Stewart
Employee
I will get the certificate subject for the user account (FQDN, i.e. user, usergroups, domain).
You're talking about in the message box, yes? This should be the subject value directly from the user's certificate.
I noticed that if I set the CRL (as was being done on the 11.3 instance), this would cause a failure.
It would probably be most helpful while troubleshooting to leave the CRL disabled.
The same certificate subject is returned
Let's focus on the broken platform. No CRL set, APM On-Demand Cert Auth agent set to request with message boxes following each branch, and the client SSL profile set to Ignore. So in this statement are you saying that you see the certificate subject in the fallback branch?
- Will_Adams_1995
Nimbostratus
yes
- Kevin_Stewart
Employee
Yes to what?
- Will_Adams_1995
Nimbostratus
Actually I got an alternate user to try and login via a Windows 8 machine.
On my session I have the following certificates expiring
"USERID" Expires 18/10/2016; 2/08/2016; 13/10/2016; 01/09/2016
"USERID2"
Expires 12/10/2016; 11/06/2016; 14/10/2016
With the loan device, USERID (me) works fine. I noticed that my certificate subject says "user_fqdn". This is without the CRL enabled.
With the load device, USERID2 breaks. I noticed that it does fail and there is no certificate subject. Hence this process goes to the fallback path and goes to the logon page option.
Help guide the future of your DevCentral Community!
What tools do you use to collaborate? (1min - anonymous)Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
