Keycloak as idp for APM

[JorenC]

Dear devcentral,

Has anyone successfully integrated keycloak as an OIDC backend for APM on F5? We are running v13.1 so this version should be able to use this feature, right?

So far I have successfully setup a provider using the autodiscover OpenID URI. Created a client application on the keycloak server with the client_id and secret. Next I’m somewhat confused on how to proceed? From what I read in the docs I need to configure the custom requests for keycloak. Though I can’t seem the find these.

Kind regards,

Joren

[Adaya]

i am also curious about this, has anyone got this to work?

[JorenC]

Adaya,

I’ve got this working for the password flow at the moment.

The documentation is there but not to easy to figure out what goes where.. (my personal opinion)

Still looking for some time to set up the authorization code flow.

Once I get this implemented i’ll try and add the setup on devcentral, stay tuned

Kind regards,

Joren

[Adaya]

Thanks for the reply. the specific implementation i’m actually looking for is the resource owner password flow. do you have the documentation you used to get this successfully working?

[JorenC]

Adaya,

I did end up using the LinkedIn lab guide.

F5 Cloud Docs

Do note, the most important piece is the creation of the custom requests.

Once you get this flow working using curl commands, it is pretty straight forward to set these up. Though I do not find the official doc clear.

I’ll see if I can upload some screenshots tomorrow (busy days at work ;))

Regards,

Joren

[Adaya]

This helps,

Appreciate the effort. If you have time, would it be too much to peak at your APM setup for the OP/IdP portion for Keycloak?

[avenging]

Hello, just wanted to know if someone finally manage to make this work with F5 APM and keycloak openid (or any custom on premise oAuth2 or openid provider) and authorisation code flow, custom request requests…

I try to do the same for OpenID auth behind an API with v15.1 and AWAF but without success…

[JorenC]

We do have it running here. But haven’t found the time to do full a write-up on the basic setup via APM..

Any specific questions you have regarding the APM integration?

[avenging]

OK I tried to use the f5 as a resource server to validate JWT token.

Did you manage to do it using the oauth provider configuration with OpenID Connect discovery like https://keycloakhost:keycloakport/auth/realms/master/.well-known/openid-configuration ?

Because for me, the discovery works but if I use the auto-JWT part, on save, f5 tell me about an issue of certificate (the CA bunble provided in config is the one of the certificate for the OpenId discovery link, may be not the same use by master realm to sign token or authorisation code). So I created the JWKS part manualy.

A request to the f5 APM VIP well redirect to the keycloack for AUTH but once authenticated the F5 tell that it cannot validate the token or auth code provided by the keycloak.

As the password grant type has been deprecated, did you manage to make it work using the authorisation code flow?

Also, wonder if for the step scope validation, you use the f5 type or have created some specific for keycloak?.

Yes, if someone have time to do a detailed write up on dev central on how to configure APM with OpenID as Client and/or Ressource server for using with keycloak or any on premise custom Provider, I think this will help a lot of people.

Thanks.

[JorenC]

yes we are indeed using the OIDC discovery.

If I’m not mistaking the CA bundle used here is just needed to connect to the keycloak endpoint. So like you said, it has nothing to do with the signing cert for tokens.

In our case, we have our keycloak running behind a virtual server on F5 with a public signed digicert certificate. So for the autodiscovery CA bundle I’ve imported the digicert bundle so F5 can validate this.

Regarding the flows, we had the password flow working more or less out of the box. The codeflow did have an issue. We kept receiving a 401 on the userinfo requests. Is this what you see? The keycloak logs will mention something like invalid_credentials.

In our case: After some tracing and the help of F5 support (big thanks to @Stanislaw Tsiarnouski) we noticed the issue is that keycloak adds the portnumber to the host header when calling introspect / userinfo. But the issuer in the token does not contain this portnumber.. This makes keycloak fail the check. (I still have a case open at RedHat for this but no solution on their part yet..)

So what I did to work around the issue is to remove the portnumber from the Host header if specific requests are passed to the VS of our server.

As a quick and dirty example:

if {[HTTP::method] eq "POST" && [HTTP::host] eq "keycloak.company.com:443"}{
        HTTP::header replace Host "keycloak.company.com"
    }

This then works for both the code flow and password flow.

Do note: All requests have been custom made for keycloak (Access > Federation : OAuth Client / Resource Server : Request). Based on the info from their doc and curl-command used to obtain tokens etc. I found this the least well documented part of the entire setup actually..

HTH

Joren

[avenging]

ok Thanks JorenC.

No I’m not able to go to scope validation part for the moment, I was just asking if we need to make custom request or if using the preconfigured F5 type was working.

I’m only at the token auth code validation internally on the F5 using JWKS or externally but I’m not able to made this works. I will post the details next week as I have no access to the F5 until monday.

[avenging]

OK, may be I will not have the issue about the port for introspect / userinfo because the token issuer already include the port number as my keycloak is published under port 8443.

I fixed some mistake about the JWKS and now it seems that token is well validated at OAuth Scope(Internal) step but failed at OAuth Scope(External) step:

Mar 3 15:20:48 f5poc notice apmd[12760]: 01490291:5: /Common/API-Authori-F5ResourceOAuth.app/API-Authori-F5ResourceOAuth_oauthResourceServer:Common:dcd521ef:/Common/API-Authori-F5ResourceOAuth.app/API-Authori-F5ResourceOAuth_oauthResourceServer_act_oauth_scope_0_ag: OAuth Scope: succeeded for jwt-provider-list '/Common/API-Authori-F5ResourceOAuth.app/API-Authori-F5ResourceOAuth_providerList_keycloak-provider'
 
Mar 3 15:20:48 f5poc notice apmd[12760]: 014902ae:5: /Common/API-Authori-F5ResourceOAuth.app/API-Authori-F5ResourceOAuth_oauthResourceServer:Common:dcd521ef:/Common/API-Authori-F5ResourceOAuth.app/API-Authori-F5ResourceOAuth_oauthResourceServer_act_oauth_scope_0_1_ag: OAuth Scope: getting list of scopes, associated with access_token, from server '/Common/API-Authori-F5ResourceOAuth.app/API-Authori-F5ResourceOAuth_oauthServer_keycloak-provider' (resource_server_id=F5-APM-Client)
 
Mar 3 15:20:48 f5poc err apmd[12760]: 01490290:3: /Common/API-Authori-F5ResourceOAuth.app/API-Authori-F5ResourceOAuth_oauthResourceServer:Common:dcd521ef:/Common/API-Authori-F5ResourceOAuth.app/API-Authori-F5ResourceOAuth_oauthResourceServer_act_oauth_scope_0_1_ag: OAuth Scope: failed for server '/Common/API-Authori-F5ResourceOAuth.app/API-Authori-F5ResourceOAuth_oauthServer_keycloak-provider' (resource_server_id=F5-APM-Client), error: HTTP error 401, Error: invalid_request: Authentication failed.

It seems that using the f5 request type is not working but I dont understand how to made custom request for keycloak IDP in f5 config.

Here is my https://keycloak.xxxxx.lu:8443/auth/realms/master/.well-known/openid-configuration (dont know if this help…) :

{
 
 "issuer": "https://keycloak.xxxxx.lu:8443/auth/realms/master",
 
 "authorization_endpoint": "https://keycloak.xxxxx.lu:8443/auth/realms/master/protocol/openid-connect/auth",
 
 "token_endpoint": "https://keycloak.xxxxx.lu:8443/auth/realms/master/protocol/openid-connect/token",
 
 "token_introspection_endpoint": "https://keycloak.xxxxx.lu:8443/auth/realms/master/protocol/openid-connect/token/introspect",
 
 "userinfo_endpoint": "https://keycloak.xxxxx.lu:8443/auth/realms/master/protocol/openid-connect/userinfo",
 
 "end_session_endpoint": "https://keycloak.xxxxx.lu:8443/auth/realms/master/protocol/openid-connect/logout",
 
 "jwks_uri": "https://keycloak.xxxxx.lu:8443/auth/realms/master/protocol/openid-connect/certs",
 
 "check_session_iframe": "https://keycloak.xxxxx.lu:8443/auth/realms/master/protocol/openid-connect/login-status-iframe.html",
 
 "grant_types_supported": [
 
   "authorization_code",
 
   "implicit",
 
   "refresh_token",
 
   "password",
 
   "client_credentials"
 
 ],
 
 "response_types_supported": [
 
   "code",
 
   "none",
 
   "id_token",
 
   "token",
 
   "id_token token",
 
   "code id_token",
 
   "code token",
 
   "code id_token token"
 
 ],
 
 "subject_types_supported": [
 
   "public",
 
   "pairwise"
 
 ],
 
 "id_token_signing_alg_values_supported": [
 
   "RS256"
 
 ],
 
 "id_token_encryption_alg_values_supported": [
 
   "RSA-OAEP",
 
   "RSA1_5"
 
 ],
 
 "id_token_encryption_enc_values_supported": [
 
   "A128GCM",
 
   "A128CBC-HS256"
 
 ],
 
 "userinfo_signing_alg_values_supported": [
 
   "RS256",
 
   "none"
 
 ],
 
 "request_object_signing_alg_values_supported": [
 
   "RS256",
 
   "none"
 
 ],
 
 "response_modes_supported": [
 
   "query",
 
   "fragment",
 
   "form_post"
 
 ],
 
 "registration_endpoint": "https://keycloak.xxxxx.lu:8443/auth/realms/master/clients-registrations/openid-connect",
 
 "token_endpoint_auth_methods_supported": [
 
   "private_key_jwt",
 
   "client_secret_basic",
 
   "client_secret_post",
 
   "client_secret_jwt"
 
 ],
 
 "token_endpoint_auth_signing_alg_values_supported": [
 
   "RS256"
 
 ],
 
 "claims_supported": [
 
   "aud",
 
   "sub",
 
   "iss",
 
   "auth_time",
 
   "name",
 
   "given_name",
 
   "family_name",
 
   "preferred_username",
 
   "email"
 
 ],
 
 "claim_types_supported": [
 
   "normal"
 
 ],
 
 "claims_parameter_supported": false,
 
 "scopes_supported": [
 
   "openid",
 
   "gatekeeperClientScope",
 
   "F5-APM-ClientScope",
 
   "address",
 
   "email",
 
   "microprofile-jwt",
 
   "offline_access",
 
   "phone",
 
   "profile",
 
   "roles",
 
   "web-origins"
 
 ],
 
 "request_parameter_supported": true,
 
 "request_uri_parameter_supported": true,
 
 "code_challenge_methods_supported": [
 
   "plain",
 
   "S256"
 
 ],
 
 "tls_client_certificate_bound_access_tokens": true,
 
 "introspection_endpoint": "https://keycloak.xxxxx.lu:8443/auth/realms/master/protocol/openid-connect/token/introspect"
 
}

Thanks in advance.

[avenging]

I go a little further on this but not yet working in F5.

For the moment using curl to target keycloak from CLI, I was ablle to have a correct reply from the keycloak for introspec/ user info:

curl --location --request POST 'https://keycloak.xxxxx.lu:8443/auth/realms/master/protocol/openid-connect/token/introspect' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJ5bzhwdzhZcHJ1UG96QzVuX0cyVHZ' \
--data-urlencode 'client_id=F5-APM-Client' \
--data-urlencode 'client_secret=db51def3-xxxx-xxx-xxx-xxx' \
--data-urlencode 'token_type_hint=access_token' \
--data-urlencode 'token=eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJ5bzhwdzhZcHJ1UG96QzVuX0cyVHZ'

but When I try to create a custom f5 keycloak scope request, I always have the same issue :  error: HTTP error 401, Error: invalid_request: Authentication failed.

So I dont understand why ? i try to made tcpdump to see what exactly F5 send to keycloak but, not helped for the moment.

I see client credentials error in keycloak logs :

But double checked the parameters > same used in curl …

[JorenC]

​@sebastien doucet​ your setup is indeed a bit different then mine but error seems related.

We are using ou F5 APM as a full OIDC client, redirecting the user to IDP logon page, requesting token etc..

Regarding the custom scope validation request, mine is more or less the same. Not so many params though, bare minimum only:

What really helped me is to place an iRule between the F5 and keycloak to capture the SSL keys so you can decode the HTTPS traffic in the TCP dump. From there you can decode the tokens etc using jwt.io and validate whats is in there.

#START
when CLIENTSSL_HANDSHAKE {
   log local0. "TCP source port client_side: [TCP::remote_port]"
   log local0. "RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]"
   log local0. "CLIENT_RANDOM [SSL::clientrandom] [SSL::sessionsecret]"
}
when SERVERSSL_HANDSHAKE {
   log local0. "TCP source port serv_side: [TCP::local_port]"
   log local0. "RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]"
   log local0. "CLIENT_RANDOM [SSL::clientrandom] [SSL::sessionsecret]"
}
#END

reference: myF5

Also: Is your keycloak server behind an F5 virtual server? If so, make sure you did config keycloak to understand it is running behind a loadbalancer. cfr: https://www.keycloak.org/docs/latest/server_installation/index.html#_setting-up-a-load-balancer-or-proxy

[avenging]

Thanks JorenC. I was finally able to made this working.

Capturing the RSA was not working due to RSA-DHE but I find another way using request logger in keycloak server : Logging requests with Undertow

And it seems that the f5 introspect request is not working as excepted in v15.1 because using the Keycloak scope request posted above the result was :

2020-03-09 13:06:30,504 INFO  [io.undertow.request.dump] (default task-15)
----------------------------REQUEST---------------------------
               URI=/auth/realms/master/protocol/openid-connect/token/introspect
 characterEncoding=null
     contentLength=1666
       contentType=[application/x-www-form-urlencoded]
            header=Proxy-Connection=Keep-Alive
            header=Accept=*/*
            header=User-Agent=F5 OAuth Client
            header=Authorization=Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJ5bzhwdzhZcHJ1UG96QzVuX0cyVHZVSVbOzSDkNlPyDi5A
            header=oauth_dns_resolver_name=/Common/172.xx.xx.xx
            header=oauth_serverssl_name=/Common/keycloak-publicssl
            header=Expect=100-continue
            header=Content-Type=application/x-www-form-urlencoded
            header=Content-Length=1666
            header=Host=keycloak.xxxxx.lu:8443
            locale=[]
            method=POST
          protocol=HTTP/1.1
       queryString=
        remoteAddr=/172.xx.xx.xx:17741
        remoteHost=172.xx.xx.xx
            scheme=https
              host=keycloak.xxxxx.lu:8443
        serverPort=8443
          isSecure=true
body=
%{session.oauth.client.last.access_token}=eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJ5bzhwdzhZcHJ1UG96QzVuX0cyVHZVSVbOzSDkNlPyDi5A
token_type_hint=access_token
--------------------------RESPONSE--------------------------
     contentLength=72
       contentType=application/json
            header=Connection=keep-alive
            header=Content-Type=application/json
            header=Content-Length=72
            header=Date=Mon, 09 Mar 2020 12:06:30 GMT
            status=401

And we see now that the F5 :

• send %{session.oauth.client.last.access_token} in place of token
• miss to send the client-id and client-secret parameters in the body
• send to much useless info to the keycloak like dns resolvers IP, backend server IP, ssl profile…

So I rebuild the request using only custom fields even for those by default like token, client-id…:

and then now it works !

I do the same for userinfo request also.

----------------------------REQUEST---------------------------
               URI=/auth/realms/master/protocol/openid-connect/token/introspect
 characterEncoding=null
     contentLength=1699
       contentType=[application/x-www-form-urlencoded]
            header=Proxy-Connection=Keep-Alive
            header=Accept=*/*
            header=User-Agent=F5 OAuth Client
            header=Authorization=Bearer hVhS2-trG8FsmeRyThTJ7zMGspLxZFGxV9kYjo
            header=oauth_dns_resolver_name=/Common/172.xxx.xx.xx
            header=oauth_serverssl_name=/Common/keycloak-publicssl
            header=Expect=100-continue
            header=Content-Length=1699
            header=Content-Type=application/x-www-form-urlencoded
            header=Host=keycloak.xxxxx.lu:8443
            locale=[]
            method=POST
          protocol=HTTP/1.1
       queryString=
        remoteAddr=/172.xx.xx.xx:47015
        remoteHost=172.xx.xx.xx
            scheme=https
              host=keycloak.xxxxx.lu:8443
        serverPort=8443
          isSecure=true
body=
client_id=F5-APM-Client
client_secret=db51def3-xxxx-xxxx-xxxx-xxxxx
token=hVhS2-trG8FsmeRyThTJ7zMGspLxZFGxV9kYjo
token_type_hint=access_token
--------------------------RESPONSE--------------------------
     contentLength=913
       contentType=application/json
            header=Connection=keep-alive
            header=Content-Type=application/json
            header=Content-Length=913
            header=Date=Mon, 09 Mar 2020 13:33:59 GMT
            status=200
 
==============================================================

Using the F5 as a Client and Ressource server for keycloak IDP is also working using f5 request without modification:

So I can confirm that another workaround for the issuer port problem you encounter is to publish your keycloak to a non standard port.

Thanks for your help, I will do a complete configuration guide when I have time available.