Authenticating Kubernetes

Authenticating Kubernetes: OAuth and mTLS; Oh My!

How do you know what your external users and services are doing in your Kubernetes cluster? Using BIG-IP Access Policy Manager and Application Security Manager we can create a secure method of authenticating users and provide visibility and control of a Kubernetes cluster.

Auth Methods in Kubernetes

Kubernetes has a myriad of methods to authenticate end-users and services including:

• mTLS
• Authenticating Proxy
• OpenID Connect (OIDC)

mTLS

Mutual TLS authentication uses client-side certificates to authenticate to a service. When you use kubectl with Kubernetes it is a common pattern to store the contents of a client certificate with the client and use it for authenticating to the cluster.

Authenticating Proxy

Kubernetes expects that the reverse proxy (i.e. BIG-IP) is configured to use client-side certificates for authenticating to the cluster. To distinguish different users or services the proxy is expected to provide additional HTTP headers that identify the users (i.e. X-Remote-User).

OpenID Connect

OpenID Connect is a flavor of OAuth2 that makes use of a token to authenticate a service. When you authenticate using OIDC you receive three types of tokens:

• access_token
• refresh_token
• id_token

These tokens are in JWT format (base64 encoded/signed JSON) and the contents of them look something like (pseudo-token):

{
  "token_type": "Bearer",
  "scope": "openid",  
  "iss": "https://oauthas.f5demo.com/f5-oauth2/v1",
  "sub": "eric",
  "iat": 1551226765,
  "exp": 1551227065,
  "nbf": 1551226465
}

When Kubernetes uses OIDC it expects that the “id_token” is being used. This looks similar to an access_token.

{
  "iss": "https://oauthas.f5demo.com/f5-oauth2/v1",
  "aud": [
    "eb209a8279b39cc00d9824a7d40002b5f02c9fe2a7624b5c"
  ],
  "sub": "eric",
  "iat": 1551226918,
  "exp": 1551227218
}

Kubernetes expects the “id_token” to be used when authenticating to the API.

Using BIG-IP to Authenticate to Kubernetes

Authentication Proxy

The easiest method of using BIG-IP APM to Authenticate to Kubernetes is to configure it as an Authentication Proxy. This requires you to import a trusted certificate into a serverssl profile and setup an API Protection profile that inserts a custom header (X-Remote-User).

An example policy that uses the LocalDB (could also be Active Directory, Radius, etc…)

Example Header Insertion

OpenID Connect

BIG-IP APM can be used either as an OAuth Authorization Server and/or a Resource Server. When configured as Authorization Server BIG-IP is responsible for authenticating users and generating JWT tokens. Acting as a Resource Server BIG-IP would validate JWT tokens generated by an external authorization server (i.e. another BIG-IP or OAuth provider).

Once you’ve configured Kubernetes to use your OIDC Authorization Server you can create an API Protection profile like the one that we created for Basic Auth.

The OAuth Bearer SSO configuration looks like the following.

mTLS

In the previous two examples we performed identity transformation from Basic Auth (username/password) to header insertion (X-Remote-User) and converting an OAuth “access_token” to an “id_token”. Using mTLS we could take a similar approach to convert the identity of an end-user from a client-certificate to use either header insertion or an OAuth bearer token.

We can also support a transformation of converting a client-side certificate that is used to authenticate to the BIG-IP to generate a dynamic/ephemeral client-side certificate that is used to authenticate to Kubernetes using mTLS. This preserves the end-to-end use of mTLS and still provide visibility of the traffic that is traversing the BIG-IP.

Due to how mTLS works the device that is performing the authentication needs to terminate the TLS connection. When you have a proxy  the typical options are:

1. Do not terminate TLS (only proxy TCP)
2. Terminate TLS and authenticate the client-certificate, forward the identity in an HTTP header

              i.e. X-Cert-Subject: foo, X-Cert-Contents: …

BIG-IP LTM supports a feature C3D, Client Certificate Constrained Delegation, that enables you to have the BIG-IP terminate the TLS connection, authenticate the client certificate, then generate a new client-side certificate with similar attributes to the original certificate. This allows backend devices/services to continue to use mTLS as long as they trust the CA that the BIG-IP is using to generate the ephemeral client-side certificates (default TTL 1 hour).

To enable this feature you need to update both your client and server SSL profiles to use C3D.

clientssl

serverssl

Oh my!

We’ve looked at 3 different ways of authenticating to Kubernetes using BIG-IP, but what if we wanted to combine them all? Also, how do we know what actions a user is taking while connected to the Kubernetes API?

We can combine the previous methods into a single API Protection Policy that uses Basic, mTLS, or OAuth for authentication. An ASM policy can also further secure and audit the use of the Kubernetes API.

The combined policy.

Corresponding ASM logs

Basic Auth

OAuth

mTLS

Denying Access

Earlier we created an API Protection Profile without importing a Swagger / OpenAPI file. You can also import your cluster’s API definitions to create fine-grained access and/or block specific access.

The following examples prevents read (GET) access to Kubernetes secrets.  This could augment existing ClusterRole permissions.

Kubernetes Dashboard

The previous examples focused on authenticating the Kubernetes API, but this can also be used to authenticate to the Kubernetes Dashboard (web application) as well. This can address a limitation of the dashboard of only being able to consume tokens as an authentication method. The following is from "Authenticating - Kubernetes"  as of version 1.12

There’s no easy way to authenticate to the Kubernetes dashboard without using the

kubectl proxy

command or a reverse proxy that injects the

id_token 

Using a similar strategy that was used previously we can create an Access Policy that transforms mTLS, OIDC access_token, and Username/Password into OIDC id_tokens.

An example of using username/password

(i.e. Active Directory converted to OAuth Bearer/JWT token)

End-User Authenticates using username and password.

End-user sees role based access to Kubernetes based on token

End-user identity captured in WAF logs

Example of using mTLS

Subject of Cert is captured in logs

Pretty Fancy

We’ve gone through several iterations of ways to authenticate to Kubernetes via the BIG-IP methods to audit and enforce access to the cluster. These ideas are adopted from previous DevCentral articles about using ASM policies to protect a cluster and how to use API WAF policies in front of API gateways. Other resources used to create this article are a lab from the 2018 Agility event and a very nice YouTube video about APM API Protection.

This article focused on Kubernetes, but these methods/approaches can be used for any API or Application that has specialized access requirements. BIG-IP makes it possible to transform the identity of your user into the format that you need it, monitor their access, and enforce policy.