Using BIG-IP Kubernetes Gateway API Controller

Preparing Kubernetes Cluster

To use BIG-IP Kubernetes Gateway API Controller("Controller" in short), we need to prepare a Kubernetes Cluster. The community versions between 1.18 and 1.24 are well tested.

We have kinds of ways to setup a Kubernetes cluster, In this article, we use Rancher(https://www.rancher.com/) to setup a 7-node K8S with Flannel Network:

$ kubectl get node
NAME             STATUS   ROLES                      AGE   VERSION
k8s-lab-host-1   Ready    controlplane,etcd,worker   52m   v1.18.20
k8s-lab-host-2   Ready    worker                     25m   v1.18.20
k8s-lab-host-3   Ready    worker                     19m   v1.18.20
k8s-lab-host-4   Ready    worker                     23m   v1.18.20
k8s-lab-host-5   Ready    worker                     27m   v1.18.20
k8s-lab-host-6   Ready    worker                     26m   v1.18.20
k8s-lab-host-7   Ready    worker                     26m   v1.18.20

Deploying Gateway API CRDs

Gateway API is a set of CRDs developed by SIGNework(https://github.com/kubernetes/community/tree/master/sig-network).

Let’s use the following command to install Gateway API CRDs.

$ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.6.0/standard-install.yaml --validate=false
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
...
job.batch/gateway-api-admission created
job.batch/gateway-api-admission-patch created

By looking into the “standard-install.yaml” file of v0.6.0, we can see the definitions of resources: “GatewayClass” , “Gateway”, “HTTPRoute” and “ReferenceGrant”. We have also “experimental” as described https://gateway-api.sigs.k8s.io/concepts/versioning/#release-channels-eg-experimental-standard.

Notes that:

If you have network connection limit to gcr.io, you have to:

  • Manually download the images and upload them to k8s nodes for gcr.io connecting limit :

             gcr.io/k8s-staging-gateway-api/admission-server:v0.6.0

             k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1

  • Change the imagePullPolicy: Always to IfNotPresent in standard-install.yaml.

Use wget to download standard-install.yaml instead of kubectl apply -f <the remote yaml file>).

After the kubectl cmd, you can see the api-resources:  

$ kubectl api-resources
…
gatewayclasses                    gc           gateway.networking.k8s.io/v1beta1   false        GatewayClass
gateways                          gtw          gateway.networking.k8s.io/v1beta1   true         Gateway
httproutes                                     gateway.networking.k8s.io/v1beta1   true         HTTPRoute
referencegrants                   refgrant     gateway.networking.k8s.io/v1beta1   true         ReferenceGrant
…

And the running Gateway API admission server deployment:

$ kubectl get deploy -n gateway-system
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
gateway-api-admission-server   1/1     1            1           3m34s

Prepare a BIG-IP Instance

We have v14 – v17 classic BIG-IP well tested with basic LTM/NET abilities, like HTTP Gateway or Terminated HTTPS Gateway, vxlan or bgp networking.

The Controller is still evoluting, we’ll add more supports per customer needs.

In this article, we use “BIG-IP 15.1.0.5 Build 0.0.8 Point Release 5” for demonstration.

Setup BIG-IP for K8S CNI

To make BIG-IP as the Gateway for the business traffic, we need to setup BIG-IP’s networks for different CNI types. Currently, Controller supports 2 CNI types: Flannel and Calico, Cilium is on the way.

Flannel

For flannel network CNI, we need to setup the very vxlan/tunnel on BIG-IP, so that the BIG-IP can forward the traffic through the tunnel. In that situation, BIG-IP is configured as virtual node of K8S cluster.

The detail of Flannel network setup can be found here: https://gateway-api.f5se.io/deploy/#in-flannel-mode.

Calico

For calico network CNI, we configure BIG-IP as a BGP neighbor of K8S nodes.

The detail of Calico network setup can be found here: https://gateway-api.f5se.io/deploy/#in-calico-mode.

The above setup processes are step by step. Fortunately, we have encapsulate the setup process in the tool: https://github.com/f5devcentral/bigip-kubernetes-gateway/tree/master/cmd/tools/cni-utils. Users only need to provide configuration items, cni-utils would help to archive the CNI setup.

Now, we are ready to deploy and use Controller.

Deploy BIG-IP Kubernetes Gateway API Controller

Create ServiceAccount and ClusterRoleBinding

To monitor resource changes from Kubernetes cluster, we need to create a service account and bind it with proper role/access.

See https://github.com/f5devcentral/bigip-kubernetes-gateway/blob/master/deploy/1.clusterrole-and-binding.yaml for the integrated YAML file.

Deploy the Controller Pod

In https://github.com/f5devcentral/bigip-kubernetes-gateway/blob/master/deploy/3.deploy-bigip-kubernetes-gateway-controller.yaml, there are several resource types and names:

  • Secrets
    • bigip-login

The BIG-IP password for admin to do iControlREST requests.

We save it into a separate secret for security concern, it will be mounted as volume to bigip-kubenetes-gateway deployment.

  • webhook-server-cert

The CA/cert/key PEM used for webhook configuration. Thus, the connection between k8s and webhook server is encrypted and verified.

  • ValidatingWebhookConfiguration
    • validating-webhook-configuration

The webhook configuration, in this configuration, we define a before-hand way to validate users’ resource request before doing real changes to api-server. The configuration includes many items, like, server ca bundle, server port and endpoints, etc.

Each resource has a separate endpoint for resource validation.

This webhook configuration is different with gateway-api-admission-server, it focuses on resource relationship validation. For example, validating the required GatewayClass existence before creating a Gateway.

The validation is configurable to enable or disable.

  • ConfigMap
    • bigip-kubernetes-gateway-configmap

This is the configuration for the downstream BIG-IP. Will be mounted as volume to bigip-kubernetes-gateway deployment.

  • Deployment
    • bigip-kubernetes-gateway

The deployment and pod where the controller process is running. The Controller is released as docker images to: https://hub.docker.com/r/f5devcentral/bigip-kubernetes-gateway.

We can find the default startup parameters and necessary input volumes in the “spec:” section. Users can adjust the parameters here for their deployments.

  • Service
    • bigip-kubernetes-gateway

The bigip-kubernetes-gateway Service is used to expose the Controller at:

8080: Prometheus metrics collection port

8081: health probing port

9443: webhook server port

Because the Controller is run with NodePort service type, no matter it’s in or out of Cluster, it can be accessed with the very ports.

Installation and Startup

To install and start controller, just run the command:

$ kubectl apply -f deploy/3.deploy-bigip-kubernetes-gateway-controller.yaml

Controller Parameters

The parameters we use to start controller is:

  "--controller-name=f5.io/gateway-controller-name",
  "--bigip-config-directory=/bigip-config",
  "--bigip-credential-directory=/bigip-credential",
  "--certificate-directory=/tmp/k8s-webhook-server/serving-certs"

By default, we need not to change them.

See the full list are as below command:

$ docker run f5devcentral/bigip-kubernetes-gateway:v0.2.1-20230411 /bigip-kubernetes-gateway-controller-linux --help

Verify the Startup

After starting the Controller, we can use the following command to monitor the logs:

$ kubectl logs -f deployment/bigip-kubernetes-gateway -c bigip-kubernetes-gateway-pod -n kube-system

Deploying a Gateway Instance

We have several usecases/examples in https://github.com/f5devcentral/bigip-kubernetes-gateway/tree/master/examples/.

As the start, you can read the simple Gateway case: https://github.com/f5devcentral/bigip-kubernetes-gateway/tree/master/examples/simple-gateway.

More Topics

HA

The controller is run as a standalone program in K8S, it has no HA feature.

When the program is terminated unexpectedly, K8S would pull up a new pod automatically to continue the Controller work.

After restarting, the resource update events would be sent to Controller again in batch from K8S. However, Controller can detect that the corresponding resources on BIG-IP are ready, no need to redeploy again.

For HA of BIG-IPs, one Controller can handle multiple BIG-IPs. They can be in HA active-standby peer.

Performance

Controller use BIG-IP iControl Rest and transaction feature to accomplish the resource deployment.

It’s a incremental way when the new resource deployment request comes. Controller will detect which part need to deploy and which part doesn’t, by comparing the existing resources and resource requests.

It’s faster than AS3 way, because, AS3 do more complex checks and validation. However, the way Controller is using only focuses on Gateway API related resources.

Prometheus

The Controller has been integrated with Prometheus, by accessing http://<controller>:8080/metrics and http://<controller>:8080/stats, we can see the metrics exposed to Prometheus. Currently, we have some metrics for runtime-manager work tracing(/metrics) and function performance tracing(/stats). In the future, we may add more metrics per customer needs.

Video Demos

Users can find more video demos from YouTube: https://www.youtube.com/playlist?list=PLAg2ep4I_zspyhcFj90n8vx_Lo8AUUyWT

BIG-IP Kubernetes Gateway API Controller Setup and simple gateway demonstration.

This video demonstrate the cross namespace routing implementation by BIGIP-Kubernetes-Gateway. See https://gateway-api.sigs.k8s.io/guide...

Splitting Traffics with HTTPRoute's multiple backendRefs.

Published Apr 14, 2023
Version 1.0
application delivery
cloud
devops
No CommentsBe the first to comment