\n\n
To use BIG-IP Kubernetes Gateway API Controller(\"Controller\" in short), we need to prepare a Kubernetes Cluster. The community versions between 1.18 and 1.24 are well tested.
\n
We have kinds of ways to setup a Kubernetes cluster, In this article, we use Rancher(https://www.rancher.com/) to setup a 7-node K8S with Flannel Network:
\n
$ kubectl get node\nNAME STATUS ROLES AGE VERSION\nk8s-lab-host-1 Ready controlplane,etcd,worker 52m v1.18.20\nk8s-lab-host-2 Ready worker 25m v1.18.20\nk8s-lab-host-3 Ready worker 19m v1.18.20\nk8s-lab-host-4 Ready worker 23m v1.18.20\nk8s-lab-host-5 Ready worker 27m v1.18.20\nk8s-lab-host-6 Ready worker 26m v1.18.20\nk8s-lab-host-7 Ready worker 26m v1.18.20
\n\n
Gateway API is a set of CRDs developed by SIGNework(https://github.com/kubernetes/community/tree/master/sig-network).
\n
Let’s use the following command to install Gateway API CRDs.
\n
$ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.6.0/standard-install.yaml --validate=false\ncustomresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created\n...\njob.batch/gateway-api-admission created\njob.batch/gateway-api-admission-patch created
\n
By looking into the “standard-install.yaml” file of v0.6.0, we can see the definitions of resources: “GatewayClass” , “Gateway”, “HTTPRoute” and “ReferenceGrant”. We have also “experimental” as described https://gateway-api.sigs.k8s.io/concepts/versioning/#release-channels-eg-experimental-standard.
\n
Notes that:
\n
If you have network connection limit to gcr.io, you have to:
\n
\n- Manually download the images and upload them to k8s nodes for gcr.io connecting limit :
\n
\n
gcr.io/k8s-staging-gateway-api/admission-server:v0.6.0
\n
k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
\n
\n- Change the imagePullPolicy: Always to IfNotPresent in standard-install.yaml.
\n
\n
Use wget to download standard-install.yaml instead of kubectl apply -f <the remote yaml file>).
\n
After the kubectl cmd, you can see the api-resources:
\n
$ kubectl api-resources\n…\ngatewayclasses gc gateway.networking.k8s.io/v1beta1 false GatewayClass\ngateways gtw gateway.networking.k8s.io/v1beta1 true Gateway\nhttproutes gateway.networking.k8s.io/v1beta1 true HTTPRoute\nreferencegrants refgrant gateway.networking.k8s.io/v1beta1 true ReferenceGrant\n…
\n
And the running Gateway API admission server deployment:
\n
$ kubectl get deploy -n gateway-system\nNAME READY UP-TO-DATE AVAILABLE AGE\ngateway-api-admission-server 1/1 1 1 3m34s
\n\n
We have v14 – v17 classic BIG-IP well tested with basic LTM/NET abilities, like HTTP Gateway or Terminated HTTPS Gateway, vxlan or bgp networking.
\n
The Controller is still evoluting, we’ll add more supports per customer needs.
\n
In this article, we use “BIG-IP 15.1.0.5 Build 0.0.8 Point Release 5” for demonstration.
\n\n
To make BIG-IP as the Gateway for the business traffic, we need to setup BIG-IP’s networks for different CNI types. Currently, Controller supports 2 CNI types: Flannel and Calico, Cilium is on the way.
\n\n
For flannel network CNI, we need to setup the very vxlan/tunnel on BIG-IP, so that the BIG-IP can forward the traffic through the tunnel. In that situation, BIG-IP is configured as virtual node of K8S cluster.
\n
The detail of Flannel network setup can be found here: https://gateway-api.f5se.io/deploy/#in-flannel-mode.
\n\n
For calico network CNI, we configure BIG-IP as a BGP neighbor of K8S nodes.
\n
The detail of Calico network setup can be found here: https://gateway-api.f5se.io/deploy/#in-calico-mode.
\n
The above setup processes are step by step. Fortunately, we have encapsulate the setup process in the tool: https://github.com/f5devcentral/bigip-kubernetes-gateway/tree/master/cmd/tools/cni-utils. Users only need to provide configuration items, cni-utils would help to archive the CNI setup.
\n
Now, we are ready to deploy and use Controller.
\n\n\n
To monitor resource changes from Kubernetes cluster, we need to create a service account and bind it with proper role/access.
\n
See https://github.com/f5devcentral/bigip-kubernetes-gateway/blob/master/deploy/1.clusterrole-and-binding.yaml for the integrated YAML file.
\n\n
In https://github.com/f5devcentral/bigip-kubernetes-gateway/blob/master/deploy/3.deploy-bigip-kubernetes-gateway-controller.yaml, there are several resource types and names:
\n
\n
The BIG-IP password for admin to do iControlREST requests.
\n
We save it into a separate secret for security concern, it will be mounted as volume to bigip-kubenetes-gateway deployment.
\n
\n
The CA/cert/key PEM used for webhook configuration. Thus, the connection between k8s and webhook server is encrypted and verified.
\n
\n- ValidatingWebhookConfiguration\n
\n- validating-webhook-configuration
\n
\n \n
\n
The webhook configuration, in this configuration, we define a before-hand way to validate users’ resource request before doing real changes to api-server. The configuration includes many items, like, server ca bundle, server port and endpoints, etc.
\n
Each resource has a separate endpoint for resource validation.
\n
This webhook configuration is different with gateway-api-admission-server, it focuses on resource relationship validation. For example, validating the required GatewayClass existence before creating a Gateway.
\n
The validation is configurable to enable or disable.
\n
\n- ConfigMap\n
\n- bigip-kubernetes-gateway-configmap
\n
\n \n
\n
This is the configuration for the downstream BIG-IP. Will be mounted as volume to bigip-kubernetes-gateway deployment.
\n
\n- Deployment\n
\n- bigip-kubernetes-gateway
\n
\n \n
\n
The deployment and pod where the controller process is running. The Controller is released as docker images to: https://hub.docker.com/r/f5devcentral/bigip-kubernetes-gateway.
\n
We can find the default startup parameters and necessary input volumes in the “spec:” section. Users can adjust the parameters here for their deployments.
\n
\n- Service\n
\n- bigip-kubernetes-gateway
\n
\n \n
\n
The bigip-kubernetes-gateway Service is used to expose the Controller at:
\n
8080: Prometheus metrics collection port
\n
8081: health probing port
\n
9443: webhook server port
\n
Because the Controller is run with NodePort service type, no matter it’s in or out of Cluster, it can be accessed with the very ports.
\n\n
To install and start controller, just run the command:
\n
$ kubectl apply -f deploy/3.deploy-bigip-kubernetes-gateway-controller.yaml
\n\n
The parameters we use to start controller is:
\n
\"--controller-name=f5.io/gateway-controller-name\",\n \"--bigip-config-directory=/bigip-config\",\n \"--bigip-credential-directory=/bigip-credential\",\n \"--certificate-directory=/tmp/k8s-webhook-server/serving-certs\"
\n
By default, we need not to change them.
\n
See the full list are as below command:
\n
$ docker run f5devcentral/bigip-kubernetes-gateway:v0.2.1-20230411 /bigip-kubernetes-gateway-controller-linux --help
\n\n
After starting the Controller, we can use the following command to monitor the logs:
\n
$ kubectl logs -f deployment/bigip-kubernetes-gateway -c bigip-kubernetes-gateway-pod -n kube-system
\n\n
We have several usecases/examples in https://github.com/f5devcentral/bigip-kubernetes-gateway/tree/master/examples/.
\n
As the start, you can read the simple Gateway case: https://github.com/f5devcentral/bigip-kubernetes-gateway/tree/master/examples/simple-gateway.
\n\n\n
The controller is run as a standalone program in K8S, it has no HA feature.
\n
When the program is terminated unexpectedly, K8S would pull up a new pod automatically to continue the Controller work.
\n
After restarting, the resource update events would be sent to Controller again in batch from K8S. However, Controller can detect that the corresponding resources on BIG-IP are ready, no need to redeploy again.
\n
For HA of BIG-IPs, one Controller can handle multiple BIG-IPs. They can be in HA active-standby peer.
\n\n
Controller use BIG-IP iControl Rest and transaction feature to accomplish the resource deployment.
\n
It’s a incremental way when the new resource deployment request comes. Controller will detect which part need to deploy and which part doesn’t, by comparing the existing resources and resource requests.
\n
It’s faster than AS3 way, because, AS3 do more complex checks and validation. However, the way Controller is using only focuses on Gateway API related resources.
\n\n
The Controller has been integrated with Prometheus, by accessing http://<controller>:8080/metrics and http://<controller>:8080/stats, we can see the metrics exposed to Prometheus. Currently, we have some metrics for runtime-manager work tracing(/metrics) and function performance tracing(/stats). In the future, we may add more metrics per customer needs.
\n\n
Users can find more video demos from YouTube: https://www.youtube.com/playlist?list=PLAg2ep4I_zspyhcFj90n8vx_Lo8AUUyWT
\n
\n
BIG-IP Kubernetes Gateway API Controller Setup and simple gateway demonstration.
\n
\n
\n
This video demonstrate the cross namespace routing implementation by BIGIP-Kubernetes-Gateway. See https://gateway-api.sigs.k8s.io/guide...
\n
\n
\n
Splitting Traffics with HTTPRoute's multiple backendRefs.
\n
\n
\n
Preparing Kubernetes Cluster
\n
To use BIG-IP Kubernetes Gateway API Controller(\"Controller\" in short), we need to prepare a Kubernetes Cluster. The community versions between 1.18 and 1.24 are well tested.
\n
We have kinds of ways to setup a Kubernetes cluster, In this article, we use Rancher(https://www.rancher.com/) to setup a 7-node K8S with Flannel Network:
\n
$ kubectl get node\nNAME STATUS ROLES AGE VERSION\nk8s-lab-host-1 Ready controlplane,etcd,worker 52m v1.18.20\nk8s-lab-host-2 Ready worker 25m v1.18.20\nk8s-lab-host-3 Ready worker 19m v1.18.20\nk8s-lab-host-4 Ready worker 23m v1.18.20\nk8s-lab-host-5 Ready worker 27m v1.18.20\nk8s-lab-host-6 Ready worker 26m v1.18.20\nk8s-lab-host-7 Ready worker 26m v1.18.20\n
Deploying Gateway API CRDs
\n
Gateway API is a set of CRDs developed by SIGNework(https://github.com/kubernetes/community/tree/master/sig-network).
\n
Let’s use the following command to install Gateway API CRDs.
\n
$ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.6.0/standard-install.yaml --validate=false\ncustomresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created\n...\njob.batch/gateway-api-admission created\njob.batch/gateway-api-admission-patch created\n
By looking into the “standard-install.yaml” file of v0.6.0, we can see the definitions of resources: “GatewayClass” , “Gateway”, “HTTPRoute” and “ReferenceGrant”. We have also “experimental” as described https://gateway-api.sigs.k8s.io/concepts/versioning/#release-channels-eg-experimental-standard.
\n
Notes that:
\n
If you have network connection limit to gcr.io, you have to:
\n
\n- Manually download the images and upload them to k8s nodes for gcr.io connecting limit :
\n
\n
gcr.io/k8s-staging-gateway-api/admission-server:v0.6.0
\n
k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
\n
\n- Change the imagePullPolicy: Always to IfNotPresent in standard-install.yaml.
\n
\n
Use wget to download standard-install.yaml instead of kubectl apply -f <the remote yaml file>).
\n
After the kubectl cmd, you can see the api-resources:
\n
$ kubectl api-resources\n…\ngatewayclasses gc gateway.networking.k8s.io/v1beta1 false GatewayClass\ngateways gtw gateway.networking.k8s.io/v1beta1 true Gateway\nhttproutes gateway.networking.k8s.io/v1beta1 true HTTPRoute\nreferencegrants refgrant gateway.networking.k8s.io/v1beta1 true ReferenceGrant\n…\n
And the running Gateway API admission server deployment:
\n
$ kubectl get deploy -n gateway-system\nNAME READY UP-TO-DATE AVAILABLE AGE\ngateway-api-admission-server 1/1 1 1 3m34s\n
Prepare a BIG-IP Instance
\n
We have v14 – v17 classic BIG-IP well tested with basic LTM/NET abilities, like HTTP Gateway or Terminated HTTPS Gateway, vxlan or bgp networking.
\n
The Controller is still evoluting, we’ll add more supports per customer needs.
\n
In this article, we use “BIG-IP 15.1.0.5 Build 0.0.8 Point Release 5” for demonstration.
\n
Setup BIG-IP for K8S CNI
\n
To make BIG-IP as the Gateway for the business traffic, we need to setup BIG-IP’s networks for different CNI types. Currently, Controller supports 2 CNI types: Flannel and Calico, Cilium is on the way.
\n
Flannel
\n
For flannel network CNI, we need to setup the very vxlan/tunnel on BIG-IP, so that the BIG-IP can forward the traffic through the tunnel. In that situation, BIG-IP is configured as virtual node of K8S cluster.
\n
The detail of Flannel network setup can be found here: https://gateway-api.f5se.io/deploy/#in-flannel-mode.
\n
Calico
\n
For calico network CNI, we configure BIG-IP as a BGP neighbor of K8S nodes.
\n
The detail of Calico network setup can be found here: https://gateway-api.f5se.io/deploy/#in-calico-mode.
\n
The above setup processes are step by step. Fortunately, we have encapsulate the setup process in the tool: https://github.com/f5devcentral/bigip-kubernetes-gateway/tree/master/cmd/tools/cni-utils. Users only need to provide configuration items, cni-utils would help to archive the CNI setup.
\n
Now, we are ready to deploy and use Controller.
\n
Deploy BIG-IP Kubernetes Gateway API Controller
\n
Create ServiceAccount and ClusterRoleBinding
\n
To monitor resource changes from Kubernetes cluster, we need to create a service account and bind it with proper role/access.
\n
See https://github.com/f5devcentral/bigip-kubernetes-gateway/blob/master/deploy/1.clusterrole-and-binding.yaml for the integrated YAML file.
\n
Deploy the Controller Pod
\n
In https://github.com/f5devcentral/bigip-kubernetes-gateway/blob/master/deploy/3.deploy-bigip-kubernetes-gateway-controller.yaml, there are several resource types and names:
\n
\n
The BIG-IP password for admin to do iControlREST requests.
\n
We save it into a separate secret for security concern, it will be mounted as volume to bigip-kubenetes-gateway deployment.
\n
\n
The CA/cert/key PEM used for webhook configuration. Thus, the connection between k8s and webhook server is encrypted and verified.
\n
\n- ValidatingWebhookConfiguration\n
\n- validating-webhook-configuration
\n
\n \n
\n
The webhook configuration, in this configuration, we define a before-hand way to validate users’ resource request before doing real changes to api-server. The configuration includes many items, like, server ca bundle, server port and endpoints, etc.
\n
Each resource has a separate endpoint for resource validation.
\n
This webhook configuration is different with gateway-api-admission-server, it focuses on resource relationship validation. For example, validating the required GatewayClass existence before creating a Gateway.
\n
The validation is configurable to enable or disable.
\n
\n- ConfigMap\n
\n- bigip-kubernetes-gateway-configmap
\n
\n \n
\n
This is the configuration for the downstream BIG-IP. Will be mounted as volume to bigip-kubernetes-gateway deployment.
\n
\n- Deployment\n
\n- bigip-kubernetes-gateway
\n
\n \n
\n
The deployment and pod where the controller process is running. The Controller is released as docker images to: https://hub.docker.com/r/f5devcentral/bigip-kubernetes-gateway.
\n
We can find the default startup parameters and necessary input volumes in the “spec:” section. Users can adjust the parameters here for their deployments.
\n
\n- Service\n
\n- bigip-kubernetes-gateway
\n
\n \n
\n
The bigip-kubernetes-gateway Service is used to expose the Controller at:
\n
8080: Prometheus metrics collection port
\n
8081: health probing port
\n
9443: webhook server port
\n
Because the Controller is run with NodePort service type, no matter it’s in or out of Cluster, it can be accessed with the very ports.
\n
Installation and Startup
\n
To install and start controller, just run the command:
\n
$ kubectl apply -f deploy/3.deploy-bigip-kubernetes-gateway-controller.yaml\n
Controller Parameters
\n
The parameters we use to start controller is:
\n
\"--controller-name=f5.io/gateway-controller-name\",\n \"--bigip-config-directory=/bigip-config\",\n \"--bigip-credential-directory=/bigip-credential\",\n \"--certificate-directory=/tmp/k8s-webhook-server/serving-certs\"\n
By default, we need not to change them.
\n
See the full list are as below command:
\n
$ docker run f5devcentral/bigip-kubernetes-gateway:v0.2.1-20230411 /bigip-kubernetes-gateway-controller-linux --help\n
Verify the Startup
\n
After starting the Controller, we can use the following command to monitor the logs:
\n
$ kubectl logs -f deployment/bigip-kubernetes-gateway -c bigip-kubernetes-gateway-pod -n kube-system
\n
Deploying a Gateway Instance
\n
We have several usecases/examples in https://github.com/f5devcentral/bigip-kubernetes-gateway/tree/master/examples/.
\n
As the start, you can read the simple Gateway case: https://github.com/f5devcentral/bigip-kubernetes-gateway/tree/master/examples/simple-gateway.
\n
More Topics
\n
HA
\n
The controller is run as a standalone program in K8S, it has no HA feature.
\n
When the program is terminated unexpectedly, K8S would pull up a new pod automatically to continue the Controller work.
\n
After restarting, the resource update events would be sent to Controller again in batch from K8S. However, Controller can detect that the corresponding resources on BIG-IP are ready, no need to redeploy again.
\n
For HA of BIG-IPs, one Controller can handle multiple BIG-IPs. They can be in HA active-standby peer.
\n
Performance
\n
Controller use BIG-IP iControl Rest and transaction feature to accomplish the resource deployment.
\n
It’s a incremental way when the new resource deployment request comes. Controller will detect which part need to deploy and which part doesn’t, by comparing the existing resources and resource requests.
\n
It’s faster than AS3 way, because, AS3 do more complex checks and validation. However, the way Controller is using only focuses on Gateway API related resources.
\n
Prometheus
\n
The Controller has been integrated with Prometheus, by accessing http://<controller>:8080/metrics and http://<controller>:8080/stats, we can see the metrics exposed to Prometheus. Currently, we have some metrics for runtime-manager work tracing(/metrics) and function performance tracing(/stats). In the future, we may add more metrics per customer needs.
\n
Video Demos
\n
Users can find more video demos from YouTube: https://www.youtube.com/playlist?list=PLAg2ep4I_zspyhcFj90n8vx_Lo8AUUyWT
\n
\n
BIG-IP Kubernetes Gateway API Controller Setup and simple gateway demonstration.
\n
\n
\n
This video demonstrate the cross namespace routing implementation by BIGIP-Kubernetes-Gateway. See https://gateway-api.sigs.k8s.io/guide...
\n
\n
\n
Splitting Traffics with HTTPRoute's multiple backendRefs.
\n
\n
![\"F5](\"https://www.f5.com/content/dam/f5/f5-logo.svg\"){kind=logo}
