CIS and Kubernetes - Part 2: Install F5 Container ingress services

In our previous article, we have setup Kubernetes and calico with our BIG-IPs. Now we will setup F5 Container Ingress Services (F5 CIS) and deploy an ingress service.

Note: in this deployment, we will setup F5 CIS in the namespace kube-system

BIG-IPs Setup

Setup our Kubernetes partition

Before deploying F5 Container Ingress Services, we need to setup our BIG-IPs:

Setup a partition that CIS will use.
Install AS3

On EACH BIG-IP, you need to do the following:

In the GUI, go to System > User > Partitions List and click on the Create button

Create a partition called "kubernetes" and click Finished

Next we need to download the AS3 extension and install it on each BIG-IP.

Install AS3

Go to GitHub . Select the release with the "latest" tag and download the rpm.

Once you have the rpm, go to iApps > Package Management LX and click the Import button (you need to run BIG-IP v12.1 or later) on EACH BIG-IP

Chose your rpm and click Upload. Once the rpm has been uploaded and loaded, you should see this:

Our BIG-IPs are setup properly now.

Next, we will deploy F5 CIS

CIS Deployment

Connect to your Kubernetes cluster. We will need to setup the following:

Create a Kubernetes secret to store our BIG-IPs credentials
Setup a service account for CIS and setup RBAC
Setup our CIS configuration. We will need 1xCIS per BIG-IP (even when BIG-IP is setup as a cluster - we don't recommend automatic sync between BIG-IPs)
Deploy one CIS per BIG-IP

Store our BIG-IP credentials in a kubernetes secret

To store your credentials (login/password) in a kubernetes secret, you can run the following command (https://clouddocs.f5.com/containers/v2/kubernetes/kctlr-secrets.html#secret-bigip-login):

kubectl create secret generic bigip-login --namespace kube-system --from-literal=username=<your_login> --from-literal=password=<your_password>

In my setup, my credentials are the following:

Login: admin
Password: D3f4ult123

So we'll run the following command:

kubectl create secret generic bigip-login --namespace kube-system --from-literal=username=admin --from-literal=password=D3f4ult123

ubuntu@ip-10-1-1-4:~$ kubectl create secret generic bigip-login --namespace kube-system --from-literal=username=admin --from-literal=password=D3f4ult123
secret/bigip-login created

Create a Service Account /RBAC for CIS

Now we need to create our service account (https://clouddocs.f5.com/containers/v2/kubernetes/kctlr-app-install.html#set-up-rbac-authentication):

Run the following command: :

kubectl create serviceaccount bigip-ctlr -n kube-system

ubuntu@ip-10-1-1-4:~$ kubectl create serviceaccount bigip-ctlr -n kube-system
serviceaccount/bigip-ctlr created

Use your favorite editor to create this file:

f5-k8s-sample-rbac.yaml:

# for use in k8s clusters only
 # for OpenShift, use the OpenShift-specific examples
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
  name: bigip-ctlr-clusterrole
 rules:
 - apiGroups: ["", "extensions"]
  resources: ["nodes", "services", "endpoints", "namespaces", "ingresses", "pods"]
  verbs: ["get", "list", "watch"]
 - apiGroups: ["", "extensions"]
  resources: ["configmaps", "events", "ingresses/status"]
  verbs: ["get", "list", "watch", "update", "create", "patch"]
 - apiGroups: ["", "extensions"]
  resources: ["secrets"]
  resourceNames: ["<secret-containing-bigip-login>"]
  verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
  name: bigip-ctlr-clusterrole-binding
  namespace: kube-system
 roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: bigip-ctlr-clusterrole
 subjects:
 - apiGroup: ""
  kind: ServiceAccount
  name: bigip-ctlr
  namespace: kube-system

Deploy this config by running the following command:

kubectl apply -f f5-k8s-sample-rbac.yaml

ubuntu@ip-10-1-1-4:~$ kubectl apply -f f5-k8s-sample-rbac.yaml
clusterrole.rbac.authorization.k8s.io/bigip-ctlr-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/bigip-ctlr-clusterrole-binding created

Next step is to setup our CIS deployment files

Create our CIS deployment configurations

Use your favorite editor to create the following files:

setup_cis_bigip1.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: k8s-bigip1-ctlr-deployment
 namespace: kube-system
spec:
 selector:
   matchLabels:
     app: k8s-bigip1-ctlr
 # DO NOT INCREASE REPLICA COUNT
 replicas: 1
 template:
   metadata:
     labels:
       app: k8s-bigip1-ctlr
   spec:
     # Name of the Service Account bound to a Cluster Role with the required
     # permissions
     serviceAccountName: bigip-ctlr
     containers:
       - name: k8s-bigip-ctlr
         image: "f5networks/k8s-bigip-ctlr"
         env:
           - name: BIGIP_USERNAME
             valueFrom:
               secretKeyRef:
                 # Replace with the name of the Secret containing your login
                 # credentials
                 name: bigip-login
                 key: username
           - name: BIGIP_PASSWORD
             valueFrom:
               secretKeyRef:
                 # Replace with the name of the Secret containing your login
                 # credentials
                 name: bigip-login
                 key: password
         command: ["/app/bin/k8s-bigip-ctlr"]
         args: [
           # See the k8s-bigip-ctlr documentation for information about
           # all config options
           # https://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest
           "--bigip-username=$(BIGIP_USERNAME)",
           "--bigip-password=$(BIGIP_PASSWORD)",
           "--bigip-url=10.1.20.11",
           "--bigip-partition=kubernetes",
           "--insecure=true",
           "--pool-member-type=cluster",
           "--agent=as3"
           ]
     imagePullSecrets:
       # Secret that gives access to a private docker registry
       - name: f5-docker-images
       # Secret containing the BIG-IP system login credentials
       - name: bigip-login

setup_cis_bigip2.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: k8s-bigip2-ctlr-deployment
 namespace: kube-system
spec:
 selector:
   matchLabels:
     app: k8s-bigip2-ctlr
 # DO NOT INCREASE REPLICA COUNT
 replicas: 1
 template:
   metadata:
     labels:
       app: k8s-bigip2-ctlr
   spec:
     # Name of the Service Account bound to a Cluster Role with the required
     # permissions
     serviceAccountName: bigip-ctlr
     containers:
       - name: k8s-bigip-ctlr
         image: "f5networks/k8s-bigip-ctlr"
         env:
           - name: BIGIP_USERNAME
             valueFrom:
               secretKeyRef:
                 # Replace with the name of the Secret containing your login
                 # credentials
                 name: bigip-login
                 key: username
           - name: BIGIP_PASSWORD
             valueFrom:
               secretKeyRef:
                 # Replace with the name of the Secret containing your login
                 # credentials
                 name: bigip-login
                 key: password
         command: ["/app/bin/k8s-bigip-ctlr"]
         args: [
           # See the k8s-bigip-ctlr documentation for information about
           # all config options
           # https://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest
           "--bigip-username=$(BIGIP_USERNAME)",
           "--bigip-password=$(BIGIP_PASSWORD)",
           "--bigip-url=10.1.20.12",
           "--bigip-partition=kubernetes",
           "--insecure=true",
           "--pool-member-type=cluster",
           "--agent=as3"
           ]
     imagePullSecrets:
       # Secret that gives access to a private docker registry
       - name: f5-docker-images
       # Secret containing the BIG-IP system login credentials
       - name: bigip-login

To deploy our CIS, run the following commands:

kubectl apply -f setup_cis_bigip1.yaml

kubectl apply -f setup_cis_bigip2.yaml

You can make sure that they got deployed successfully:

kubectl get pods -n kube-system

You can review if it launched successfully with the commands:

kubectl logs <pod_name> -n kube system

ubuntu@ip-10-1-1-4:~$ kubectl logs k8s-bigip2-ctlr-deployment-6f674c8d58-bbzqv -n kube-system
2020/01/03 12:04:06 [INFO] Starting: Version: 1.12.0, BuildInfo: n2050-623590021
2020/01/03 12:04:06 [INFO] ConfigWriter started: 0xc0002bb950
2020/01/03 12:04:06 [INFO] Started config driver sub-process at pid: 14
2020/01/03 12:04:07 [INFO] NodePoller (0xc000b7c090) registering new listener: 0x11bfea0
2020/01/03 12:04:07 [INFO] NodePoller started: (0xc000b7c090)
2020/01/03 12:04:07 [INFO] Watching Ingress resources.
2020/01/03 12:04:07 [INFO] Watching ConfigMap resources.
2020/01/03 12:04:07 [INFO] Handling ConfigMap resource events.
2020/01/03 12:04:07 [INFO] Handling Ingress resource events.
2020/01/03 12:04:07 [INFO] Registered BigIP Metrics
2020/01/03 12:04:07 [INFO] [2020-01-03 12:04:07,663 __main__ INFO] entering inotify loop to watch /tmp/k8s-bigip-ctlr.config563475778/config.json
2020/01/03 12:04:08 [INFO] Successfully Sent the FDB Records

You may also check your BIG-IPs configuration: if CIS has been able to connect successfully, it will have created *another* partition called "kubernetes_AS3".This is explained here: https://clouddocs.f5.com/containers/v2/kubernetes/kctlr-use-as3-backend.html

CIS will create partition kubernetes_AS3 to store LTM objects such as pools, and virtual servers. FDB, and Static ARP entries are stored in kubernetes. These partitions should not be managed manually.

Application Deployment

In this section, we will do the following:

Deploy an application with 2 replicas (ie 2 instances of our app)
Setup an ingress service to connect to your application

Deploy our application and service

Create the following service and deployment configuration files:

f5-hello-world-app-http-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
 name: f5-hello-world
 namespace: default
spec:
 replicas: 2
 selector:
   matchLabels:
     app: f5-hello-world
 template:
   metadata:
     labels:
       app: f5-hello-world
   spec:
     containers:
     - env:
       - name: service_name
         value: f5-hello-world
       image: f5devcentral/f5-hello-world:latest
       imagePullPolicy: Always
       name: f5-hello-world
       ports:
       - containerPort: 8080
         protocol: TCP

f5-hello-world-app-http-service.yaml

apiVersion: v1
kind: Service
metadata:
 name: f5-hello-world
 namespace: default
 labels:
   app: f5-hello-world
spec:
 ports:
 - name: f5-hello-world
   port: 8080
   protocol: TCP
   targetPort: 8080
 type: NodePort
 selector:
   app: f5-hello-world

Apply your deployment and service:

kubectl apply -f f5-hello-world-app-http-deployment.yaml

kubectl apply -f f5-hello-world-app-http-service.yaml

ubuntu@ip-10-1-1-4:~$ kubectl apply -f f5-hello-world-app-http-deployment.yaml
deployment.apps/f5-hello-world created
ubuntu@ip-10-1-1-4:~$ kubectl apply -f f5-hello-world-app-http-service.yaml
service/f5-hello-world created

You can review your deployment and service with kubectl get :

ubuntu@ip-10-1-1-4:~$ kubectl get pods
NAME                             READY  STATUS   RESTARTS  AGE
f5-hello-world-847698f5c6-59mmn  1/1    Running  0         2m3s
f5-hello-world-847698f5c6-pcz9v  1/1    Running  0         2m3s
ubuntu@ip-10-1-1-4:~$ kubectl get svc
NAME            TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)         AGE
f5-hello-world  NodePort   10.97.83.208  <none>       8080:31380/TCP  2m
kubernetes      ClusterIP  10.96.0.1     <none>       443/TCP         63d

Define our ingress service

CIS ingress annotations can be reviewed here: https://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/v1.11/#ingress-resources

Create the following file:

f5-as3-ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: singleingress1
 namespace: default
 annotations:
 # See the k8s-bigip-ctlr documentation for information about
 # all Ingress Annotations
 # https://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest/#supported-ingress-annotations
   virtual-server.f5.com/ip: "10.1.10.80"
   virtual-server.f5.com/http-port: "443"
   virtual-server.f5.com/partition: "kubernetes_AS3"
   virtual-server.f5.com/health: '[{"path": "/", "send": "HTTP GET /", "interval": 5, "timeout": 10}]'
spec:
 backend:
   # The name of the Service you want to expose to external traffic
   serviceName: f5-hello-world
   servicePort: 8080

you can review the BIG-IP configuration by going into the kubernetes_AS3 partition