SSL offload would need a SSL cert on the F5. - https://gitlab.com/abgmbh/kitchen_sink/-/blob/master/k8s%20and%20Nginx/Kubernetes_IC/F5_Container_Ingress_service/CIS_fronting_NIC/f5-k8s-bigip-ctlr-ConfigMap_with_cert.yaml

SSL passthrough is a straight up L4 VIP. - https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/declarations/non-http-services.html#tcp-load-balanced-to-icap-with-custom-monitor (without the ICAP)

With XFF, you can either use a HTTP profile ( https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/refguide/schema-reference.html#http-profile-insertheader ) to insert the header or use an irule and have AS3 to attach that irule.

For use cases, please take a look at this ( https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/declarations/ ).

The AS3 should be the same for the Kubernetes setup as well, the BIG-IP gets to process it regardless where the declaration comes from.

To assist with writing AS3 declaration, you might want to use Visual Studio Code and take advantage of syntax checking feature. ( https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/validate.html )

Thanks  . Three other questions for you:

1. How can I do SSL offloading? One article I found online suggested I set my ServiceMain `class` to `Service_TCP`. That didn't seem to work.
2. My NGINX-Ingress has `externalTrafficPolicy: "Local"` set so that the source IP is reserved but I still don't see it working. How do I get the F5 to set `x-forwarded-for`?
3. Third question is do you have a handy reference guide where I can find out what other things I can do using AS3? It seems like the AS3 reference guide doesn't closely follow what we are doing here. In other words, it looks like AS3 for Kubernetes setup is a bit different than your regular AS3 declarations. Correct me if I'm wrong.

Thanks!

Edit: Figured out how to get SSL passthrough working manually. Had to change my VS to Performance (L4). Time to figure out how to do it via AS3 🙂

Use Common and Shared, see below. The objects will be create in /Common/Shared location.

"Common": {
            "class": "Tenant",
            "enable": true,
            "Shared": {
                "class": "Application",
                "template": "shared",
...

https://gitlab.com/abgmbh/kitchen_sink/-/blob/master/AS3/Common.json#L10

Hey @Chris, thanks for the response!

host-gw is a mode in flannel. You're right -- I believe it uses the machine's routing table to move traffic in and out of the overlay network. Unfortunately I cannot reach pods from within the F5, that is a limitation and only supported in Flannel VXLAN it seems (unless I do static routes as you mentioned but these seems very complicated).

Thanks for clarifying the partition question. Do you know how we can specify in the AS3 declaration so that the nodes get created in the correct partition? I'd like to do it for organizational purposes.

Thanks again!

Hey Ali,

When you say host-gw, are you referring to adding static routes on the F5 with the pod IP space pointing to the Kubernetes nodes as gateways? It is routing similar to what BGP does but I have not tested it myself. Are you able to reach the Pods from within the F5? You might also need to add routes on the Kubernetes nodes for returning traffic.

From a routing's perspective it sounds like a feasible solution. But I don't know if there are intricacies that Calico as a CNI does with regards to routing that we don't know about.

Regarding NGINX_IC partition, my understanding is that it's all done by AS3 via ConfigMap. Partition is only an administrative function, creating nodes in Common should not affect data traffic.

Thanks,

Chris

Hey  ,

Thanks for the article, it's helpful. I'm trying to run through this setup and unfortunately I'm running into problems. I want to preface this by saying I am using the open-source version of NGINX. I know that this tutorial is based off of nginx-plus but I don't see anything obvious why this wouldn't work with the open source version. Please correct me if I'm wrong.

So I am able to get to the point where the big-ip controller on my Kubernetes cluster creates a virtual server, node pool and my nodes. The nodes created on the F5 are created using the NGINX pod's Cluster IP's. The nodes are shown as "down" -- as expected, as I see in your screenshot they're also down and the F5 doesn't know how to ping the pods.

When trying to hit my website via web browser, I get "ERR_CONNECTION_TIMED_OUT".

Here are the configuration args I am setting on my big-ip-controller:

- --credentials-directory=/var/run/secrets/credentials
          - --bigip-url=xxxxx
          - --pool-member-type=cluster
          - --bigip-partition=$(BIGIP_PARTITION)
          - --insecure=true
          - --manage-ingress=false
          - --as3-validation=true
          - --manage-configmaps=true
          - --log-as3-response=true

My ConfigMap for the AS3 declaration is the same as yours except that I changed the client certificate. I have tried the original ConfigMap as well, no luck there.

Here is what my NGINX Service looks like:

apiVersion: v1
kind: Service
metadata:
 name: ali-service
 namespace: xxx
 labels:
   cis.f5.com/as3-tenant: Nginx_IC
   cis.f5.com/as3-app: Nginx_IC_vs
   cis.f5.com/as3-pool: Nginx_IC_pool
spec:
 type: ClusterIP
 ports:
   - port: 443
     targetPort: 443
     protocol: TCP
     name: https
 selector:
    app: nginx-ingress
    component: controller
    release: nginx-ingress

And finally, I did try making my service a `NodePort` (instead of ClusterIP) to make sure that I can reach my service via NGINX. This works. I've reverted the change back to ClusterIP.

I'm not exactly sure what I'm doing wrong and I would appreciate some guidance on what to check.

My company does have a F5 support plan so if you think a ticket would be better, I can get one going. Alternatively, if you have a moment to respond here or on a phone call that would work for me as well.

Thanks Chris!

Edit: One thing I want to point out is my nodes on the F5 are created in the "Common" partition, not in the "NGINX_IC" partition. It's hard to tell whether it should be created in the NGINX_IC partition. Either way, the Node Pool is referencing the nodes. But I don't know if this is contributing to my problem.

Edit 2: I think I understand where I'm going wrong, now that I think of it. My cluster CNI is flannel and I'm using host-gw. When running the big-ip controller in "cluster" mode, you're essentially integrating the F5 within the overlay network, which means I need to be running VXLAN. Can I confirm with you, Chris, that host-gw won't work? I think this is an obvious no because then the big-ip controller adds your actual Kubernetes VM nodes to the F5 node pool, not the pods of the nginx controller. I've tried setting the controller mode to "nodeport" which also doesn't seem to work. Thanks!

Hey Kunal,

I just did a test with the latest CIS (e.g., 2.0) by changing the deployment file with the following, and everything seems to be working.

image: "f5networks/k8s-bigip-ctlr:latest"

Please take a look at this folder in my repository ( https://gitlab.com/abgmbh/kitchen_sink/-/tree/master/k8s%20and%20Nginx/Kubernetes_IC/F5_Container_Ingress_service ).

On a side note, can you please use code quote for code or logs in future replies? It might be much easier to read as the logs are quite long.

Thanks,

Chris

Hello  

I hope you are doing well.

Just wanted to check if this solution still works. I am trying to recreate the environment but its not working, its giving me below error

2020/06/08 20:47:18 [ERROR] [AS3] Response from BIG-IP: code: ERR_REQUEST_FAILED --- tenant:Nginx_IC --- message: declaration failed

2020/06/08 20:47:18 [ERROR] [AS3] Response from BIG-IP: code: 200 --- tenant:k8s-AS3_AS3 --- message: no change

I have tried this setup with CIS 2.0.0 and f5appsvc 3.20.0 and also CIS 1.14.0 and f5appsvc 3.17.1, i am using same working configuration from march. but getting below error

nginx SVC config

root@master-1:~# kubectl describe svc nginx-ingress2 -n nginx-ingress

Name:       nginx-ingress2

Namespace:     nginx-ingress

Labels:      cis.f5.com/as3-app=Nginx_vs

          cis.f5.com/as3-pool=Nginx_IC_pool

          cis.f5.com/as3-tenant=Nginx_IC

Annotations:    <none>

Selector:     app=nginx-ingress

Type:       ClusterIP

IP:        10.111.160.103

Port:       https 443/TCP

TargetPort:    443/TCP

Endpoints:     10.1.2.191:443

Session Affinity: None

Events:      <none>

Configmap for CIS and F5 integration

root@master-1:~# kubectl describe configmap nginx-as3 -n kube-system

Name:     nginx-as3

Namespace:  kube-system

Labels:    as3=true

       f5type=virtual-server

Annotations: <none>

Data

====

template:

----

{

  "class": "AS3",

  "action": "deploy",

  "persist": true,

  "declaration": {

   "class": "ADC",

   "schemaVersion": "3.13.0",

   "id": "1847a369-5a25-4d1b-8cad-5740988d4423",

   "label": "APP Template",

   "remark": "HTTP application",

   "Nginx_IC": {

       "class": "Tenant",

       "Nginx_IC_vs": {

         "class": "Application",

         "template": "generic",

         "app_80_vs": {

          "class": "Service_HTTP",

          "remark": "app",

          "virtualAddresses": [

           "10.165.36.141"

           ],

          "virtualPort": 80,

          "profileTCP": {

          "bigip": "/Common/f5-tcp-lan"

          },

       "pool": "Nginx_IC_pool"

          },

          "Nginx_IC_pool": {

          "class": "Pool",

          "members": [

          {

           "servicePort": 80,

           "shareNodes": true,

           "serverAddresses": []

          }

         ]

        }

       }

      }

    }

 }

Events: <none>

CIS:

root@master-1:~# kubectl describe pod k8s-bigip-ctlr-deployment-6759c46587-tdk79 -n kube-system

Name:     k8s-bigip-ctlr-deployment-6759c46587-tdk79

Namespace:  kube-system

Priority:   0

Node:     worker-2/192.168.5.22

Start Time:  Mon, 08 Jun 2020 20:40:16 +0000

Labels:    app=k8s-bigip-ctlr

       pod-template-hash=6759c46587

Annotations: <none>

Status:    Running

IP:      10.1.2.192

IPs:

 IP:      10.1.2.192

Controlled By: ReplicaSet/k8s-bigip-ctlr-deployment-6759c46587

Containers:

 k8s-bigip-ctlr:

  Container ID: docker://4f4bfd89700af786bfa3920e5287160003a4500370c4e133c159cc33c62ed984

  Image:     f5networks/k8s-bigip-ctlr:1.14.0

  Image ID:   docker-pullable://f5networks/k8s-bigip-ctlr@sha256:25bdfc947ed4cdd172a68e37c51dbaa8ca87fcbc4d894622b42a260755a2bf68

  Port:     <none>

  Host Port:   <none>

  Command:

   /app/bin/k8s-bigip-ctlr

  Args:

   --bigip-username=$(BIGIP_USERNAME)

   --bigip-password=$(BIGIP_PASSWORD)

   --bigip-url=https://192.168.5.210

   --bigip-partition=k8s-AS3

   --pool-member-type=cluster

   --agent=as3

   --manage-ingress=false

   --insecure=true

   --as3-validation=true

   --node-poll-interval=30

   --verify-interval=30

   --log-level=INFO

  State:     Running

   Started:   Mon, 08 Jun 2020 20:40:20 +0000

  Ready:     True

  Restart Count: 0

  Environment:

   BIGIP_USERNAME: <set to the key 'username' in secret 'bigip-login'> Optional: false

   BIGIP_PASSWORD: <set to the key 'password' in secret 'bigip-login'> Optional: false

  Mounts:

   /var/run/secrets/kubernetes.io/serviceaccount from bigip-ctlr-token-r6rvn (ro)

Conditions:

 Type       Status

 Initialized    True

 Ready       True

 ContainersReady  True

 PodScheduled   True

Volumes:

 bigip-ctlr-token-r6rvn:

  Type:    Secret (a volume populated by a Secret)

  SecretName: bigip-ctlr-token-r6rvn

  Optional:  false

QoS Class:    BestEffort

Node-Selectors: <none>

Tolerations:   node.kubernetes.io/not-ready:NoExecute for 300s

         node.kubernetes.io/unreachable:NoExecute for 300s

Events:

 Type  Reason   Age    From        Message

 ----  ------   ----    ----        -------

 Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/k8s-bigip-ctlr-deployment-6759c46587-tdk79 to worker-2

 Normal Pulling  17m    kubelet, worker-2 Pulling image "f5networks/k8s-bigip-ctlr:1.14.0"

 Normal Pulled   17m    kubelet, worker-2 Successfully pulled image "f5networks/k8s-bigip-ctlr:1.14.0"

 Normal Created  17m    kubelet, worker-2 Created container k8s-bigip-ctlr

 Normal Started  17m    kubelet, worker-2 Started container k8s-bigip-ctlr

Any help is greatly appreciated.

Thanks

Kunal

When you use ClusterIP, the BIG-IP needs to be able to deliver traffic to that IP space. If you use Calico (BGP), that traffic is routed. If you use Flannel (VXLAN), that traffic is sent inside the VXLAN tunnel.

With BGP, the route table will have the next-hop set to the k8s nodes. Traffic is routed to k8s nodes and those nodes further route traffic to the NGINX IC pods.

With VXLAN, the BIG-IP establishes a tunnel with the k8s nodes at the other end. Traffic is sent inside the tunnel and arrives at the k8s nodes, and traffic is taken out from the tunnel and delivered to the NGINX IC pods.

Hello Chris... thanks for responding to the queries... In above example clusterip is used which advertised nginx ingress controller IP to the F5. can you please confirm what would be the data plane forwarding, is it like below

F5-->k8'sNode-->Services-->POD where NGINX is running or is it different?

Thanks again for your help