on 27-Jan-2021 09:24
First thing first. What is GitOps?
In a nutshell, GitOps is a practice (Git Operation) that allows you to use GIT and code repository as your configuration source of truth (Declarative Infrastructure as Code and Application Delivery as Code) couple with various supporting tools. The state of your git repository syncs with your infrastructure and application states. As operation team runs daily operations (CRUD - "Create, Read, Update and Delete") leveraging the goodness and philosophy of DevOps, they no longer required to store configuration manifest onto various configuration systems. THe Git repo will be the source of truth. Typically, the target systems or infrastructures runs on Kubernetes base platform. For further details and better explanation of GitOps, please refer to below or Google search.
https://www.weave.works/technologies/gitops/
https://www.atlassian.com/git/tutorials/gitops
Why practice GitOps?
I have been managing my lab environment for many years. I use my lab for research of technologies, customer demos/Proof-of-Concepts, applications testing, and code development. Due to the nature of constant changes to my environment (agile and dynamic nature), especially with my multiple versions of Kubernetes platform, I have been spending too much time updating, changing, building, deploying and testing various cloud native apps. Commands like docker build, kubectl, istioctl and git have been constantly and repetitively used to operate environments. Hence, I practice GitOps for my Kubernetes infrastructure. Of course, task/operation can be automated and orchestrated with tools such as Ansible, Terraform, Chef and Puppet. You may not necessarily need GitOps to achieve similar outcome. I managed with GitOps practice partly to learn the new "language" and to experience first-hand the full benefit of GitOps. Here are some of my learnings and operations experience that I have been using to manage F5's NGINX App Protect and many demo apps protected by it, which may benefit you and give you some insight on how you can run your own GitOps. You may leverage your own GitOps workflow from here. For details and description on F5's NGINX App Protect, please refer to https://www.nginx.com/products/nginx-app-protect/
Key architecture decision of my GitOps Workflow.
Use Case and desirable outcome
My Problem Statement
I need to ensure that my infrastructure (Kubernetes Ingress controller) and web application firewall (NGINX App Protect) is kept up to date with ease. For example, when there are new NGINX-ingress and NGINX App Protect updates (e.g., new version, attack signature and threat campaign signature), I would like to seamlessly push changes out to NGINX-ingress and NGINX App Protect (as it protects my backend apps) without impacting applications protected by NGINX App Protect.
GitOps Workflow
Start small, start with clear workflow. Below is a depiction of the overall GitOps workflow.
NGINX App Protect is the target application. Hence, before description of the full GitOps Workflow, let us understand deployment options for NGINX App Protect.
NGINX App Protect Deployment model
There are four deployment models for NGINX App Protect. A common deployment models are:
Pipeline demonstrated in this article will work with either NGINX App Protect deployed as Ingress Controller (2) or per-service proxy model (3). For the purposes of this article, NGINX App Protect is deployed at the Ingress controller at the entry point to Kubernetes (Kubernetes Edge Proxy).
For those who prefer video,
Video Demonstration
Part 1 /3 – GitOps with NGINX Plus Ingress and NGINX App Protect - Overview
Part 2 /3 – GitOps with NGINX Plus Ingress and NGINX App Protect – Demo in Action
Part 3/3 - GitOps with NGINX Plus Ingress and NGINX App Protect - WAF Security Policy Management.
Description of GitOps workflow
Note:
Snippet on where Gitlab CI update nginx-plus-ingress deployment manifest (Flow#6).
Each new image build will be tagged with <branch>-hash-<version>
... spec: imagePullSecrets: - name: regcred serviceAccountName: nginx-ingress containers: - image: reg.foobz.com.au/apps/nginx-plus-ingress:master-f660306d-1.9.1 imagePullPolicy: IfNotPresent name: nginx-plus-ingress ...
Gitlab CI/CD Pipeline
Successful run of CI/CD pipeline to build, test, scan and push container image to private repository and execute code commit onto nginx-plus-ingress repo.
Note:
Trivy scanning report will be uploaded or committed back to the same repo. To prevent Gitlab CI triggering another build process ("pipeline loop"), the code commit is tagged with [skip ci].
ArgoCD continuous deployment
ArgoCD constantly (default every 3 mins) syncs desired application state with my Kubernetes cluster. Its ensures configuration manifest stored in Git repository is always synchronised with the target environment.
Mytrain-dev apps are protected by nginx-ingress + NGINX App Protect. Specific (per service/URI enforcement). NGINX App Protect policy is applied onto this service.
nginx-ingress + NGINX App Protect is deployed as an Ingress Controller in Kubernetes. pod-template-hash=xxxx is labeled and tracked by ArgoCD.
$ kubectl -n nginx-ingress get pod --show-labels NAME READY STATUS RESTARTS AGE LABELS nginx-ingress-776b64dc89-pdtv7 1/1 Running 0 8h app=nginx-ingress,pod-template-hash=776b64dc89 nginx-ingress-776b64dc89-rwk8w 1/1 Running 0 8h app=nginx-ingress,pod-template-hash=776b64dc89
Please refer to the attached video links above for full demo in actions.
References
Tools involved
Source repo used for this demonstration
Repo for building nginx-ingress + NGINX App Protect image repo
https://github.com/fbchan/nginx-plus-ingress.git
Repo use for deployment manifest of nginx-ingress controller with the NGINX App Protect policy.
https://github.com/fbchan/my-kubernetes-apps.git
Summary
GitOps perhaps is a new buzzword. It may or may not make sense in your environment. It definitely makes sense for me. It integrated well with NGINX App Protect and allows me to constantly update and push new code changes into my environment with ease. A few months down the road, when I need to update nginx-ingress and NGINX App Protect, I just need to trigger a CI job, and then everything works like magic. Your mileage may vary. Experience leads me to think along the line of - start small, start simple by "GitOps-ing" on one of your apps that may require frequency changes. Learn, revise and continuously improve from there. The outcome that GitOps provides will ease your operational burden with "do more with less". Ease of integration of nginx-ingress and NGINX App Protect into your declarative infrastructure and application delivery with GitOps and F5's industry leading Web Application firewall protection will definitely alleviate your organisation's risk exposure to external and internal applications threat.
I think this GitOps workflow demo is very nice.
I'm trying to practice GitOps. But it's hard to create an environment.
I would like to know how you created this demo environment.
Hi Shingo,
Key comments in this demo are ArgoCD, gitlab repository and Kubernetes. I use kubespray to spin up k8s/k3s. Install gitlab in a separate VM and ArgoCD on k3s. K3S can run on same VM as the gitlab. I been running my env quite a while. Probably you can start of with a k8s/k3s with manual deploy simple apps. Then have a gitlab or free public github as your repository. Then only add argocd later on.