CIS and Kubernetes - Part 1: Install Kubernetes and Calico

Welcome to this series to see how to:

Install Kubernetes and Calico (Part 1)
Deploy F5 Container Ingress Services (F5 CIS) to tie applications lifecycle to our application services (Part 2)

Here is the setup of our lab environment:

BIG-IP Version: 15.0.1

Kubernetes component: Ubuntu 18.04 LTM

We consider that your BIG-IPs are already setup and running:

Licensed and setup as a cluster
The networking setup is already done

Part 1: Install Kubernetes and Calico

Setup our systems before installing kubernetes

Step1: Update our systems and install docker

To run containers in Pods, Kubernetes uses a container runtime. We will use docker and follow the recommendation provided here

As root on ALL Kubernetes components (Master and Node):

# Install packages to allow apt to use a repository over HTTPS
apt-get -y update && apt-get install -y apt-transport-https ca-certificates curl software-properties-common

# Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

# Add Docker apt repository.
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

# Install Docker CE.
apt-get -y update && apt-get install -y docker-ce=18.06.2~ce~3-0~ubuntu

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

We may do a quick test to ensure docker run as expected:

docker run hello-world

Step2: Setup Kubernetes tools (kubeadm, kubelet and kubectl)

To setup Kubernetes, we will leverage the following tools:

kubeadm: the command to bootstrap the cluster.

kubelet: the component that runs on all of the machines in your cluster and does things like starting pods and containers.

kubectl: the command line util to talk to your cluster.

As root on ALL Kubernetes components (Master and Node):

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get -y update

We can review which version of kubernetes is supported with F5 Container Ingress Services here

At the time of this article, the latest supported version is v1.13.4. We'll make sure to install this specific version with our following step

apt-get install -qy kubelet=1.13.4-00 kubeadm=1.13.4-00 kubectl=1.13.4-00 kubernetes-cni=0.6.0-00
apt-mark hold kubelet kubeadm kubectl

Install Kubernetes

Step1: Setup Kubernetes with kubeadm

We will follow the steps provided in the documentation here

As root on the MASTER node (make sure to update the api server address to reflect your master node IP):

kubeadm init --apiserver-advertise-address=10.1.20.20 --pod-network-cidr=192.168.0.0/16

Note: SAVE somewhere the kubeadm join command. It is needed to "assimilate" the node later. In my example, it looks like the following (YOURS WILL BE DIFFERENT):

kubeadm join 10.1.20.20:6443 --token rlbc20.va65z7eauz89mmuv --discovery-token-ca-cert-hash sha256:42eca5bf49c645ff143f972f6bc88a59468a30276f907bf40da3bcf5127c0375

Now you should NOT be ROOT anymore. Go back to your non root user. Since i use Ubuntu, i'll use the default "ubuntu" user

Run the following commands as highlighted in the screenshot above:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step2: Install the networking component of Kubernetes

The last step is to setup the network related to our k8s infrastructure. In our kubeadm init command, we used --pod-network-cidr=192.168.0.0/16 in order to be able to setup next on network leveraging Calico as documented here

kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml

You may monitor the deployment by running the command:

kubectl get pods --all-namespaces

After some time (<1 min), everything should have a "Running" status.

Make sure that CoreDNS started also properly. If everything is up and running, we have our master setup properly and can go to the node to setup k8s on it.

Step3: Add the Node to our Kubernetes Cluster

Now that the master is setup properly, we can assimilate the node. You need to retrieve the "kubeadmin join …" command that you received at the end of the "kubeadm init …" cmd. You must run the following command as ROOT on the Kubernetes NODE (remember that you got a different hash and token, the command below is an example):

kubeadm join 10.1.20.20:6443 --token rlbc20.va65z7eauz89mmuv --discovery-token-ca-cert-hash sha256:42eca5bf49c645ff143f972f6bc88a59468a30276f907bf40da3bcf5127c0375

We can check the status of our node by running the following command on our MASTER (ubuntu user)

kubectl get nodes

Both component should have a "Ready" status.

Last step is to setup Calico between our BIG-IPs and our Kubernetes cluster

Setup Calico

We need to setup Calico on our BIG-IPs and k8S components. We will setup our environment with the following AS Number: 64512

Step1: BIG-IPs Calico setup

F5 has documented this procedure here

We will use our self IPs on the internal network. Therefore we need to make sure of the following:

The self IP has a portlock down setup to "Allow All"
Or add a TCP custom port to the self IP: TCP port 179

You need to allow BGP on the default route domain 0 on your BIG-IPs. Connect to the BIG-IP GUI on go into Network > Route domain. Click on Route Domain "0" and allow BGP

Click on "Update"

Once this is done, connect via SSH and get into a bash shell on both BIG-IPs

Run the following commands:

#access the IMI Shell
imish

#Switch to enable mode
enable

#Enter configuration mode
config terminal
#Setup route bgp with AS Number 64512
router bgp 64512

#Create BGP Peer group
neighbor calico-k8s peer-group

#assign peer group as BGP neighbors
neighbor calico-k8s remote-as 64512

#we need to add all the peers: the other BIG-IP, our k8s components
neighbor 10.1.20.20 peer-group calico-k8s
neighbor 10.1.20.21 peer-group calico-k8s

#on BIG-IP1, run
neighbor 10.1.20.12 peer-group calico-k8s

#on BIG-IP2, run
neighbor  10.1.20.11 peer-group calico-k8s

#save configuration
write

#exit
end

You can review your setup with the command

show ip bgp neighbors

Note: your other BIG-IP should be identified with a router ID and have a BGP state of "Active". The k8s node won't have a router ID since BGP hasn't already been setup on those nodes.

Keep your BIG-IP SSH sessions open. We'll re-use the imish terminal once our k8s components have Calico setup

Step2: Kubernetes Calico setup

On the MASTER node (not as root), we need to retrieve the calicoctl binary

curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.10.0/calicoctl
chmod +x calicoctl
sudo mv calicoctl /usr/local/bin

We need to setup calicoctl as explained here

sudo mkdir /etc/calico

Create a file /etc/calico/calicoctl.cfg with your preferred editor (you'll need sudo privilegies). This file should contain the following

apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: "kubernetes"
  kubeconfig: "/home/ubuntu/config"

Note: you may have to change the path specified by the kubeconfig parameter based on the user you use to do kubectl command

To make sure that calicoctl is properly setup, run the command

calicoctl get nodes

You should get a list of your Kubernetes nodes

Now we can work on our Calico/BGP configuration as documented here

On the MASTER node:

cat << EOF | calicoctl create -f -
 apiVersion: projectcalico.org/v3
 kind: BGPConfiguration
 metadata:
   name: default
 spec:
   logSeverityScreen: Info
   nodeToNodeMeshEnabled: true
   asNumber: 64512
EOF

Note: Because we setup nodeToNodeMeshEnabled to True, the k8s node will receive the same config

We may now setup our BIG-IP BGP peers. Replace the peerIP Value with the IP of your BIG-IPs

cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-global-bigip1
spec:
  peerIP: 10.1.20.11
  asNumber: 64512
EOF

cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-global-bigip2
spec:
  peerIP: 10.1.20.12
  asNumber: 64512
EOF

Review your setup with the command: