How to secure egress with F5 Service Proxy for Kubernetes
Outline:
- Securing Egress Challenges
- How F5 can help
- Technical bit on how it works
Getting traffic into your clusters to your workloads is just a small part of the cluster admin's tasks, and there are many options available.
Controlling the packets going out is harder and often ignored. This makes your clusters more vulnerable to security risks because they don’t follow the same strict rules as your traditional networks.
This article will dive deeper into how SPK can control traffic exiting your clusters, even when your application workload uses multus to attach additional external interfaces.
Secure Egress Challenges
By default, a pod deployed using calico CNI will follow the default route to get out of the cluster. Traffic will look like it’s coming from the worker host’s external IP address on the management interface.
While Kubernetes NetworkPolicies can be used for egress, it becomes painful to manage the lifecycle of hundreds or thousands of policies across all namespaces as the cluster grows.
If you deploy a pod with multus interfaces, as commonly seen with telco applications, you add another way for that pod to bypass any NetworkPolicies applied within the cluster.
What if there was a way to manage egress dynamically (as pods are spun up and down) and easily so that the cluster admin could centrally configure and control traffic flowing out of the cluster?
How F5 can help
Service Proxy for Kubernetes (SPK) is a cloud-native application traffic management solution, designed for communication service provider (CoSP) 5G networks and other application workloads.
With SPK and its Calico egress gateway feature, managing a pod's default calico network interface as well as any multus interfaces becomes easy and consistent with the CSRC daemonset.
Kernel routes are automatically configured so that the pods traffic will always be routed via the SPK pod where you can apply consistent, namespace-aware network policies, source NAT translation, and other controls.
If the "watched" application workload is deleted, the corresponding host rules also get removed.
Technical Overview
This section will provide an overview of how to configure the above scenario.
Host Prerequisites
On the host, two shims of type macvlan bridges are created on physical interfaces, one for the application pod's calico traffic and one for the macvlan traffic, which will forward packets on to SPK. These interfaces allow connectivity to the SPK's "internal" and "external2" interfaces, respectively.
ip link add spk-shim link ens224 type macvlan mode bridge
ip addr add 10.1.30.244/24 dev shim1
ip link set shim1 up
ip link add spk-shim2 link ens256 type macvlan mode bridge
ip addr add 10.1.10.244/24 dev shim2
ip link set shim2 upApplication Prerequisites and Configuration
In the SPK controller values.yaml file, configure your application workload namespaces in the watchNamespace block.
watchNamespace:
- "spk-apps"
- "spk-apps-2"
Since we want SPK to do the source NAT for pod egress traffic, we create an IPPool with natOutgoing set to false. This IPPool will be used by the applications.
apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
name: app-ip-pool
spec:
cidr: 10.124.0.0/16
ipipMode: Always
natOutgoing: false
Ensure that the application namespaces are annotated like below to use the IPPool.
kubectl annotate namespace spk-apps "cni.projectcalico.org/ipv4pools"=[\"app-ip-pool\"]
kubectl annotate namespace spk-apps-2 "cni.projectcalico.org/ipv4pools"=[\"app-ip-pool\"]
Deploy your application. See below for an example deployment manifest for the application. Note that I'm attaching a secondary macvlan interface, which is in addition to the default calico interface. It will get an IP address automatically as configured in the corresponding NetworkAttachmentDefinition. Note the specific labels used by SPK, which allows you to enable traffic routing to SPK on a per application basis.
Additionally, the enableSecureSPK=true label will instruct SPK to create additional listeners that will pick up traffic coming from the pod's secondary macvlan interface. (Will show these listeners later)
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
annotations:
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
annotations:
k8s.v1.cni.cncf.io/networks: '[
{ "name": "macvlan-conf-ens256-myapp1" } ]'
labels:
app: nginx
enableSecureSPK: "true"
enablePseudoCNI: "true"
secureSPKPort: "8050"
secureSPKCNFPodIfName: "net1"
secondaryCNINodeIfName: "spk-shim2"
primaryCNINodeIfName: "spk-shim"
secureSPKNetAttachDefName: "macvlan-conf-ens256"
secureSPKEgressVlanName: "external"SPK Configuration
Deploy the custom resource that will configure a listener that does two things:
- listen for traffic coming from the internal vlan, or the calico interface of targeted application pods
- SNAT the traffic so that the source IP is an IP address of SPK
apiVersion: "k8s.f5net.com/v1"
kind: F5SPKEgress
metadata:
name: egress-crd
namespace: ns-f5-spk
spec:
#leave commented out for snat automap
#egressSnatpool: "snatpool-1"
dualStackEnabled: false
maxTmmReplicas: 1
vlans:
vlanList: [internal]
disableListedVlans: false
Next, we deploy the CSRC Daemonset that dynamically creates the kernel rules and routes for us. Note that I am setting the daemonsetMode to "pseudoCNI" which means I want to route both primary (calico) and secondary interface traffic to SPK.
values-csrc.yaml
image:
repository: gitlab.tky.lab:5050/registry/spk/200
# daemonset mode, regular, secureSPK, or pseudoCNI
#daemonsetMode: "regular"
daemonsetMode: "pseudoCNI"
ipFamily: "ipv4"
imageCredentials:
name: f5-common-pull-creds
config:
iptableid: 200
interfacename: "spk-shim"
#tmmpodinterfacename: "internal"
json:
ipPoolCidrInfo:
cidrList:
- name: cluster-cidr0
value: "172.21.107.192/26"
- name: cluster-cidr1
value: "172.21.66.0/26"
- name: cluster-cidr2
value: "10.124.18.192/26"
- name: node-cidr0
value: "10.1.11.0/24"
- name: node-cidr1
value: "10.1.10.0/24"
ipPoolList:
- name: default-ipv4-ippool
value: "172.21.64.0/18"
- name: spk-app1-pool
value: "10.124.0.0/16"Testing
You can then log onto the worker node that is hosting the applications and confirm the routes and rules are created. Essentially, the rules are making calico interfaces use a custom route table that ensures that the default route is via the SPK.
# ip rule
0: from all lookup local
32254: from all to 172.21.107.192/26 lookup main
32254: from all to 172.21.66.0/26 lookup main
32254: from all to 10.124.18.192/26 lookup main
32254: from all to 172.28.15.0/24 lookup main
32254: from all to 10.1.10.0/24 lookup main
32257: from 10.124.18.207 lookup ns-f5-spkshim1ipv4257 <--match on app pod1 calico IP!!!
32257: from 10.124.18.211 lookup ns-f5-spkshim1ipv4257 <--match on app pod2 calico IP!!!
32258: from 10.1.10.171 lookup ns-f5-spkshim2ipv4258 <--match on app pod1 macvlan IP!!!
32258: from 10.1.10.170 lookup ns-f5-spkshim2ipv4258 <--match on app pod2 macvlan IP!!!
32766: from all lookup main
32767: from all lookup default
# ip route show table ns-f5-spkshim1ipv4257
default via 10.1.30.242 dev shim1
10.1.30.242 via 10.1.30.242 dev shim1
# ip route show table ns-f5-spkshim2ipv4258
default via 10.1.10.160 dev shim2
10.1.10.160 via 10.1.10.160 dev shim2
If I then try to execute a curl command towards a server that exists in a network segment beyond SPK, the application pod will hit the CSRC-configured ip rule and then forwarded to its new default gateway, which is SPK.
Since SPK has Source NAT enabled, the "Client IP" from the server perspective is the self-IP of SPK. This means you can apply firewall policies to application workloads in a deterministic way as well as have visibility into what kind of traffic is coming out of your clusters.
k exec -it nginx-7d7699f86c-hsx48 -n my-app1 -- curl 10.1.70.30
================================================
___ ___ ___ _
| __| __| | \ ___ _ __ ___ /_\ _ __ _ __
| _||__ \ | |) / -_) ' \/ _ \ / _ \| '_ \ '_ \
|_| |___/ |___/\___|_|_|_\___/ /_/ \_\ .__/ .__/
|_| |_|
================================================
Node Name: F5 Docker vLab
Short Name: server.tky.f5se.com
Server IP: 10.1.70.30
Server Port: 80
Client IP: 10.1.30.242
Client Port: 59248
Client Protocol: HTTP
Request Method: GET
Request URI: /
host_header: 10.1.70.30
user-agent: curl/7.88.1
A simple tcpdump command run in the debug container of SPK confirms that the pod's calico interface IP (10.124.18.192) is the source IP of the incoming traffic on SPK, and after Source NAT is applied using the self-IP of SPK (10.1.30.242), the packet is sent out towards the server.
/tcpdump -nni 0.0 tcp port 80
----snip----
12:34:51.964200 IP 10.124.18.192.48194 > 10.1.70.30.80: Flags [P.], seq 1:75, ack 1, win 225, options [nop,nop,TS val 4077628853 ecr 777672368], length 74: HTTP: GET / HTTP/1.1 in slot1/tmm0 lis=egress-ipv4 port=1.1 trunk=
----snip----
12:34:51.964233 IP 10.1.30.242.48194 > 10.1.70.30.80: Flags [P.], seq 1:75, ack 1, win 225, options [nop,nop,TS val 4077628853 ecr 777672368], length 74: HTTP: GET / HTTP/1.1 out slot1/tmm0 lis=egress-ipv4 port=1.1 trunk=
Let's take a look at egress application traffic that is using the secondary macvlan interface. In this case, I have not configured Source NAT so SPK will forward the traffic out, retaining the original pod IP.
k exec -it nginx-7d7699f86c-g4hpv -n my-app1 -- curl 10.1.80.30
================================================
___ ___ ___ _
| __| __| | \ ___ _ __ ___ /_\ _ __ _ __
| _||__ \ | |) / -_) ' \/ _ \ / _ \| '_ \ '_ \
|_| |___/ |___/\___|_|_|_\___/ /_/ \_\ .__/ .__/
|_| |_|
================================================
Node Name: F5 Docker vLab
Short Name: ue-client3
Server IP: 10.1.80.30
Server Port: 80
Client IP: 10.1.10.170
Client Port: 56436
Client Protocol: HTTP
Request Method: GET
Request URI: /
host_header: 10.1.80.30
user-agent: curl/7.88.1
Another tcpdump command run in the debug container of SPK shows that it receives the above GET request and sends it out without Source NAT in this case.
/tcpdump -nni 0.0 tcp port 80
----snip----
13:54:40.389281 IP 10.1.10.170.56436 > 10.1.80.30.80: Flags [P.], seq 1:75, ack 1, win 229, options [nop,nop,TS val 4087715696 ecr 61040149], length 74: HTTP: GET / HTTP/1.1 in slot1/tmm0 lis=secure-egress-ipv4-virtual-server port=1.2 trunk=
----snip----
13:54:40.389305 IP 10.1.10.170.56436 > 10.1.80.30.80: Flags [P.], seq 1:75, ack 1, win 229, options [nop,nop,TS val 4087715696 ecr 61040149], length 74: HTTP: GET / HTTP/1.1 out slot1/tmm0 lis=secure-egress-ipv4-virtual-server port=1.2 trunk=
You can use the familiar tmctl command inside the debug container of SPK to confirm the statistics for both listeners that process the pod's primary (egress-ipv4) and secondary (secure-egress-ipv4-virtual-server) interface egress traffic.
/tmctl -f /var/tmstat/blade/tmm0 virtual_server_stat -s name,clientside.bytes_in,clientside.bytes_out,no_staged_acl_match_accept -w 200
name clientside.bytes_in clientside.bytes_out no_staged_acl_match_accept
---------------------------------------------- ------------------- -------------------- --------------------------
secure-egress-ipv4-virtual-server 394 996 1
egress-ipv4 394 1011 1
Now that you have egress traffic routed to the SPK data plane pods, you can use the below F5 published custom resource definitions (CRDs) to apply granular access control lists (ACLs) to meet your security requirements. The firewall configuration is defined as code (YAML manifests) so it natively integrates with K8s and portable across clusters.
F5BigContextGlobal: CRD to define the default global firewall behavior and reference the firewall policy.
F5BigFwPolicy: CRD to define your firewall rules.
In summary, the above diagrams and configuration snippets show how SPK can capture all egress traffic in a dynamic way so that you don't have to sacrifice security and control in your ever-changing Kubernetes clusters.
Employee