When cloud-native meets monolithic
According to CNCF’s Cloud Native Survey 2020 published on 17 Nov 2020, containers in production jump 300% from the first survey in 2016. Year 2020 itself increased to 92% from 84% in 2019. (https://www.cncf.io/cncf-cloud-native-survey-2020). In addition, according to F5's 2020 State of Application Services Report (https://www.f5.com/state-of-application-services-report#get-the-report), 80% of organisations are executing on digital transformation and of this organisation, there are more likely to deploy modern app architecture and app services at higher rate. So, cloud-native modern application architecture is gaining great momentum. Industries and majority organisation embracing and pivoting toward cloud-native technologies. Cloud-native provides multitude of benefits – which is not the subject of this article. F5’s BIG-IP (a.k.a classic BIG-IP) is not cloud-native. How F5’s classic BIG-IP be relevant in the cloud-native world? This article demonstrates how cloud-native meets and needs classic BIG-IP (monolithic). FYI: F5’s BIG-IP SPK (Service Proxy for Kubernetes) is BIG-IP delivered in a containerized form factor. It is cloud-native (https://www.f5.com/products/service-proxy-for-kubernetes). BIG-IP SPK will be discussed in future article. How both of them need each other? Typically, it take years for organisation who embracing cloud-native to move into a fully cloud-native technologies/infrastructure. There are use cases where modern cloud-native application needs to integrate with traditional or existing monolithic applications. Modern apps living along with traditional apps or infrastructure are common for most enterprises. F5’s classic BIG-IP can bridge those gaps. This article about use cases that we solved in one of our customer environment where how we leverage classic BIG-IP to bridge gaps between cloud-native and monolithic apps. First, let’s be clear on what cloud-native really means. To set the record straight, cloud-native doesn’t just mean running workload in the cloud, although it is partially true. There are many definition and perspective on what cloud-native really means. For the sake of this article, I will base on the official definition of cloud-native from the CNCF (Cloud Native Computing Foundation), which defines “Cloud native technologies empower organisations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil” My takeaway (characteristic of cloud-native): Scalable apps that design to run in dynamic environment (public, private and hybrid clouds). Typically delivered in the form of microservices/containers which loosely coupled systems. Easily adapted and integrated into automation system. CI/CD part of it ecosystem – frequent release, patch and updates cycle. Immutable (cattle service model) instead of mutable (Pets service model) Declarative API Kubernetes is one of the example of a cloud-naive technologies. What is the Problem Statement? Uniquely identify apps/workload/containers deployed in Kubernetes platform and apply appropriate external security control (e.g. network/application firewall) for containerized apps when it communicates with existing legacy applications deployed outside of Kubernetes (egress from Kubernetes). What are those challenges? Containers that egress off Kubernetes are by design source network address translated (SNAT) from Kubernetes nodes. External security control system such as network firewall may not be able to identify the right authorised source apps as it is hidden behind NATing. How to ensure that only authorised apps deployed in Kubernetes environment authorised to access critical legacy apps (e.g. billing or financial system) protected by network/application firewall? For multi-tenant environment with multiple namespaces in Kubernetes, how to ensure pods or namespaces have a unique identity and enforce control access to egress endpoints (outside of Kubernetes). Unique workload identity is important for end-to-end correlation, audit and traceability. How to provide end-to-end and correlated view from source to target apps. How F5 solved this with classic BIG-IP ADC and Aspen Mesh Service Mesh Architecture Overview This solution article is an extension to the original article – “Expanding Service Mesh without Envoy” published by my colleague Eric Chen. For details of that article please refer to https://aspenmesh.io/expanding-service-mesh-without-envoy/ Aspen Mesh, an innovation from F5, is an enterprise-ready service mesh built on Istio. It is tested and hardened distribution of Istio with complete support by F5. For details, please refer to https://aspenmesh.io. For the purpose of this solution, Aspen Mesh and Istio will be used interchangeably. Solution in a nutshell Each pod had its own workload identity. Part of native capabilities of Aspen Mesh (AM). The identity is in a form of client certificate managed by AM (istiod/Citadel) and generated from an organisation Intermediate CA loaded onto Istio control plane. BIG-IP on-boarded with workload identity (client certificate) and signed from the same organisation Intermediate CA (or root CA). This client certificate NOT managed by AM. F5 Virtual Server (VS) configured with client-side profile to perform mutual TLS (mTLS). F5 VS is registered onto AM. F5 VS service can be discovered from internal service registry. On egress, pod will perform mTLS with F5 VS. As F5 client certificate issues from same organisation Intermediate CA, both parties will negotiate and mutually trust each other and exchange mTLS key. An optional iRule can be implemented on BIG-IP to inspect pod identity (certificate SAN) upon successful mTLS and permit/reject request. BIG-IP implement SNAT and present a unique network identifier (e.g IP address) to network firewall. Environment BIGIP LTM (v14.x) Aspen Mesh - v1.16.x Kubernetes 1.18.x Use Case Permit microservices apps (e.g. bookinfo) to use organisation forward proxy (tinyproxy) to get to Internet which sit behind enterprise network firewall and reject all other microservices apps on the same Kubernetes cluster. Classic BIG-IP Only vs_aspenmesh-bookinfo-proxy-srv-mtls-svc configuration will be demonstrated. Similar configuration can be applied on other VS. F5 Virtual Server configuration F5's VS client profile configuration. "Client Certificate = require" require pods deployed inside AM present a valid trusted client certificate. An optional iRule to only permit pods from bookinfo namespace. Optional irule_bookinfo_spiffee to permit bookinfo apps and reject other apps. when CLIENTSSL_CLIENTCERT { set client_cert [SSL::cert 0] #log local0. "Client cert extensions - [X509::extensions $client_cert]" #Split the X509::extensions output on each newline character and log the values foreach item [split [X509::extensions [SSL::cert 0]] \n] { log local0. "$item" } if {[SSL::cert 0] ne ""} { set santemp [findstr [X509::extensions [SSL::cert 0]] "Subject Alternative Name" 43 " "] set spiffe [findstr $santemp "URI" 4] log local0. "Source SPIFFEE-->$spiffe" if { ($spiffe starts_with "spiffe://cluster.local/ns/bookinfo/") } { log local0. "Aspen Mesh mTLS: PEMITTED==>$spiffe" # Allow and SNAT from defined SNAT Pool } else { log local0. "Aspen Mesh mTLS: REJECTED==>$spiffe" reject } } } Note: As of Istio version 1.xx, client-side envoy (istio sidecar) will start a mTLS handshake with server-side BIG-IP VS (F5's client side profile). During the handshake, the client-side envoy also does a secure naming check to verify that the service account presented in the server certificate is authorised to run the target service. Then only the client-side envoy and server-side BIG-IP will establish a mTLS connection. Hence, the client certificate generated loaded onto BIG-IP have to conform to the secure naming information, which maps the server identities to the service names. For details on secure naming, please refer to https://istio.io/latest/docs/concepts/security/#secure-naming Example to generate a SPIFFE friendly certificate openssl req -new -out bookinfo.istio-spiffee-req.pem -subj "/C=AU/ST=Victoria/L=Melbourne/O=F5/OU=SE/CN=bookinfo.spiffie" -keyout bookinfo.istio-spiffee-key.pem -nodes cat > v3.ext <<-EOF authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1=spiffe://cluster.local/ns/bookinfo/sa/default EOF openssl x509 -req -sha512 -days 365 \ -extfile v3.ext \ -CA ../ca1/ca-cert.pem -CAkey ../ca1/ca-key.pem -CAcreateserial \ -in bookinfo.istio-spiffee-req.pem \ -out bookinfo.istio-spiffee-cert.pem where ca1 is the intermediate CA use for Aspen Mesh. Aspen Mesh Pods and Services before registration of F5 VS $ kubectl -n bookinfo get pod,svc NAME READY STATUS RESTARTS AGE pod/details-v1-78d78fbddf-4vmdr 2/2 Running 0 4d1h pod/productpage-v1-85b9bf9cd7-f6859 2/2 Running 0 4d1h pod/ratings-v1-6c9dbf6b45-9ld6f 2/2 Running 0 4d1h pod/reviews-v1-564b97f875-bjx2r 2/2 Running 0 4d1h pod/reviews-v2-568c7c9d8f-zzn8r 2/2 Running 0 4d1h pod/reviews-v3-67b4988599-pdk25 2/2 Running 0 4d1h pod/traffic-generator-productpage-fc97f5595-pdhvv 2/2 Running 0 6d11h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/details ClusterIP 10.235.14.186 <none> 9080/TCP 6d11h service/productpage ClusterIP 10.235.37.112 <none> 9080/TCP 6d11h service/ratings ClusterIP 10.235.40.239 <none> 9080/TCP 6d11h service/reviews ClusterIP 10.235.1.21 <none> 9080/TCP 6d11h service/traffic-generator-productpage ClusterIP 10.235.17.158 <none> 80/TCP 6d11h Register bigip-proxy-svc onto Aspen Mesh $ istioctl register -n bookinfo bigip-proxy-svc 10.4.0.201 3128 --labels apps=bigip-proxy 2020-12-15T23:14:33.286854Z warn Got 'services "bigip-proxy-svc" not found' looking up svc 'bigip-proxy-svc' in namespace 'bookinfo', attempting to create it 2020-12-15T23:14:33.305890Z warn Got 'endpoints "bigip-proxy-svc" not found' looking up endpoints for 'bigip-proxy-svc' in namespace 'bookinfo', attempting to create them $ kubectl -n bookinfo get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE bigip-proxy-svc ClusterIP 10.235.45.250 <none> 3128/TCP 26s details ClusterIP 10.235.14.186 <none> 9080/TCP 6d11h productpage ClusterIP 10.235.37.112 <none> 9080/TCP 6d11h ratings ClusterIP 10.235.40.239 <none> 9080/TCP 6d11h reviews ClusterIP 10.235.1.21 <none> 9080/TCP 6d11h traffic-generator-productpage ClusterIP 10.235.17.158 <none> 80/TCP 6d11h $ kubectl -n bookinfo describe svc bigip-proxy-svc Name: bigip-proxy-svc Namespace: bookinfo Labels: apps=bigip-proxy Annotations: alpha.istio.io/kubernetes-serviceaccounts: default Selector: <none> Type: ClusterIP IP: 10.235.45.250 Port: 3128 3128/TCP TargetPort: 3128/TCP Endpoints: 10.4.0.201:3128 Session Affinity: None Events: <none> To test egress from bookinfo pod to external forward proxy (tinyproxy). Run "curl" accessing to Internet (www.f5.com) pointing to bigip-proxy-svc registered on Aspen Mesh. Example below shown executing curl binary inside "traffic-generator-productpage" pod. $ kubectl -n bookinfo exec -it $(kubectl -n bookinfo get pod -l app=traffic-generator-productpage -o jsonpath={.items..metadata.name}) -c traffic-generator -- curl -Ikx bigip-proxy-svc:3128 https://www.f5.com HTTP/1.0 200 Connection established Proxy-agent: tinyproxy/1.8.3 HTTP/1.1 200 OK Content-Type: text/html;charset=utf-8 Content-Length: 132986 Connection: keep-alive Accept-Ranges: bytes Cache-Control: no-cache="set-cookie" Content-Security-Policy: frame-ancestors 'self' *.cybersource.com *.salesforce.com *.force.com ; form-action *.cybersource.com *.salesforce.com *.force.com 'self' Date: Wed, 16 Dec 2020 06:19:48 GMT ETag: "2077a-5b68b3c0c5be0" Last-Modified: Wed, 16 Dec 2020 02:00:07 GMT Strict-Transport-Security: max-age=16070400; X-Content-Type-Options: nosniff X-Dispatcher: dispatcher1uswest2 X-Frame-Options: SAMEORIGIN X-Vhost: publish Via: 1.1 sin1-bit21, 1.1 24194e89802a1a492c5f1b22dc744e71.cloudfront.net (CloudFront) Vary: Accept-Encoding X-Cache: Hit from cloudfront X-Amz-Cf-Pop: MEL50-C2 X-Amz-Cf-Id: 7gE6sEaBP9WonZ0KjngDsr90dahHWFyDG0MwbuGn91uF7EkEJ_wdrQ== Age: 15713 Logs shown on classic BIG-IP Classic BIG-IP successfully authenticate with bookinfo with mTLS and permit access. Logs shown on forward proxy (tinyproxy). Source IP is SNATed to IP configured on classic BIG-IP. IP also allowed on network firewall. From other namespace (e.g. sm-apigw-a), try to access bigip-proxy-svc. Attempt shown rejected by classic BIG-IP. Example below shown executing curl binary in "nettools" pod. $ kubectl -n sm-apigw-a get pod NAME READY STATUS RESTARTS AGE httpbin-api-78bdd794bd-hfwkj 2/2 Running 2 22d nettools-9497dcc86-nhqmr 2/2 Running 2 22d podinfo-bbb7bf7c-j6wcs 2/2 Running 2 22d sm-apigw-a-85696f7455-rs9zh 3/3 Running 0 7d21h fbchan@logos:~/k8s-clusterX/k8s$ $ kubectl -n sm-apigw-a exec -it $(kubectl -n sm-apigw-a get pod -l app=nettools -o jsonpath={.items..metadata.name}) -c nettools -- curl -kIx bigip-proxy-svc.bookinfo.svc.cluster.local:3128 https://devcentral.f5.com curl: (56) Recv failure: Connection reset by peer command terminated with exit code 56 Classic BIG-IP Logs Classic BIG-IP reject sm-apigw-a namespace from using bigip-proxy-svc service. Summary Aspen Mesh is cloud-native Enterprise Ready Istio service mesh. Classic BIG-IP is a features rich application delivery controller (ADC). With Aspen Mesh, microservices are securely authenticated with mTLS with classic BIG-IP. Classic BIG-IP able to securely authenticate microservices apps and deliver application services based on your business and security requirement. This article addresses egress use cases. What about Ingress to Kubernetes cluster? How classic BIG-IP or cloud-native SPK coherently work together with Aspen Mesh to provides secure and consistent multi-cloud, multi-cluster application delivery services to your Kubernetes environment. This will be shared in future article. Stay tune.
3 Ways to Connect BIG-IP to Istio
Istio, a service mesh, uses “zero trust” to authenticate services.We’ll look at 3 ways to connect BIG-IP to Istio. 1. TCP The first method that we will use will be TCP.This will allow the BIG-IP to passthrough client traffic to Istio’s Ingress Gateway. 2. Mutual TLS (mTLS) The second method is to use the Client Certificate Constrained Delegation (C3D) feature of BIG-IP to authenticate client connections via mTLS and then generate a new client certificate (with similar attributes to the original) and use that newly minted certificate to authenticate to Istio. This second example is useful for scenarios where you are unable to install a trusted (externally CA signed) certificate into Istio (corporate policy prohibits it) and/or you want to establish a TLS DMZ.Despite the connection using mTLS the BIG-IP can inspect the traffic (i.e. log to Splunk), apply policy (i.e. insert XFF headers, WAF protection), etc… 3. JSON Web Tokens (JWT) Istio can use JWT tokens to authenticate users, but not all enterprise systems speak JWT.Using BIG-IP Access Policy Manager (APM) we can create an access policy that performs Single-Sign On (SSO) with an OAuth bearer token (JWT).This enables us to authenticate a client with username / password and convert the identity into a JWT token that is understood by Istio. Video Please These 3 methods are discussed and demo’d in the following YouTube video. Thanks for reading/watching!Building Trust with Service Mesh
The Istio Service Mesh is built on a “zero-trust network”; but how do you ensure that you can trust Istio? When trusted and untrusted services are co-mingled in a single Kubernetes cluster all services appear to originate from a single IP address. Only being able to use an IP address as an identifier makes traditional firewall and forward web proxies unable to provide different levels of assurance. We’ll look at 3 different ways that we can secure egress traffic from the service mesh Istio using both built-in mechanisms as well as BIG-IP and F5 SSL Orchestrator. What is a Service Mesh A Service Mesh is a pattern of deploying many “small” proxies to secure communication, provide additional audit capability, and monitoring. The Istio project uses the Envoy proxy and mutual TLS to secure communications between services deployed in a Kubernetes cluster. Trusting Istio When you deploy Istio you can opt to have all egress traffic blocked and create specific rules to permit traffic to specific endpoints. The first method to secure egress traffic is to use the built-in methods of creating an “egress gateway”. This allows you to restrict outbound connections from the service mesh by IP, Port, Name, and/or TLS ServerName. Ideally you would also like to audit that Istio’s policies are accurate by layering in additional security. Trust and Audit Istio uses mutual TLS to secure communication within the service mesh, but when traffic departs the mesh it loses the identity of the original service. A request from a trusted service will look the same as an untrusted service. Using Istio’s built-in mutual TLS we can extend the circle of trust to include a BIG-IP. This enables the BIG-IP to capture the original identity of the service and distinguish between trusted and untrusted services. An example of the information that we can capture when the BIG-IP has joined the mesh and configured as an explicit proxy: ..<http_proxy_request>: service: spiffe://cluster.local/ns/trusted/sa/sleep GET http://www.f5.com/company ..<http_proxy_request>: service: spiffe://cluster.local/ns/trusted/sa/sleep CONNECT www.f5.com:443 ..<http_proxy_request>: service: spiffe://cluster.local/ns/untrusted/sa/sleep GET http://www.f5.com/company ..<http_proxy_request>: service: spiffe://cluster.local/ns/untrusted/sa/sleep CONNECT www.f5.com:443 Service Chain of Trust In the previous example we saw the complete request for cleartext, HTTP, traffic, but encrypted, HTTPS, traffic is not visible. We can create a proxy chain from the BIG-IP to another F5 SSL Orchestrator device that can selectively intercept HTTPS traffic for untrusted services. In this example we can configure the SSL Orchestrator to act as an explicit proxy and send decrypted traffic to a passive L2 device. The Demo Please watch the demo video to see an example of this in a lab environment. You may be also interested in the following article about connecting legacy services (like a database) to a service mesh and securing its communication over mutual TLS.
