Announcing F5 NGINX Gateway Fabric 2.0.0 with a New Distributed Architecture
Gateway Fabric 2.0 marks our transition into a distributed architecture that is highly scalable, secure, and flexible. This architecture also easily enables more advanced capabilities and prepares us for integration for observability and fleet management with F5 NGINX One. Here are the big highlights for this major release: Control and Data Plane Separation Multiple Gateway Support HTTP Request Mirror Listener Isolation As always, bug fixesAnnouncing F5 NGINX Gateway Fabric 2.0.0 with a New Distributed Architecture
Today, F5 NGINX Gateway Fabric is reaching an important milestone. The release of NGINX Gateway Fabric 2.0 marks our transition into a distributed architecture that is highly scalable, secure, and flexible. This architecture also easily enables more advanced capabilities and prepares us for integration for observability and fleet management with F5 NGINX One. Here are the big highlights for this major release: Control and Data Plane Separation Multiple Gateway Support HTTP Request Mirror Listener Isolation As always, bug fixes Data and Control Plane Separation Before 2.0, NGINX Gateway Fabric contained both the “control plane,” the container responsible for reading and applying configuration, and the “data plane,” the NGINX container where all traffic flows through, within the same pod. This meant if you scaled the NGINX Gateway Fabric pod, you would be forced to scale the data and control plane together. Now with our distributed architecture, the data plane deploys in a separate pod from the control plane. This allows us to enable more flexible use cases such as multiple gateways per control plane, a highly available control plane, and directly scaling NGINX replicas per Gateway. This makes NGINX Gateway Fabric much more resource efficient at scale. This change also improves NGINX Gateway Fabric’s security posture by limiting how much is accessible if a single pod is compromised. While NGINX Gateway Fabric and NGINX have always been secure by default, this architecture enables a two-tier defense against potential attacks: any security intrusion on the control plane has no way to directly access traffic, nor can the data plane directly access control plane interfaces. Multiple Gateways NGINX Gateway Fabric has historically been limited to a single Gateway object. We first chose this architecture for the short term because Routes are a good way to separate routing and access control for developers and cluster operators. As our product matures, we know that more advanced use cases require the isolation of infrastructure for separate teams, customers, or SLAs. Our new architecture enables you to do just that: In NGINX Gateway Fabric 2.0, every time you define a Gateway object, the control plane will provision an NGINX deployment, which can then be independently scaled by adding more replicas if needed. With this pattern, it’s easy to create a second, third, or many more Gateways. You can choose to give each customer or team in your cluster their own Gateway so they can each own their own infrastructure. Or you may want to separate infrastructure to apply separate policies based on hostname or product group. Each Gateway can also be scaled independently for varying levels of traffic, all managed by a single control plane. HTTP Request Mirrors Request mirrors are often useful when you want to mirror traffic to analyze traffic for security issues or to test a new version of an application with production traffic without impacting current users. All request responses to a mirrored location are ignored, so request mirrors serve as another useful tool for testing and analysis. These mirrors are added using a filter on the route rule. They will only affect a small group of traffic you choose. If you need more, you can add as many as you need. Listener Isolation We decided to also include the concept of listener isolation in this release to make advanced configurations more intuitive to work with and guard against accidental misconfiguration. Listener isolation means that any request should match at most one Listener within a Gateway. The Gateway API lists this example below: If Listeners are defined for "foo.example.com" and "*.example.com", a request to "foo.example.com" SHOULD only be routed using routes attached to the "foo.example.com" Listener and NOT the "*.example.com" Listener. The alternative is that request may match against multiple listeners unintentionally, which can become a problem when different policies are applied to the other listeners, or the request is routed to the wrong location. Now, NGINX Gateway Fabric will ensure that a Route will only match the most specific Listener on the Gateway it is attached to. What’s Next Our next release will be primarily focused around delivering more extended features from the Gateway API in an effort to support all extended features at the extended support level. If you are unfamiliar, the Gateway API has three separate support levels for every feature in the specification: Core: Portable features that all implementations should support. Extended: Portable features that are not universally supported across implementations. Implementations that support the feature should have the same behavior and semantics. Some extended features may eventually move to core. Implementation-specific: These features are not portable and vendor-specific. These features are far less defined in API and schema. You can see the full description of support levels here. So far, NGINX Gateway Fabric has supported all core features since 1.0 and has slowly been adding extended features over the past few releases. For 2.1, we will have a greater focus on these features to enable more advanced use cases. We plan to have all extended features available in NGINX Gateway Fabric by 2.2. For F5 NGINX Plus users, 2.1 will bring connection to the F5 NGINX One Console with basic fleet management capabilities. We plan on expanding these capabilities to include Observability for all NGINX deployments in your organization in the near future. 2.1 will also include support for F5 NGINX App Protect Web Application Firewall to protect your applications from any incoming malicious traffic. Specifically, NGINX Gateway Fabric will be implementing support for v5, with a new configuration experience coming later this year. Resources For the complete changelog for NGINX Gateway Fabric 2.0.0, see the Release Notes. To try NGINX Gateway Fabric for Kubernetes with NGINX Plus, start your free 30-day trial today or contact us to discuss your use cases. If you would like to get involved, see what is coming next, or see the source code for NGINX Gateway Fabric, check out our repository on GitHub! We have weekly community meetings on Tuesdays at 9:30AM Pacific/12:30PM Eastern/5:30PM GMT. Meeting links, updates, agenda, and notes are on the NGINX Gateway Fabric Meeting Calendar. Links are also always available from our GitHub readme.
Can I deploy the application study tool on Kubernetes?(AST)
Hi, everyone. I followed the AST guide to ensure that AST works on Docker. But, I want to deploy this on Kubernetes, is there a how-to or guide for that? If you have any suggestions on availability, or any alternative links or instructions I can refer to, I would really appreciate it. Thank you for your help.BIG-IP Next does not work on KubeVirt
Hi all, I use a KubeVirt based Hypervisor called Harvester (https://harvesterhci.io) and tried to start a BIG-IP Next Instance with the BIG-IP-Next-20.3.0-2.716.2+0.0.50.qcow2.tar.gz image. The VM does start but I cannot see any service on tcp port 5443 running after doing setup. What I have seen when I checked the logs of the f5-platform-manager deployment is the following: 2025-02-06T12:52:25.985842479Z stdout F "ts"="2025-02-06 12:52:25.985"|"l"="info"|"m"="Found an unknown Bios vendor"|"id"="19093-000230"|"lt"="A"|"vendor"="KubeVirt"|"pod"="f5-platform-manager-6f78695744-p48tr"|"ct"="f5-platform-manager"|"v"="1.0"|"src"="surveyor/z100_detector.go:60" 2025-02-06T12:52:25.98585795Z stdout F "ts"="2025-02-06 12:52:25.985"|"l"="error"|"m"="Failed to run surveyor probe"|"id"="19093-000259"|"lt"="A"|"error"="Unsupported virtual platform: '{ChassisAssetTag: MachineId: Mfr:KubeVirt Product:None Serial: Uuid:aeada22c-3bf3-5220-a678-91f04ac6db0d Version:pc-q35-7.1}'"|"pod"="f5-platform-manager-6f78695744-p48tr"|"ct"="f5-platform-manager"|"v"="1.0"|"src"="surveyor/surveyor.go:344" So what can I do to get this BIG-IP Next Instance running on the VM? Thanks, PeterKubernetes architecture options with F5 Distributed Cloud Services
Summary F5 Distributed Cloud Services (F5 XC) can both integrate with your existing Kubernetes (K8s) clusters and/or host a K8s workload itself. Within these distinctions, we have multiple architecture options. This article explores four major architectures in ascending order of sophistication and advantages. Architecture #1: External Load Balancer (Secure K8s Gateway) Architecture #2: CE as a pod (K8s site) Architecture #3: Managed Namespace (vK8s) Architecture #4: Managed K8s (mK8s) Kubernetes Architecture Options As K8s continues to grow, options for how we run K8s and integrate with existing K8s platforms continue to grow. F5 XC can both integrate with your existing K8s clusters and/or run a managed K8s platform itself. Multiple architectures exist within these offerings too, so I was thoroughly confused when I first heard about these possibilities. A colleague recently laid it out for me in a conversation: "Michael, listen up: XC can either integrate with your K8s platform, run inside your K8s platform, host virtual K8s (Namespace-aaS), or run a K8s platform in your environment." I replied, "That's great. Now I have a mental model for differentiating between architecture options." This article will overview these architectures and provide 101-level context: when, how, and why would you implement these options? Side note 1: F5 XC concepts and terms F5 XC is a global platform that can provide networking and app delivery services, as well as compute (K8s workloads). We call each of our global PoP's a Regional Edge (RE). RE's are highly meshed to form the backbone of the global platform. They connect your sites, they can expose your services to the Internet, and they can run workloads. This platform is extensible into your data center by running one or more XC Nodes in your network, also called a Customer Edge (CE). A CE is a compute node in your network that registers to our global control plane and is then managed by a customer as SaaS. The registration of one or more CE's creates a customer site in F5 XC. A CE can run on a hypervisor (VMWare/KVM/Etc), a Hyperscaler (AWS, Azure, GCP, etc), baremetal, or even as a k8s pod, and can be deployed in HA clusters. XC Mesh functionality provides connectivity between sites, security services, and observability. Optionally, in addition, XC App Stack functionality allows a large and arbitrary number of managed clusters to be logically grouped into a virtual site with a single K8s mgmt interface. So where Mesh services provide the networking, App Stack services provide the Kubernetes compute mgmt. Our first 2 architectures require Mesh services only, and our last two require App Stack. Side note 2: Service-to-service communication I'm often asked how to allow services between clusters to communicate with each other. This is possible and easy with XC. Each site can publish services to every other site, including K8s sites. This means that any K8s service can be reachable from other sites you choose. And this can be true in any of the architectures below, although more granular controls are possible with the more sophisticated architectures. I'll explore this common question more in a separate article. Architecture 1: External Load Balancer (Secure K8s Gateway) In a Secure Kubernetes Gateway architecture, you have integration with your existing K8s platform, using the XC node as the external load balancer for your K8s cluster. In this scenario, you create a ServiceAccount and kubeconfig file to configure XC. The XC node then performs service discovery against your K8s API server. I've covered this process in a previous article, but the advantage is that you can integrate with existing K8s platforms. This allows exposing both NodePort and ClusterIP services via the XC node. XC is not hosting any workloads in this architecture, but it is exposing your services to your local network, or remote sites, or the Internet. In the diagram above, I show a web application being accesssed from a remote site (and/or the Internet) where the origin pool is a NodePort service discovered in a K8s cluster. Architecture 2: Run a site within a K8s cluster (K8s site type) Creating a K8s site is easy - just deploy a single manifest found here. This file deploys multiple resources in your cluster, and together these resources work to provide the services of a CE, and create a customer site. I've heard this referred to as "running a CE inside of K8s" or "running your CE as a pod". However, when I say "CE node" I'm usually referring to a discreet compute node like a VM or piece of hardware; this architecture is actually a group of pods and related resources that run within K8s to create a XC customer site. With XC running inside your existing cluster, you can expose services within the cluster by DNS name because the site will resolve these from within the cluster. Your service can then be exposed anywhere by the F5 XC platform. This is similar to Architecture 1 above, but with this model, your site is simply a group of pods within K8s. An advantage here is the ability to expose services of other types (e.g. ClusterIP). A site deployed into a K8s cluster will only support Mesh functionality and does not support AppStack functionality (i.e., you cannot run a cluster within your cluster). In this architecture, XC acts as a K8s ingress controller with built-in application security. It also enables Mesh features, such as publishing of other sites' services on this site, and publishing of this site's discovered services on other sites. Architecture 3: vK8s (Namespace-as-a-Service) If the services you use include AppStack capabilities, then architectures #3 and #4 are possible for you. In these scenarios, our XC node actually runs your K8s on your workloads. We are no longer integrating XC with your existing K8s platform. XC is the platform. A simple way to run K8s workloads is to use a virtual k8s (vK8s) architecture. This could be referred to as a "managed Namespace" because by creating a vK8s object in XC you get a single namespace in a virtual cluster. Your Namespace can be fully hosted (deployed to RE's) or run on your VM's (CE's), or both. Your kubeconfig file will allow access to your Namespace via the hosted API server. Via your regular kubectl CLI (or via the web console) you can create/delete/manage K8s resources (Deployments, Services, Secrets, ServiceAccounts, etc) and view application resource metrics. This is great if you have workloads that you want to deploy to remote regions where you do not have infrastructure and would prefer to run in F5's RE's, or if you have disparate clusters across multiple sites and you'd like to manage multiple K8s clusters via a single centralized, virtual cluster. Best practice guard rails for vK8s With a vK8s architecture, you don't have your own cluster, but rather a managed Namespace. So there are some restrictions (for example, you cannot run a container as root, bind to a privileged port, or to the Host network). You cannot create CRD's, ClusterRoles, PodSecurityPolicies, or Namespaces, so K8s operators are not supported. In short, you don't have a managed cluster, but a managed Namespace on a virtual cluster. Architecture 4: mK8s (Managed K8s) In managed k8s (mk8s, also known as physical K8s or pk8s) deployment, we have an enterprise-level K8s distribution that is run at your site. This means you can use XC to deploy/manage/upgrade K8s infrastructure, but you manage the Kubernetes resources. The benefits include what is typical for 3rd-party K8s mgmt solutions, but also some key differentiators: multi-cloud, with automation for Azure, AWS, and GCP environments consumed by you as SaaS enterprise-level traffic control natively allows a large and arbitrary number of managed clusters to be logically managed with a single K8s mgmt interface You can enable kubectl access against your local cluster and disable the hosted API server, so your kubeconfig file can point to a global URL or a local endpoint on-prem. Another benefit of mK8s is that you are running a full K8s cluster at your site, not just a Namespace in a virtual cluster. The restrictions that apply to vK8s (see above) do not apply to mK8s, so you could run privileged pods if required, use Operators that make use of ClusterRoles and CRDs, and perform other tasks that require cluster-wide access. Traffic management controls with mK8s Because your workloads run in a cluster managed by XC, we can apply more sophisticated and native policies to K8s traffic than non-managed clusters in earlier architectures: Service isolation can be enforced within the cluster, so that pods in a given namespace cannot communicate with services outside of that namespace, by default. More service-to-service controls exist so that you can decide which services can reach with other services with more granularity. Egress control can be natively enforced for outbound traffic from the cluster, by namespace, labels, IP ranges, or other methods. E.g.: Svc A can reach myapi.example.com but no other Internet service. WAF policies, bot defense, L3/4 policies, etc—all of these policies that you have typically applied with network firewalls, WAF's, etc—can be applied natively within the platform. This architecture took me a long time to understand, and longer to fully appreciate. But once you have run your workloads natively on a managed K8s platform that is connected to a global backbone and capable of performing network and application delivery within the platform, the security and traffic mgmt benefits become very compelling. Conclusion: As K8s continues to expand, management solutions of your clusters make it possible to secure your K8s services, whether they are managed by XC or exist in disparate clusters. With F5 XC as a global platform consumed as a service—not a discreet installation managed by you—the available architectures here are unique and therefore can accommodate the diverse (and changing!) ways we see K8s run today. Related Articles Securely connecting Kubernetes Microservices with F5 Distributed Cloud Multi-cluster Multi-cloud Networking for K8s with F5 Distributed Cloud - Architecture Pattern Multiple Kubernetes Clusters and Path-Based Routing with F5 Distributed CloudAnnouncing F5 NGINX Gateway Fabric 1.5.0 with NGINX Code Snippets
Today we are announcing the release of F5 NGINX Gateway Fabric 1.5.0 which comes with one of the most prominent features from F5 NGINX Ingress Controller: code snippets. As always, we have made this feature available through an extension of the Gateway API, a new resource called a SnippetFilter. With it, you can define custom NGINX configuration that can be applied to any Route rule! In addition to SnippetFilters, some other highlights of this release include: Ability to retain client IP information via Proxy Pass or XForwardedFor header Configurable NGINX error log level and reduced “info” log verbosity A new Getting Started guide for installing NGF for the first time F5 NGINX Plus R33 support Bug fixes! You can see the full changelog with details here! NGINX Code Snippets via SnippetFilters With a 20 year old legacy, there is a massive range of features that F5 NGINX provides that are not yet explicitly exposed through NGINX Gateway Fabric via the Gateway API. If you are familiar with NGINX directives and configuration and just want to use a native NGINX feature, you were unable to because our control plane would overwrite any configuration you wrote. We needed a way for your directives to merge with the configuration we pushed. With SnippetFilters, we have not only enabled you to append directives to whatever context you need them in but also built them as an extension to the Gateway API. Once you define a SnippetFilter, any route rule can then use that snippet filter to apply NGINX configuration for the context you supplied. You can even change higher level contexts using these snippets, such as main, to do things like importing modules – just make sure you coordinate with everyone when making these changes! As SnippetFilters do open the opportunity for misconfiguration in a way that may impact other application teams using the same Gateway, they do have to be explicitly enabled. But if you find yourself needing a feature from NGINX that is not yet available in NGINX Gateway Fabric, or you want to include your own NGINX logic, the SnippetFilter will fill that gap between our current first-class Gateway API features and the full power of NGINX. What’s Next Next release we will be focusing on a big architectural shift in NGINX Gateway Fabric that we have been planning for a while: separating our data and control planes. Currently, both our control plane (what configures NGINX for us) and our data plane (NGINX itself) are contained within the same pod. Ideally, these should be separate for security, performance, and scalability reasons. As we are also looking at supporting multiple Gateway resources for a single NGINX Gateway Fabric installation, we thought this was a good time to make that change, prompting our next release to be 2.0! In 2.0, you’ll see the control plane start as its own pod. As you create Gateways, or as you scale existing ones, the control plane will spin up new instances of NGINX in their own pods. This translates into less overhead, better RBAC security, and the ability to provision multiple Gateway resources. For more information, check out our design on the separation! Resources For the complete changelog for NGINX Gateway Fabric 1.5.0, see the Release Notes. To try NGINX Gateway Fabric for Kubernetes with NGINX Plus, start your free 30-day trial today or contact us to discuss your use cases. If you would like to get involved, see what is coming next, or see the source code for NGINX Gateway Fabric, check out our repository on GitHub! We have weekly community meetings on Tuesdays at 9:30AM Pacific/12:30PM Eastern/5:30PM GMT. The meeting link, updates, agenda, and notes are on the NGINX Gateway Fabric Meeting Calendar. Links are also always available from our GitHub README.5 Technical Sessions That Should Be Great: F5 AppWorld 2025
These F5 Academy sessions explore modern app delivery, security, and operations. The full list of sessions is on the F5 AppWorld 2025 Academy page - if you haven't yet registered you can do so here: Register for F5 AppWorld 2025 LAB - F5 Distributed Cloud: Discovering & Securing APIs API security has never been more critical, and this lab dives straight into the tough stuff. Learn how to find hidden endpoints, detect sensitive data and authentication states, and apply integrated API security measures to keep your environment locked tight. TECHNICAL BRIEFING - LLM Security and Delivery with F5’s Distributed Cloud Security Ecosystem AI is fueling the next wave of applications—but it’s also introducing new security blind spots. This briefing explores how to secure LLMs and integrate the right solutions to ensure your AI-driven workloads remain fast, cost-effective, and protected. LAB - F5 NGINX Plus Ingress as an API Gateway for Kubernetes Containerized environments and microservices are here to stay, and this lab helps you navigate the complexity. Configure NGINX Plus Ingress as a powerful API gateway for your Kubernetes workloads, enabling schema enforcement, authorization, and rate-limiting all in one streamlined solution. LAB - Zero Trust at Scale With F5 NGINX Zero trust principles become a whole lot more meaningful when you can scale them. Get hands-on with NGINX Plus and BIG-IP GTM to build a robust, scalable zero trust architecture, ensuring secure and seamless app access across enterprises and multi-cluster Kubernetes environments. LAB - F5 Distributed Cloud: Security Automation & Zero Day Mitigation In this lab, you’ll learn how to leverage advanced matching criteria and custom rules to quickly respond to emerging threats. Shore up your defenses with automated policies that deliver frictionless security and agile zero-day mitigation. Session Updates Coming in January 🚨 AppWorld's Breakout Sessions officially drop in January 2025 but here is a sneak preview! Check back in January to add these to your agenda. Global App Delivery With a Global Network How Generative AI Breaks Traditional Application Security and What You Can Do About It The New Wave of Bots: A Deep Dive into Residential IP Proxy Networks From ZTNA to Universal ZTNA: Expanding Your App Security Strategy --- See you at F5 AppWorld 2025! #AppWorld25NGINX Unit running in Distributed Cloud vK8s
F5 Distributed Cloud provides a mechanism to easily deploy applications using virtual Kubernetes (vK8s) across a global network. This results in applications running closer to the end user. This article will demonstrate creating a NGINX Unit container to run a simple server-side WebAssembly (Wasm) application and deliver it via Distributed Cloud Regional Edge (RE) sites. Distributed Cloud provides a vK8s infrastructure for deploying modern apps as well as load balancing and security services to securely and reliably deliver applications at the edge. NGINX Unit Pod A Kubernetes pod is the smallest execution unit that can be deployed in Kubernetes. A pod is a single instance of an application and may contain single or multiple containers. For this article, a single NGINX Unit container will make up the application pod and replicas of the pod will be deployed to each of the configured Distributed Cloud Points of Presence (PoPs). Building the NGINX Unit Container The first step is to build a server-side WebAssemby (Wasm) application that Unit can execute. Instructions on creating a simple Hello World Wasm application are available here. After the Hello World Wasm component is built, a NGINX Unit Container needs to be created to execute the application. To build the NGINX Unit container, download the Unit Wasm Dockerfile that is available on the NGINX Unit github repo. The Dockerfile needs to be modified to allow the container to run as a non-root user in the Distributed Cloud virtual Kubernetes (vK8s) environment. Below is the Dockerfile that I used for this demonstration. FROM debian:bullseye-slim LABEL org.opencontainers.image.title="Unit (wasm)" LABEL org.opencontainers.image.description="Official build of Unit for Docker." LABEL org.opencontainers.image.url="https://unit.nginx.org" LABEL org.opencontainers.image.source="https://github.com/nginx/unit" LABEL org.opencontainers.image.documentation="https://unit.nginx.org/installation/#docker-images" LABEL org.opencontainers.image.vendor="NGINX Docker Maintainers <docker-maint@nginx.com>" LABEL org.opencontainers.image.version="1.32.1" RUN set -ex \ && savedAptMark="$(apt-mark showmanual)" \ && apt-get update \ && apt-get install --no-install-recommends --no-install-suggests -y ca-certificates git build-essential libssl-dev libpcre2-dev curl pkg-config \ && mkdir -p /usr/lib/unit/modules /usr/lib/unit/debug-modules \ && mkdir -p /usr/src/unit \ && mkdir -p /unit/var/lib/unit \ && mkdir -p /unit/var/run \ && mkdir -p /unit/var/log \ && mkdir -p /unit/var/tmp \ && mkdir /app \ && chmod -R 777 /unit/var/lib/unit \ && chmod -R 777 /unit/var/run \ && chmod -R 777 /unit/var/log \ && chmod -R 777 /unit/var/tmp \ && chmod 777 /app \ && cd /usr/src/unit \ && git clone --depth 1 -b 1.32.1-1 https://github.com/nginx/unit \ && cd unit \ && NCPU="$(getconf _NPROCESSORS_ONLN)" \ && DEB_HOST_MULTIARCH="$(dpkg-architecture -q DEB_HOST_MULTIARCH)" \ && CC_OPT="$(DEB_BUILD_MAINT_OPTIONS="hardening=+all,-pie" DEB_CFLAGS_MAINT_APPEND="-Wp,-D_FORTIFY_SOURCE=2 -fPIC" dpkg-buildflags --get CFLAGS)" \ && LD_OPT="$(DEB_BUILD_MAINT_OPTIONS="hardening=+all,-pie" DEB_LDFLAGS_MAINT_APPEND="-Wl,--as-needed -pie" dpkg-buildflags --get LDFLAGS)" \ && CONFIGURE_ARGS_MODULES="--prefix=/usr \ --statedir=/unit/var/lib/unit \ --debug \ --control=unix:/unit/var/run/control.unit.sock \ --runstatedir=/unit/var/run \ --pid=/unit/var/run/unit.pid \ --logdir=/unit/var/log \ --log=/unit/var/log/unit.log \ --tmpdir=/unit/var/tmp \ --openssl \ --libdir=/usr/lib/$DEB_HOST_MULTIARCH" \ && CONFIGURE_ARGS="$CONFIGURE_ARGS_MODULES \ --njs" \ && make -j $NCPU -C pkg/contrib .njs \ && export PKG_CONFIG_PATH=$(pwd)/pkg/contrib/njs/build \ && ./configure $CONFIGURE_ARGS --cc-opt="$CC_OPT" --ld-opt="$LD_OPT" --modulesdir=/usr/lib/unit/debug-modules --debug \ && make -j $NCPU unitd \ && install -pm755 build/sbin/unitd /usr/sbin/unitd-debug \ && make clean \ && ./configure $CONFIGURE_ARGS --cc-opt="$CC_OPT" --ld-opt="$LD_OPT" --modulesdir=/usr/lib/unit/modules \ && make -j $NCPU unitd \ && install -pm755 build/sbin/unitd /usr/sbin/unitd \ && make clean \ && apt-get install --no-install-recommends --no-install-suggests -y libclang-dev \ && export RUST_VERSION=1.76.0 \ && export RUSTUP_HOME=/usr/src/unit/rustup \ && export CARGO_HOME=/usr/src/unit/cargo \ && export PATH=/usr/src/unit/cargo/bin:$PATH \ && dpkgArch="$(dpkg --print-architecture)" \ && case "${dpkgArch##*-}" in \ amd64) rustArch="x86_64-unknown-linux-gnu"; rustupSha256="0b2f6c8f85a3d02fde2efc0ced4657869d73fccfce59defb4e8d29233116e6db" ;; \ arm64) rustArch="aarch64-unknown-linux-gnu"; rustupSha256="673e336c81c65e6b16dcdede33f4cc9ed0f08bde1dbe7a935f113605292dc800" ;; \ *) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \ esac \ && url="https://static.rust-lang.org/rustup/archive/1.26.0/${rustArch}/rustup-init" \ && curl -L -O "$url" \ && echo "${rustupSha256} *rustup-init" | sha256sum -c - \ && chmod +x rustup-init \ && ./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch} \ && rm rustup-init \ && rustup --version \ && cargo --version \ && rustc --version \ && make -C pkg/contrib .wasmtime \ && install -pm 755 pkg/contrib/wasmtime/target/release/libwasmtime.so /usr/lib/$(dpkg-architecture -q DEB_HOST_MULTIARCH)/ \ && ./configure $CONFIGURE_ARGS_MODULES --cc-opt="$CC_OPT" --modulesdir=/usr/lib/unit/debug-modules --debug \ && ./configure wasm --include-path=`pwd`/pkg/contrib/wasmtime/crates/c-api/include --lib-path=/usr/lib/$(dpkg-architecture -q DEB_HOST_MULTIARCH)/ && ./configure wasm-wasi-component \ && make -j $NCPU wasm-install wasm-wasi-component-install \ && make clean \ && ./configure $CONFIGURE_ARGS_MODULES --cc-opt="$CC_OPT" --modulesdir=/usr/lib/unit/modules \ && ./configure wasm --include-path=`pwd`/pkg/contrib/wasmtime/crates/c-api/include --lib-path=/usr/lib/$(dpkg-architecture -q DEB_HOST_MULTIARCH)/ && ./configure wasm-wasi-component \ && make -j $NCPU wasm-install wasm-wasi-component-install \ && cd \ && rm -rf /usr/src/unit \ && for f in /usr/sbin/unitd /usr/lib/unit/modules/*.unit.so; do \ ldd $f | awk '/=>/{print $(NF-1)}' | while read n; do dpkg-query -S $n; done | sed 's/^\([^:]\+\):.*$/\1/' | sort | uniq >> /requirements.apt; \ done \ && apt-mark showmanual | xargs apt-mark auto > /dev/null \ && { [ -z "$savedAptMark" ] || apt-mark manual $savedAptMark; } \ && /bin/true \ && mkdir -p /var/lib/unit/ \ && mkdir -p /docker-entrypoint.d/ \ && apt-get update \ && apt-get --no-install-recommends --no-install-suggests -y install curl $(cat /requirements.apt) \ && apt-get purge -y --auto-remove build-essential \ && rm -rf /var/lib/apt/lists/* \ && rm -f /requirements.apt \ && ln -sf /dev/stderr /var/log/unit.log WORKDIR /app COPY --chmod=755 hello_wasi_http.wasm /app COPY ./*.json /docker-entrypoint.d/ COPY --chmod=755 docker-entrypoint.sh /usr/local/bin/ COPY welcome.* /usr/share/unit/welcome/ STOPSIGNAL SIGTERM ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"] EXPOSE 8000 CMD ["unitd", "--no-daemon", "--control", "unix:/unit/var/run/control.unit.sock"] The changes to the Dockerfile are due to the non-root user not being able to write to the /var directory. Rather than modify the permissions on /var and its subdirectories, this docker file creates a /unit directory that is used for storing process information and logs. The docker-entrypoint.sh script also needed to be modified to use the /unit directory structure. The updated docker-entrypoint.sh script is shown below. #!/bin/sh set -e WAITLOOPS=5 SLEEPSEC=1 curl_put() { RET=$(/usr/bin/curl -s -w '%{http_code}' -X PUT --data-binary @$1 --unix-socket /unit/var/run/control.unit.sock http://localhost/$2) RET_BODY=$(echo $RET | /bin/sed '$ s/...$//') RET_STATUS=$(echo $RET | /usr/bin/tail -c 4) if [ "$RET_STATUS" -ne "200" ]; then echo "$0: Error: HTTP response status code is '$RET_STATUS'" echo "$RET_BODY" return 1 else echo "$0: OK: HTTP response status code is '$RET_STATUS'" echo "$RET_BODY" fi return 0 } if [ "$1" = "unitd" ] || [ "$1" = "unitd-debug" ]; then if /usr/bin/find "/unit/var/lib/unit/" -mindepth 1 -print -quit 2>/dev/null | /bin/grep -q .; then echo "$0: /unit/var/lib/unit/ is not empty, skipping initial configuration..." else echo "$0: Launching Unit daemon to perform initial configuration..." /usr/sbin/$1 --control unix:/unit/var/run/control.unit.sock for i in $(/usr/bin/seq $WAITLOOPS); do if [ ! -S /unit/var/run/control.unit.sock ]; then echo "$0: Waiting for control socket to be created..." /bin/sleep $SLEEPSEC else break fi done # even when the control socket exists, it does not mean unit has finished initialisation # this curl call will get a reply once unit is fully launched /usr/bin/curl -s -X GET --unix-socket /unit/var/run/control.unit.sock http://localhost/ if /usr/bin/find "/docker-entrypoint.d/" -mindepth 1 -print -quit 2>/dev/null | /bin/grep -q .; then echo "$0: /docker-entrypoint.d/ is not empty, applying initial configuration..." echo "$0: Looking for certificate bundles in /docker-entrypoint.d/..." for f in $(/usr/bin/find /docker-entrypoint.d/ -type f -name "*.pem"); do echo "$0: Uploading certificates bundle: $f" curl_put $f "certificates/$(basename $f .pem)" done echo "$0: Looking for JavaScript modules in /docker-entrypoint.d/..." for f in $(/usr/bin/find /docker-entrypoint.d/ -type f -name "*.js"); do echo "$0: Uploading JavaScript module: $f" curl_put $f "js_modules/$(basename $f .js)" done echo "$0: Looking for configuration snippets in /docker-entrypoint.d/..." for f in $(/usr/bin/find /docker-entrypoint.d/ -type f -name "*.json"); do echo "$0: Applying configuration $f"; curl_put $f "config" done echo "$0: Looking for shell scripts in /docker-entrypoint.d/..." for f in $(/usr/bin/find /docker-entrypoint.d/ -type f -name "*.sh"); do echo "$0: Launching $f"; "$f" done # warn on filetypes we don't know what to do with for f in $(/usr/bin/find /docker-entrypoint.d/ -type f -not -name "*.sh" -not -name "*.json" -not -name "*.pem" -not -name "*.js"); do echo "$0: Ignoring $f"; done else echo "$0: /docker-entrypoint.d/ is empty, creating 'welcome' configuration..." curl_put /usr/share/unit/welcome/welcome.json "config" fi echo "$0: Stopping Unit daemon after initial configuration..." kill -TERM $(/bin/cat /unit/var/run/unit.pid) for i in $(/usr/bin/seq $WAITLOOPS); do if [ -S /unit/var/run/control.unit.sock ]; then echo "$0: Waiting for control socket to be removed..." /bin/sleep $SLEEPSEC else break fi done if [ -S /unit/var/run/control.unit.sock ]; then kill -KILL $(/bin/cat /unit/var/run/unit.pid) rm -f /unit/var/run/control.unit.sock fi echo echo "$0: Unit initial configuration complete; ready for start up..." echo fi fi exec "$@" The Dockerfile copies all files ending in .json to a directory named /docker-entrypoint.d. The docker-entrypoint.sh script checks the /docker-entrypoint.d directory for configuration files that are used to configure NGINX Unit. The following config.json file was created in the same directory as the Dockerfile and docker-entrypoint.sh script. { "listeners": { "*:8000": { "pass": "applications/wasm" } }, "applications": { "wasm": { "type": "wasm-wasi-component", "component": "/app/hello_wasi_http.wasm" } } } This config file instructs NGINX unit to listen on port 8000 and pass any client request to the Wasm application component located at /app/hello_wasi_http.wasm. The last step before building the container is to copy the Wasm application component into the same directory as the Dockerfile, docker-entrypoint.sh, and config.json files. With these four files in the same directory, the docker build command is used to create a container: docker build --no-cache --platform linux/amd64 -t hello-wasi-xc:1.0 . Here I use the --platform option to specify that the container will run on a linux/amd64 platform. I also use -t to give the image a name. The portion of the name after the ":" can be used for versioning. For example, if I added a new feature to my application, I could create a new image with -t hello-wasi-xc:1.1 to represent version 1.1 of the hello-wasi-xc application. Once the image is built, it can be uploaded to the container registry of your choice. I have access to an Azure Container Registry, so that is what I chose to use. Distributed Cloud supports pulling images from both public and private container registries. To push the image to my registry, I had to first tag the image with the registry location ({{registry_name}} should be replaced with the name of your registry): docker tag hello-wasi-xc:1.0 {{registry_name}}/examples/hello-wasi-xc:1.0 Next, I logged into my Azure registry: az login az acr login --name {{registry_name}} I then pushed the image to the registry: docker push {{registry_name}}/examples/hello-wasi-xc:1.0 F5 Distributed Cloud Virtual Kubernetes F5 Distributed Cloud Services support a Kubernetes compatible API for centralized orchestration of applications across a fleet of sites (customer sites or F5 Distributed Cloud Regional Edges). Distributed Cloud utilizes a distributed control plane to manage scheduling and scaling of applications across multiple sites. Kubernetes Objects Virtual Kubernetes supports the following Kubernetes objects: Deployments, StatefulSets, Jobs, CronJob, DaemonSet, Service, ConfigMap, Secrets, PersistentVolumeClaim, ServiceAccount, Role, and Role Binding. For this article I will focus on the Deployments object. A deployment is commonly used for stateless applications like web-servers, front-end web-ui, etc. A deployment works well for this use case because the NGINX Unit container is serving up a stateless web application. Workload Object A workload within Distributed Cloud is used to configure and deploy components of an application in Virtual Kubernetes. Workload encapsulates all the operational characteristics of Kubernetes workload, storage, and network objects (deployments, statefulsets, jobs, persistent volume claims, configmaps, secrets, and services) configuration, as well as configuration related to where the workload is deployed and how it is advertised using L7 or L4 load balancers. Within Distributed Cloud there are four types of workloads: Simple Service, Service, Stateful Service, and Job. Namespaces In Distributed Cloud, tenant configuration objects are grouped under namespaces. Namespaces can be thought of as administrative domains. Within each Namespace, a user can create an object called vK8s and use that for application management. Each Namespace can have a maximum of one vK8s object. Distributed Cloud vK8s Configuration vK8s is configured under the Distributed Apps tile within the Distributed Cloud console. After clicking on that tile, I next created a new Virtual K8s. By clicking Add Virtual K8s from the Virtual K8s menu under Applications. This brings up the Virtual K8s configuration form. I provided a name for my vK8s site and click Save and Exit. This initiates the vK8s build. Deploy the Application Once the vK8s cluster is ready, I click on the name of my vk8s to configure the deployment of my application. For this demo, I am going to deploy my application via Workloads. I click on Workloads and then Add VK8s Workload. This brings up the Workload configuration form. I provide my Workload a name, select Service for Workload type and click Configure. Using Service allows me to have more granular controls on how my application will be advertised. Next, I click Add Item under containers to specify to Distribtued Cloud where it can pull my container image from. In the resulting form, I supply a name for my container, supply the public registry FQDN along with the container name and version and click Apply. Next I configure where to advertise the application. I want to manually configure my HTTP LB, so I chose to advertise in Cluster. Manually configuring the LB allows me to configure additional options such as Web Application Firewall (WAF). Within the Advertise in Cluster form, I specify port 8000, because that is the port my container is listening on. Click Apply and then click Save and Exit. Create an HTTP Load Balancer to Advertise the Application The next step is to create an HTTP Load Balancer to expose the application to the Internet. This can be done by clicking Manage, selecting Load Balancers, and clicking on HTTP Load Balancers. Next, click on Add HTTP Load Balancer to create a new load balancer. I gave the LB a Name, Domain, Load Balancer Type, and Checked the box to Automatically Manage DNS Records. Then I configured the Origin Pool by clicking Add Item in the Origin Pool section and then clicking Add Item under the Origin Pool. On the resulting form, I created a name for my Origin Pool and then clicked Add Item under Origin Servers to add an origin Server. In the Origin Server form, I selected K8s Service Name of Origin Server on given Sites, Service Name, and provided my vK8s workload name along with my namespace. I also selected Virtual Site and created a Virtual Site for the REs I wanted to use to access my application and clicked Continue. Back on the Origin Server form, I selected vK8s Networks on Site from the Select Network on the site drop down and then clicked Apply. On the Origin Pool form, I specified port 8000 for my origin server Port and then clicked Continue. I then clicked Apply to return to the LB configuration form. I added a WAF policy to protect my application and then clicked Save and Exit. At this point my application is now accessible on the domain name I specified in the HTTP Load Balancer config (http://aconley-wasm.amer-ent.f5demos.com/). Conclusion This is a basic demo of using NGINX Unit and Distributed Cloud Virtual Kubernetes to deploy a server-side WASM application. The principals in this demo could be expanded to build more complex applications.
BIG-IP for SIP resources running in Kubernetes
Hello, We are trying to setup Virtual Server using BIG-IP that would server as a Load Balancer for SIP traffic for resources that are deployed in Kubernetes cluster and exposed through NodePort. Our F5 is not part of the Kubernetes cluster and it is a standalone Virtual Machine that sends its traffic to NodePort service of our SIP resources. We are facing few issues and hope someone can help us understand them. UDP not working When we try to use UDP the problem is that F5 (10.224.64.223) sends SIP OPTIONS to ip address/port that we defined as access point for SIP elements in Kubernetes (Node IP and NodePort port, 10.224.64.222, port:31131). But due to Kubernetes deployment, responses are sent from different IP address and port (10.224.64.222, port 30834). And this gets rejected by the F5. 10:17:23.695039 IP 10.224.64.223.51938 > 10.224.64.222.31131: UDP, length 575 out slot1/tmm1 lis=mon_mrf_sip_udp port=1.2 trunk= 10:17:23.700849 IP 10.224.64.220.30834 > 10.224.64.223.51938: UDP, length 520 in slot1/tmm0 lis= port=1.2 trunk= 10:17:23.700949 IP 10.224.64.223 > 10.224.64.220: ICMP 10.224.64.223 udp port 51938 unreachable, length 36 out slot1/tmm0 lis= port=1.2 trunk= Even the usage of macvlan on Kubernetes pods does not help. With macvlan we manage to achieve that IP address is preserved (10.226.64.225), but still the port changes (5060 -> 25404). And F5 rejects it. 10:42:07.370926 IP 10.224.64.223.54412 > 10.224.64.225.5060: SIP: OPTIONS sip:10.224.64.225:5060 SIP/2.0 out slot1/tmm0 lis= port=1.2 trunk= 10:42:07.378237 IP 10.224.64.225.25404 > 10.224.64.223.54412: UDP, length 425 in slot1/tmm0 lis= port=1.2 trunk= 10:42:07.378325 IP 10.224.64.223 > 10.224.64.225: ICMP 10.224.64.223 udp port 54412 unreachable, length 36 out slot1/tmm0 lis= port=1.2 trunk= So I guess there is no way to have it working for UDP at all with resources being deployed in Kubernets cluster? (host-network is not an option). TCP (in Message Routing mode) not working When we try to use TCP we found out that "Standard (SIP - legacy profile)" mode behaves differently then "Message Routing" one. In case when we use "Legacy" SIP monitor via TCP it establishes a TCP connection with destination server prior to sending the SIP Options message. This is OK for us. But when we try to use "Message Routing" (from what I understood this is generally advisable for SIP traffic) for TCP monitoring, TCP connection is not established before OPTIONS message is sent and this is not acceptable by our SIP servers. So I have few questions: Is it even possible to use F5 BIG-IP TLM VE as SIP LB for SIP resources operating in Kubernetes cluster (for both UDP and TCP) or the ONLY option is to use F5 BIG-IP Next Service Proxy Kubernetes (SPK) for SIP traffic? Is there a way to somehow force F5 that does Monitoring usin Message Routing mode to open TCP connection prior to sending SIP requests? Due to UDP problem above (that probably is solvable only if SPK version is used) is some way for F5 to do the UDP-2-TCP conversion of SIP traffic? Kind Regards, Zvonimir
