NGINX Unit running in Distributed Cloud vK8s
F5 Distributed Cloud provides a mechanism to easily deploy applications using virtual Kubernetes (vK8s) across a global network. This results in applications running closer to the end user. This article will demonstrate creating a NGINX Unit container to run a simple server-side WebAssembly (Wasm) application and deliver it via Distributed Cloud Regional Edge (RE) sites. Distributed Cloud provides a vK8s infrastructure for deploying modern apps as well as load balancing and security services to securely and reliably deliver applications at the edge. NGINX Unit Pod A Kubernetes pod is the smallest execution unit that can be deployed in Kubernetes. A pod is a single instance of an application and may contain single or multiple containers. For this article, a single NGINX Unit container will make up the application pod and replicas of the pod will be deployed to each of the configured Distributed Cloud Points of Presence (PoPs). Building the NGINX Unit Container The first step is to build a server-side WebAssemby (Wasm) application that Unit can execute. Instructions on creating a simple Hello World Wasm application are available here. After the Hello World Wasm component is built, a NGINX Unit Container needs to be created to execute the application. To build the NGINX Unit container, download the Unit Wasm Dockerfile that is available on the NGINX Unit github repo. The Dockerfile needs to be modified to allow the container to run as a non-root user in the Distributed Cloud virtual Kubernetes (vK8s) environment. Below is the Dockerfile that I used for this demonstration. FROM debian:bullseye-slim LABEL org.opencontainers.image.title="Unit (wasm)" LABEL org.opencontainers.image.description="Official build of Unit for Docker." LABEL org.opencontainers.image.url="https://unit.nginx.org" LABEL org.opencontainers.image.source="https://github.com/nginx/unit" LABEL org.opencontainers.image.documentation="https://unit.nginx.org/installation/#docker-images" LABEL org.opencontainers.image.vendor="NGINX Docker Maintainers <docker-maint@nginx.com>" LABEL org.opencontainers.image.version="1.32.1" RUN set -ex \ && savedAptMark="$(apt-mark showmanual)" \ && apt-get update \ && apt-get install --no-install-recommends --no-install-suggests -y ca-certificates git build-essential libssl-dev libpcre2-dev curl pkg-config \ && mkdir -p /usr/lib/unit/modules /usr/lib/unit/debug-modules \ && mkdir -p /usr/src/unit \ && mkdir -p /unit/var/lib/unit \ && mkdir -p /unit/var/run \ && mkdir -p /unit/var/log \ && mkdir -p /unit/var/tmp \ && mkdir /app \ && chmod -R 777 /unit/var/lib/unit \ && chmod -R 777 /unit/var/run \ && chmod -R 777 /unit/var/log \ && chmod -R 777 /unit/var/tmp \ && chmod 777 /app \ && cd /usr/src/unit \ && git clone --depth 1 -b 1.32.1-1 https://github.com/nginx/unit \ && cd unit \ && NCPU="$(getconf _NPROCESSORS_ONLN)" \ && DEB_HOST_MULTIARCH="$(dpkg-architecture -q DEB_HOST_MULTIARCH)" \ && CC_OPT="$(DEB_BUILD_MAINT_OPTIONS="hardening=+all,-pie" DEB_CFLAGS_MAINT_APPEND="-Wp,-D_FORTIFY_SOURCE=2 -fPIC" dpkg-buildflags --get CFLAGS)" \ && LD_OPT="$(DEB_BUILD_MAINT_OPTIONS="hardening=+all,-pie" DEB_LDFLAGS_MAINT_APPEND="-Wl,--as-needed -pie" dpkg-buildflags --get LDFLAGS)" \ && CONFIGURE_ARGS_MODULES="--prefix=/usr \ --statedir=/unit/var/lib/unit \ --debug \ --control=unix:/unit/var/run/control.unit.sock \ --runstatedir=/unit/var/run \ --pid=/unit/var/run/unit.pid \ --logdir=/unit/var/log \ --log=/unit/var/log/unit.log \ --tmpdir=/unit/var/tmp \ --openssl \ --libdir=/usr/lib/$DEB_HOST_MULTIARCH" \ && CONFIGURE_ARGS="$CONFIGURE_ARGS_MODULES \ --njs" \ && make -j $NCPU -C pkg/contrib .njs \ && export PKG_CONFIG_PATH=$(pwd)/pkg/contrib/njs/build \ && ./configure $CONFIGURE_ARGS --cc-opt="$CC_OPT" --ld-opt="$LD_OPT" --modulesdir=/usr/lib/unit/debug-modules --debug \ && make -j $NCPU unitd \ && install -pm755 build/sbin/unitd /usr/sbin/unitd-debug \ && make clean \ && ./configure $CONFIGURE_ARGS --cc-opt="$CC_OPT" --ld-opt="$LD_OPT" --modulesdir=/usr/lib/unit/modules \ && make -j $NCPU unitd \ && install -pm755 build/sbin/unitd /usr/sbin/unitd \ && make clean \ && apt-get install --no-install-recommends --no-install-suggests -y libclang-dev \ && export RUST_VERSION=1.76.0 \ && export RUSTUP_HOME=/usr/src/unit/rustup \ && export CARGO_HOME=/usr/src/unit/cargo \ && export PATH=/usr/src/unit/cargo/bin:$PATH \ && dpkgArch="$(dpkg --print-architecture)" \ && case "${dpkgArch##*-}" in \ amd64) rustArch="x86_64-unknown-linux-gnu"; rustupSha256="0b2f6c8f85a3d02fde2efc0ced4657869d73fccfce59defb4e8d29233116e6db" ;; \ arm64) rustArch="aarch64-unknown-linux-gnu"; rustupSha256="673e336c81c65e6b16dcdede33f4cc9ed0f08bde1dbe7a935f113605292dc800" ;; \ *) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \ esac \ && url="https://static.rust-lang.org/rustup/archive/1.26.0/${rustArch}/rustup-init" \ && curl -L -O "$url" \ && echo "${rustupSha256} *rustup-init" | sha256sum -c - \ && chmod +x rustup-init \ && ./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch} \ && rm rustup-init \ && rustup --version \ && cargo --version \ && rustc --version \ && make -C pkg/contrib .wasmtime \ && install -pm 755 pkg/contrib/wasmtime/target/release/libwasmtime.so /usr/lib/$(dpkg-architecture -q DEB_HOST_MULTIARCH)/ \ && ./configure $CONFIGURE_ARGS_MODULES --cc-opt="$CC_OPT" --modulesdir=/usr/lib/unit/debug-modules --debug \ && ./configure wasm --include-path=`pwd`/pkg/contrib/wasmtime/crates/c-api/include --lib-path=/usr/lib/$(dpkg-architecture -q DEB_HOST_MULTIARCH)/ && ./configure wasm-wasi-component \ && make -j $NCPU wasm-install wasm-wasi-component-install \ && make clean \ && ./configure $CONFIGURE_ARGS_MODULES --cc-opt="$CC_OPT" --modulesdir=/usr/lib/unit/modules \ && ./configure wasm --include-path=`pwd`/pkg/contrib/wasmtime/crates/c-api/include --lib-path=/usr/lib/$(dpkg-architecture -q DEB_HOST_MULTIARCH)/ && ./configure wasm-wasi-component \ && make -j $NCPU wasm-install wasm-wasi-component-install \ && cd \ && rm -rf /usr/src/unit \ && for f in /usr/sbin/unitd /usr/lib/unit/modules/*.unit.so; do \ ldd $f | awk '/=>/{print $(NF-1)}' | while read n; do dpkg-query -S $n; done | sed 's/^\([^:]\+\):.*$/\1/' | sort | uniq >> /requirements.apt; \ done \ && apt-mark showmanual | xargs apt-mark auto > /dev/null \ && { [ -z "$savedAptMark" ] || apt-mark manual $savedAptMark; } \ && /bin/true \ && mkdir -p /var/lib/unit/ \ && mkdir -p /docker-entrypoint.d/ \ && apt-get update \ && apt-get --no-install-recommends --no-install-suggests -y install curl $(cat /requirements.apt) \ && apt-get purge -y --auto-remove build-essential \ && rm -rf /var/lib/apt/lists/* \ && rm -f /requirements.apt \ && ln -sf /dev/stderr /var/log/unit.log WORKDIR /app COPY --chmod=755 hello_wasi_http.wasm /app COPY ./*.json /docker-entrypoint.d/ COPY --chmod=755 docker-entrypoint.sh /usr/local/bin/ COPY welcome.* /usr/share/unit/welcome/ STOPSIGNAL SIGTERM ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"] EXPOSE 8000 CMD ["unitd", "--no-daemon", "--control", "unix:/unit/var/run/control.unit.sock"] The changes to the Dockerfile are due to the non-root user not being able to write to the /var directory. Rather than modify the permissions on /var and its subdirectories, this docker file creates a /unit directory that is used for storing process information and logs. The docker-entrypoint.sh script also needed to be modified to use the /unit directory structure. The updated docker-entrypoint.sh script is shown below. #!/bin/sh set -e WAITLOOPS=5 SLEEPSEC=1 curl_put() { RET=$(/usr/bin/curl -s -w '%{http_code}' -X PUT --data-binary @$1 --unix-socket /unit/var/run/control.unit.sock http://localhost/$2) RET_BODY=$(echo $RET | /bin/sed '$ s/...$//') RET_STATUS=$(echo $RET | /usr/bin/tail -c 4) if [ "$RET_STATUS" -ne "200" ]; then echo "$0: Error: HTTP response status code is '$RET_STATUS'" echo "$RET_BODY" return 1 else echo "$0: OK: HTTP response status code is '$RET_STATUS'" echo "$RET_BODY" fi return 0 } if [ "$1" = "unitd" ] || [ "$1" = "unitd-debug" ]; then if /usr/bin/find "/unit/var/lib/unit/" -mindepth 1 -print -quit 2>/dev/null | /bin/grep -q .; then echo "$0: /unit/var/lib/unit/ is not empty, skipping initial configuration..." else echo "$0: Launching Unit daemon to perform initial configuration..." /usr/sbin/$1 --control unix:/unit/var/run/control.unit.sock for i in $(/usr/bin/seq $WAITLOOPS); do if [ ! -S /unit/var/run/control.unit.sock ]; then echo "$0: Waiting for control socket to be created..." /bin/sleep $SLEEPSEC else break fi done # even when the control socket exists, it does not mean unit has finished initialisation # this curl call will get a reply once unit is fully launched /usr/bin/curl -s -X GET --unix-socket /unit/var/run/control.unit.sock http://localhost/ if /usr/bin/find "/docker-entrypoint.d/" -mindepth 1 -print -quit 2>/dev/null | /bin/grep -q .; then echo "$0: /docker-entrypoint.d/ is not empty, applying initial configuration..." echo "$0: Looking for certificate bundles in /docker-entrypoint.d/..." for f in $(/usr/bin/find /docker-entrypoint.d/ -type f -name "*.pem"); do echo "$0: Uploading certificates bundle: $f" curl_put $f "certificates/$(basename $f .pem)" done echo "$0: Looking for JavaScript modules in /docker-entrypoint.d/..." for f in $(/usr/bin/find /docker-entrypoint.d/ -type f -name "*.js"); do echo "$0: Uploading JavaScript module: $f" curl_put $f "js_modules/$(basename $f .js)" done echo "$0: Looking for configuration snippets in /docker-entrypoint.d/..." for f in $(/usr/bin/find /docker-entrypoint.d/ -type f -name "*.json"); do echo "$0: Applying configuration $f"; curl_put $f "config" done echo "$0: Looking for shell scripts in /docker-entrypoint.d/..." for f in $(/usr/bin/find /docker-entrypoint.d/ -type f -name "*.sh"); do echo "$0: Launching $f"; "$f" done # warn on filetypes we don't know what to do with for f in $(/usr/bin/find /docker-entrypoint.d/ -type f -not -name "*.sh" -not -name "*.json" -not -name "*.pem" -not -name "*.js"); do echo "$0: Ignoring $f"; done else echo "$0: /docker-entrypoint.d/ is empty, creating 'welcome' configuration..." curl_put /usr/share/unit/welcome/welcome.json "config" fi echo "$0: Stopping Unit daemon after initial configuration..." kill -TERM $(/bin/cat /unit/var/run/unit.pid) for i in $(/usr/bin/seq $WAITLOOPS); do if [ -S /unit/var/run/control.unit.sock ]; then echo "$0: Waiting for control socket to be removed..." /bin/sleep $SLEEPSEC else break fi done if [ -S /unit/var/run/control.unit.sock ]; then kill -KILL $(/bin/cat /unit/var/run/unit.pid) rm -f /unit/var/run/control.unit.sock fi echo echo "$0: Unit initial configuration complete; ready for start up..." echo fi fi exec "$@" The Dockerfile copies all files ending in .json to a directory named /docker-entrypoint.d. The docker-entrypoint.sh script checks the /docker-entrypoint.d directory for configuration files that are used to configure NGINX Unit. The following config.json file was created in the same directory as the Dockerfile and docker-entrypoint.sh script. { "listeners": { "*:8000": { "pass": "applications/wasm" } }, "applications": { "wasm": { "type": "wasm-wasi-component", "component": "/app/hello_wasi_http.wasm" } } } This config file instructs NGINX unit to listen on port 8000 and pass any client request to the Wasm application component located at /app/hello_wasi_http.wasm. The last step before building the container is to copy the Wasm application component into the same directory as the Dockerfile, docker-entrypoint.sh, and config.json files. With these four files in the same directory, the docker build command is used to create a container: docker build --no-cache --platform linux/amd64 -t hello-wasi-xc:1.0 . Here I use the --platform option to specify that the container will run on a linux/amd64 platform. I also use -t to give the image a name. The portion of the name after the ":" can be used for versioning. For example, if I added a new feature to my application, I could create a new image with -t hello-wasi-xc:1.1 to represent version 1.1 of the hello-wasi-xc application. Once the image is built, it can be uploaded to the container registry of your choice. I have access to an Azure Container Registry, so that is what I chose to use. Distributed Cloud supports pulling images from both public and private container registries. To push the image to my registry, I had to first tag the image with the registry location ({{registry_name}} should be replaced with the name of your registry): docker tag hello-wasi-xc:1.0 {{registry_name}}/examples/hello-wasi-xc:1.0 Next, I logged into my Azure registry: az login az acr login --name {{registry_name}} I then pushed the image to the registry: docker push {{registry_name}}/examples/hello-wasi-xc:1.0 F5 Distributed Cloud Virtual Kubernetes F5 Distributed Cloud Services support a Kubernetes compatible API for centralized orchestration of applications across a fleet of sites (customer sites or F5 Distributed Cloud Regional Edges). Distributed Cloud utilizes a distributed control plane to manage scheduling and scaling of applications across multiple sites. Kubernetes Objects Virtual Kubernetes supports the following Kubernetes objects: Deployments, StatefulSets, Jobs, CronJob, DaemonSet, Service, ConfigMap, Secrets, PersistentVolumeClaim, ServiceAccount, Role, and Role Binding. For this article I will focus on the Deployments object. A deployment is commonly used for stateless applications like web-servers, front-end web-ui, etc. A deployment works well for this use case because the NGINX Unit container is serving up a stateless web application. Workload Object A workload within Distributed Cloud is used to configure and deploy components of an application in Virtual Kubernetes. Workload encapsulates all the operational characteristics of Kubernetes workload, storage, and network objects (deployments, statefulsets, jobs, persistent volume claims, configmaps, secrets, and services) configuration, as well as configuration related to where the workload is deployed and how it is advertised using L7 or L4 load balancers. Within Distributed Cloud there are four types of workloads: Simple Service, Service, Stateful Service, and Job. Namespaces In Distributed Cloud, tenant configuration objects are grouped under namespaces. Namespaces can be thought of as administrative domains. Within each Namespace, a user can create an object called vK8s and use that for application management. Each Namespace can have a maximum of one vK8s object. Distributed Cloud vK8s Configuration vK8s is configured under the Distributed Apps tile within the Distributed Cloud console. After clicking on that tile, I next created a new Virtual K8s. By clicking Add Virtual K8s from the Virtual K8s menu under Applications. This brings up the Virtual K8s configuration form. I provided a name for my vK8s site and click Save and Exit. This initiates the vK8s build. Deploy the Application Once the vK8s cluster is ready, I click on the name of my vk8s to configure the deployment of my application. For this demo, I am going to deploy my application via Workloads. I click on Workloads and then Add VK8s Workload. This brings up the Workload configuration form. I provide my Workload a name, select Service for Workload type and click Configure. Using Service allows me to have more granular controls on how my application will be advertised. Next, I click Add Item under containers to specify to Distribtued Cloud where it can pull my container image from. In the resulting form, I supply a name for my container, supply the public registry FQDN along with the container name and version and click Apply. Next I configure where to advertise the application. I want to manually configure my HTTP LB, so I chose to advertise in Cluster. Manually configuring the LB allows me to configure additional options such as Web Application Firewall (WAF). Within the Advertise in Cluster form, I specify port 8000, because that is the port my container is listening on. Click Apply and then click Save and Exit. Create an HTTP Load Balancer to Advertise the Application The next step is to create an HTTP Load Balancer to expose the application to the Internet. This can be done by clicking Manage, selecting Load Balancers, and clicking on HTTP Load Balancers. Next, click on Add HTTP Load Balancer to create a new load balancer. I gave the LB a Name, Domain, Load Balancer Type, and Checked the box to Automatically Manage DNS Records. Then I configured the Origin Pool by clicking Add Item in the Origin Pool section and then clicking Add Item under the Origin Pool. On the resulting form, I created a name for my Origin Pool and then clicked Add Item under Origin Servers to add an origin Server. In the Origin Server form, I selected K8s Service Name of Origin Server on given Sites, Service Name, and provided my vK8s workload name along with my namespace. I also selected Virtual Site and created a Virtual Site for the REs I wanted to use to access my application and clicked Continue. Back on the Origin Server form, I selected vK8s Networks on Site from the Select Network on the site drop down and then clicked Apply. On the Origin Pool form, I specified port 8000 for my origin server Port and then clicked Continue. I then clicked Apply to return to the LB configuration form. I added a WAF policy to protect my application and then clicked Save and Exit. At this point my application is now accessible on the domain name I specified in the HTTP Load Balancer config (http://aconley-wasm.amer-ent.f5demos.com/). Conclusion This is a basic demo of using NGINX Unit and Distributed Cloud Virtual Kubernetes to deploy a server-side WASM application. The principals in this demo could be expanded to build more complex applications.
Declaring disaster when using a BIG-IP DNS Controller driven disaster recovery data center transition
I'm re-asking this question, because we still don't have a solution, and I'm hoping that potential answers may have been missed on the first ask - and it just "feels" to me like someone who is more experienced with Big-IP DNS Controllers (GTMs) would know a way to do this. The original posting is here. In our organization, we're planning on using our GTMs to control disaster recovery - that is, we have a backup data center, which is cold, but we want to spin it up in case of a disaster, and only after it's ready, "flip a switch" to declare a disaster, and at that point have all GTM-managed names start returning the alternate data center's IPs. Up until that switch-flip, all wide IPs should continue to return their original data center IP values (or, potentially, return nothing at all, failing to resolve). My question is, what are the general recommendations for implementation of this manual "switch"? What have people done in terms of creating a construct that is manually controlled, in order to control the GTM's logic for declaring disaster, and affecting a broad array of wide IPs, etc. managed by the GTMs? (sorry, "BIG-IP DNS") It was suggested to simply use global availability - but that doesn't quite fit, because we don't want the alternate data center's IPs returned by the wide IPs until after this "switch" is thrown. Is there some capability at the data center construct level to effect this behavior? Via distributed applications? I'm hoping for some built-in configurable capability, without having to do significant iRule coding or iControl scripting. At that point, it may simply be easier to manage it all via short TTLs and manual imports to our primary DNS servers (which are not GTMs). Thank you for any help!
