Automating F5 ADSP — Part 1: F5 XC and BIG-IP for Delivery and Security
What this use case demonstrates This use case covers three of the four ADSP areas: Delivery, Security, and Deployment. Delivery — F5 Distributed Cloud (XC) load balancer at the edge, F5 BIG-IP LTM handling traffic management inside the VPC. Security — XC WAF at the edge, BIG-IP Advanced WAF (AWAF) applying in-path policy before traffic reaches the application servers. Deployment — XC consumed as SaaS, BIG-IP deployed as a Virtual Edition in GCP. Same article, two deployment models, both provisioned from code. You get two layers of delivery and two layers of WAF, across a SaaS edge and a self-managed VE. The whole stack, VPC through XC load balancer, comes up from a single git push. Architecture What gets deployed: A GCP VPC with management, external, internal, and application subnets BIG-IP with AWAF in a single-NIC configuration OWASP Juice Shop and crAPI as target applications F5 Distributed Cloud HTTP load balancer, origin pool, and WAF policy pointing at the BIG-IP The vulnerabilities in the apps are deliberate. They let you exercise the WAF stack against real attack signatures and API abuse patterns. Without them, you only know the controls deployed, not that they work. DevSecOps in practice The lead-in covers the approach. For UC1, that means: Terraform handles infrastructure, BIG-IP configuration, and F5 Distributed Cloud objects. No click-ops. State lives in a GCS bucket the workflow creates on the first run, with a separate state file per module. The same bucket carries the AS3 declaration BIG-IP pulls on boot, so the runner never needs network access to BIG-IP. GitHub Actions runs the pipeline. Branch names trigger deployments, so git history shows what was meant to happen. GCP Workload Identity Federation replaces static service account keys. The F5 XC API certificate lives in GitHub Actions secrets, not the repo. The pipeline Pushing to a branch runs the workflow. There is no manual terraform to apply. Action Branch Validate, plan, and apply deploy-adsp-uc1 Validate only (no apply) test-adsp-uc1 Destroy all resources destroy-adsp-uc1 This keeps intent visible in git, makes destroy as easy as deploying, and gives reviewers a real PR to look at when something changes. What's in the repo f5devcentral/F5-ADSP-Automation: Directory Purpose infra/gcp/ VPC, subnets, firewall rules compute/gcp/ Juice Shop and crAPI f5/ BIG-IP base config and AWAF policy config/uc1/gcp/env.json GCP project, region, prefix config/uc1/xc/env.json F5 Distributed Cloud config .github/workflows/ CI/CD workflows Demo Try it Prerequisites, secrets, and troubleshooting are in the Use Case Deployment Guide. Contribute Issues and PRs welcome at f5devcentral/F5-ADSP-Automation/Issues. Resources: F5 Application Delivery and Security Platform GitHub Repo and Automation Guide ADSP Architecture Article Series: Automating F5 ADSP Deployments (Intro) Automating F5 ADSP Deployments (Part 1 - F5 XC WAF and BIG-IP Adv. WAF) Automating F5 ADSP Deployments (Part 2 - F5 XC WAF and NGINX App Protect) Automating F5 ADSP Deployments (Part 3 - F5 XC API Protection and NGINX Ingress) Automating F5 ADSP Deployments (Part 4 - F5 XC BOT Defense and BIG-IP AdvWAF) Automating F5 ADSP Deployments (Part 5 - F5 XC, BIG-IP APM, CIS, and NGINX Ingress) Minimizing Security Complexity: Managing Distributed WAF Policies
Automating F5 Application Delivery and Security Platform Deployments
The F5 ADSP Architecture Automation Project The F5 Application Delivery and Security Platform (ADSP) reduces the complexity of modern applications by integrating operations, traffic management, performance optimization, and security controls into a single platform with multiple deployment options. This series outlines practical steps anyone can take to put these ideas into practice using the F5 ADSP Architectures GitHub repo and related projects. Each article in the series highlights a different deployment example. The examples can be run locally or integrated into CI/CD pipelines following DevSecOps practices. The repositories are community-supported and intended as reference code for demos, workshops, or as a stepping stone for your own F5 ADSP deployments. If you find any bugs or have any enhancement requests, open an issue, or better yet, contribute. The F5 Application Delivery and Security Platform (F5 ADSP) F5 ADSP addresses four core areas: how you operate day-to-day, how you deploy at scale, how you secure against evolving threats, and how you deliver reliably across environments. Each comes with its own challenges, but together they define the foundation for keeping systems fast, stable, and safe. xOps — day-to-day operations, observability, and lifecycle management Deployment — getting workloads where they need to go, at the scale they need Delivery — traffic management across hybrid and multi-cloud environments Security — protecting applications and APIs from current threats Each architecture deployment example in this series is designed to cover at least three of the four core areas. This ensures the examples demonstrate how multiple components of the platform work together in practice, rather than showcasing any single feature in isolation. DevSecOps: Integrating security into the software delivery lifecycle is a necessary part of building and maintaining secure applications. This project incorporates DevSecOps practices by using supported APIs and tooling, with each use case including a GitHub repository containing IaC code, CI/CD integration examples, and telemetry options. In practice across the series, that means Terraform for infrastructure and F5 configuration, GitHub Actions as the pipeline runner, federated cloud identity instead of long lived keys, secrets handled through the platform rather than committed, and vulnerable target applications so the security controls can be exercised end-to-end. Contribute The repos are community-supported. Open an issue, send a PR, or port a use case to another cloud. Resources: F5 Application Delivery and Security Platform GitHub Repo and Automation Guide ADSP Architecture Article Series: Automating F5 ADSP Deployments (Intro) Automating F5 ADSP Deployments (Part 1 - F5 XC WAF and BIG-IP Adv. WAF) Automating F5 ADSP Deployments (Part 2 - F5 XC WAF and NGINX App Protect) Automating F5 ADSP Deployments (Part 3 - F5 XC API Protection and NGINX Ingress) Automating F5 ADSP Deployments (Part 4 - F5 XC BOT Defense and BIG-IP AdvWAF) Automating F5 ADSP Deployments (Part 5 - F5 XC, BIG-IP APM, CIS, and NGINX Ingress) Minimizing Security Complexity: Managing Distributed WAF PoliciesAutomating F5 ADSP — Part 2: F5 XC and NGINX for Delivery and Security
What this use case demonstrates This use case deploys NGINX Ingress Controller (NIC) running NGINX Plus with NGINX App Protect V5 (NAP V5) as the in-cluster data plane on GKE. WAF runs on two layers: NAP V5 enforcing inside the cluster, F5 Distributed Cloud (XC) enforcing at the edge. XC also provides API protection driven by an OpenAPI spec. It covers all four ADSP areas: Delivery, Security, Deployment, and xOps. Delivery: F5 Distributed Cloud HTTPS load balancer at the edge, NGINX Ingress Controller handling in-cluster delivery through the NIC VirtualServer CRD. Security: Two layers of WAF. NAP V5 runs as NIC sidecars (waf-enforcer and waf-config-mgr) and enforces the WAF policy attached to the VirtualServer. XC WAF runs at the edge in blocking mode. XC API protection is driven by an OpenAPI spec. Deployment: XC consumed as SaaS, GKE Standard with private nodes, NIC and NAP installed via OCI Helm chart, the application installed via a separate OCI Helm chart. xOps: NAP policy lives in config/uc2/nap/policy.json. The workflow compiles it with the NAP waf-compiler container, uploads the compiled bundle to GCS, and NIC mounts the bundle read-only via the GCS Fuse CSI driver. The waf-config-mgr sidecar watches the mount and pushes updates to the waf-enforcer. Change the policy, push, and NAP follows. Architecture What gets deployed: A GCP VPC with a dedicated k8s subnet (with secondary ranges for pods and services), management subnet, and NAT for private nodes A GKE Standard zonal cluster with private nodes and a control plane locked down by authorized networks NGINX Ingress Controller running NGINX Plus, with NAP V5 enforcer and config-mgr sidecars Comfy Capybara deployed via an OCI Helm chart, exposed through a NIC VirtualServer that references the waf-policy CRD in the nginx-ingress namespace An F5 Distributed Cloud HTTP load balancer with WAF and API protection. The origin pool is resolved from the NIC LoadBalancer IP via Terraform remote state. The VirtualServer attaches waf-policy both server-wide and on the /api route by default, so the policy enforces everywhere as a baseline. DevSecOps in practice for UC2 The lead-in covers the approach. For UC2, that means: Terraform handles infrastructure, the GKE cluster, NIC and NAP, the application Helm release, and all F5 Distributed Cloud objects. No click-ops. State lives in a GCS bucket the workflow creates on the first run, with a separate state file per module. The same bucket carries the compiled NAP policy bundle that NIC mounts via the GCS Fuse CSI driver. The XC origin pool reads the NIC LoadBalancer IP from state/uc2/nic, so no IP is pasted between configs. GitHub Actions runs the pipeline. Branch names trigger deployments, so git history shows what was meant to happen. GCP Workload Identity Federation replaces static service account keys for the runner. NIC pods also use Workload Identity to impersonate the runtime service account when mounting the NAP bundle from GCS. The XC API certificate, NGINX Plus JWT, and NGINX registry credentials live in GitHub Actions secrets, not the repo. The OpenAPI spec at config/uc2/app/oas/openapi.json is base64-encoded by the workflow and referenced inline by the XC API definition. Change the spec, push, and API protection follows. The pipeline Pushing to a branch runs the workflow. There is no manual terraform apply or helm install. Action Branch Validate, plan, and apply deploy-adsp-uc2 Validate only (no apply) test-adsp-uc2 Destroy all resources destroy-adsp-uc2 Modules deploy sequentially: state bucket - infra - GKE - compile NAP policy - NIC and NAP - app - XC. Destroy runs in reverse. What's in the repo f5devcentral/F5-ADSP-Automation: Directory Purpose infra/gcp/ VPC, subnets with pod and service secondary ranges, NAT, firewall k8s/gcp/ GKE Standard cluster and node pool f5/nic/gcp/ NGINX Ingress Controller and NAP V5 Helm release f5/xc/ F5 Distributed Cloud HTTP LB, WAF, API definition (shared with other XC use cases) app/gcp/ Comfy Capybara Helm release and VirtualServer config/uc2/gcp/env.json GCP, GKE, and NIC config config/uc2/nap/policy.json NAP policy source, compiled in the workflow config/uc2/app/env.json Application chart and VirtualServer config config/uc2/app/oas/openapi.json OpenAPI spec the XC API definition is built from config/uc2/xc/env.json XC tenant, LoadBalancer, WAF and API feature flags .github/workflows/ CI/CD workflows Prerequisites, secrets, and troubleshooting are in the UC2 deployment guide. Demo Try it Fork f5devcentral/F5-ADSP-Automation, set the secrets and tfvars from the deployment guide, and push to deploy-adsp-uc2. Push to destroy-adsp-uc2 to tear it down. Contribute Issues and PRs welcome at f5devcentral/F5-ADSP-Automation.log local0. is not a debugging strategy!
But let's be honest...with iRules, it's pretty much all we've had. If you have ever debugged an iRule, you know the ritual. Former F5er Jibin_Han in an article once called the log command is "the crudest of debug facilities." He was not wrong. It is the printf of application delivery, and at high traffic the logging pipeline will happily truncate your output just to keep things spicy. So back in TMOS 13.1, he shipped something much better: ltm rule-profiler. It is a passive tracer baked right into TMM. (Did you know this? Most don't, including most F5ers I talk to!) You do not touch your script. You tell it which virtual server, events, and occurrences to watch, you flip it on, and it emits a timestamped execution trace of everything your iRule did. Events firing, rules matching, the Tcl VM grinding through bytecode, native commands getting dispatched, variables changing. The whole shebang. There is just one catch. Ok actually two. It is tmsh-only. No GUI, no REST endpoint, no nothing. You configure it by hand and you start and stop it by hand. The not so fun part, the output looks like this: 1780079189187194,RP_EVENT_ENTRY,/Common/testvip-http,CLIENT_ACCEPTED,22623,0x70373707000576,10.1.10.6,36086,0,10.1.10.50,80,0 1780079189187210,RP_RULE_ENTRY,/Common/testvip-http,/Common/myrule,22623,0x70373707000576,... 1780079189187225,RP_RULE_VM_ENTRY,... 1780079189187240,RP_CMD_BYTECODE,/Common/testvip-http,push1,... Now multiply that by a few hundred lines, sprinkle in microsecond timestamps you are expected to subtract in your head, and remember that every single custom iRules command is actually a round trip out of the Tcl VM and back into TMM. A command, inside a VM, inside the microkernel. We need to go deeper. Reading it raw is less "performance analysis" and more "staring at the green rain in The Matrix and pretending you can see the woman in the red dress." The data is genuinely great. It's just wearing a CSV trench coat and refusing to make eye contact. But that ends now. Let's talk about Rültracer. What Rültracer is Rültracer is an iApps LX extension that gives ltm rule-profiler the face it always deserved. When Jibin_Han released his 3-part article series (linked at the bottom of this article) introducing the rule-profiler, we had a couple interns who built an analysis engine called Campfire that used a perl-based flamegraph package to display the trace in context of the "lift" of each occurence within a trace. The challenge was you had to manually configure everything on BIG-IP, send the logs somewhere, collect those logs, then import them into where you built campfire to run. A lot of manual work to get to the good. But Rültracer? It runs on the BIG-IP. An on-box Node worker handles the unglamorous parts: it configures the profiler, sets up (and tears down) the log publisher, captures the trace stream into a per-session file, and serves that file to a browser app. The browser does all the parsing and visualization client side, so the box just ships raw CSV and gets out of the way. What you get: A sequence diagram and step-through debugger. The trace becomes a UML-style sequence diagram across six lifelines (Users, Event, Rule, Rule VM, Command VM, Command), with the TMM and Tcl VM sides color coded so you can actually see every handoff between the microkernel and the interpreter. Which matters, because that back-and-forth is exactly where iRule inefficiency likes to hide. Next to it sits a linked step-through with a timeline scrubber, variable state, and command replay. Click the diagram, the table follows. Click the table, the diagram follows. There's a Tclsh disassembler you can enable as well, and as long as you're willing to modify the parts of the iRules code under test. I wrote an article years ago on this disassembly functionality, it's worth the read to see what this functionality affords you. iRule source mapping. It pulls your actual iRule source and lights it up: which commands fired (with microseconds and counts), which branches never ran, and which lines were ambiguous multi-matches. Your code, annotated by what the trace really did. (this part is early stages, it needs work.) Flamegraphs, with diff. An interactive flamegraph where width equals inclusive time, so the widest frame is your prime suspect. Find the slow command without playing Where's Waldo. Captured a "before" and an "after"? The diff view paints frames red and blue by how their self-time shifted, so you can prove your optimization actually optimized. Cycles versus CPU. Rültracer takes the box's own ltm rule stats hardware cycle counters and turns them into honest performance tables: cycles to microseconds, percent of a CPU per request, and max requests per second before your iRule becomes the bottleneck. It even reconciles the authoritative cycle counts against the trace-derived numbers, and the gap between them is the profiler's own overhead. These are numbers you can bring to a capacity-planning meeting without getting laughed out of the room. This is based on Deb Allen's yesteryear work in excel for computing capacity with iRules and my update in doing this with python much later. Reports and exports. Export a capture as self-contained HTML, JSON, a Mermaid sequence diagram, or Brendan Gregg folded stacks. Share it, attach it to a ticket, or feed it to your own tooling. Multi-TMM aware. Captures that span multiple TMMs get partitioned by context id with a scope selector, because of course your traffic did not politely land on a single TMM. No (post RPM install) build steps, no cloud, no telemetry, no agent. Vanilla JS in the browser, a small ES5 worker on the box, fully self-contained. And it is a lab tool on purpose: tracing adds significant TMM overhead, so this is not something you run in production. Rültracer tears the profiler and publisher down for you on teardown, so you never accidentally leave the tap open. Installing it Rültracer installs over SSH. You build the RPM on your workstation, copy it to the BIG-IP, and run the installer on the box as root. The installer provisions the persistent data directory, installs the package through the iApps LX framework, runs the post-install step, and confirms the workers came up. Replace and with your BIG-IP's SSH host and port. If it answers on plain old port 22, drop the -P / -p flags. First time on a fresh box The installer script lives outside the RPM, so it rides along once: Copy the installer on the box (one time only) scp -O -P <port> build/install-onbox.sh root@<host>:/shared/images/ Build, ship, and install ./build/build-rpm.sh 0.7.1 0001 scp -O -P <port> build/dist/rultracer-0.7.1-0001.noarch.rpm root@<host>:/shared/images/ ssh -p <port> root@<host> /shared/images/install-onbox.sh 0.7.1-0001 Because the installer runs as root, it creates the session data directory owned by the restnoded worker user before the workers start (the worker is uid 198 and cannot create directories under /shared/ on its own). When it finishes, it prints your UI URL: https://BIG-IP-host/mgmt/shared/rultracer/ui/ Open that, and you are in. Follow-on updates The installer is already on the box, so the next rounds are just bump, build, ship, install like above. That is an in-place upgrade and it keeps your saved sessions. If you ever want a clean slate, pass --reinstall, but note it wipes session data, so hit the Sessions tab's "Download backup" button first if you care about what is in there. See it in action (The walkthrough covers a live capture end to end: pointing the profiler at a virtual server, driving a little traffic, then digging through the sequence diagram, flamegraph, and cycle stats on a real trace.) Conclusion This was a fun project to bring together something I've played with a lot since the v13.1 release but could never quite figure out a packaging solution to make it functional enough to rely on. You can find the code in the Rültracer repo on Github. Let me know in the comments if you take a look at this and submit any bugs ore feature requests as an issue out on Github.
Secure and Harden Forward Proxies in NGINX Plus
Most people know NGINX as a reverse proxy, sitting in front of your servers to handle incoming traffic. Forward proxy works in the opposite direction. It sits between your internal users (or applications) and the outside world, managing outbound connections. NGINX Plus R36 introduced this capability through support for the HTTP CONNECT method. Before this, organizations often needed separate tools for inbound and outbound traffic control. Now you can handle both with a single platform. This unification in a single platform reduces operational overhead, streamlines application delivery and management, and reduces the attack surface of your application infrastructure.API Discovery and Enforcement with API Security Local Edition
API Security Local Edition is a self-hosted platform that discovers APIs from BIG-IP traffic insights, builds and maintains an inventory with risk scoring, and pushes enforcement back to BIG-IP. This article covers the architecture, the data flows between components, and the operator workflow from discovery to enforcement.
NGINX Gateway Fabric - Data Plane Programmability with NGINX JavaScript
This post walks through a pattern for injecting NGINX JavaScript logic into NGINX Gateway Fabric using Kubernetes-native extension points to enable data plane programmability, with an F5 AI Guardrails integration as a worked example.Explicit write control for iRules subtables
Note to the reader...apparently what is old is new again. There are some threads here on DevCentral that have already solved for this, albeit in different ways. The few brought to my attention by MVP Kai_Wilke are included in the list below for your benefit to read through. That said, the journey of discovery here in this article is worth your time to understand the nuances of how data is passed in a multi-TMM system. Dealing with iRule $variables for HTTP2 workload while HTTP MRF Router is enabled | DevCentral https://github.com/KaiWilke/F5-iRule-RADIUS-Server-Stack SPDY/HTTP2 Profile Impact on Variable Use | DevCentral The TL;DR TMM subtables on BIG-IP are partitioned across TMMs by hashing the subtable name. Writing to a subtable from a non-owner TMM is roughly 1000x slower than writing from the owner...single-digit clock clicks vs. tens of thousands. If you want fast per-TMM local storage, you cannot pick the subtable name yourself; you have to *discover* a locally-owned name by timing trial writes. Deterministic naming schemes do not work, even when they look obviously correct. The Problem A colleague had an iRule that maintained per-connection state across many CLIENT_DATA events. The natural data structure was a TMM session subtable. His quick experimenting showed the writes were slow enough to push the system CPU under modest load and needed to understand why before scaling further. There's an example proc library from Nat_Thirasuttakorn "LOCALDB" that uses a clever timing trick: it generates a random subtable name, times a probe write, and only keeps the name if the write completes under some threshold (50 clock clicks in the original). The implication was that most random names produce slow writes and only a few are fast. I read the code, figured I understood it, and rewrote it "cleanly" using deterministic per-TMM names: `localdb_tmm_0`, `localdb_tmm_1`, `localdb_tmm_2`, ... one per TMM, no probing required. Each TMM would write only to its own name. Done, right? Wrong. The diagram above is the mental model the rest of this post leans on. Two independent hashes are happening: the DAG hashes the inbound 4-tuple to choose which TMM accepts the connection, and TMOS separately hashes the subtable name to choose which TMM *owns* the storage for that name. A write succeeds only when both hashes agree; when the TMM that received the connection is also the owner of the subtable being written to. When they disagree, the write costs roughly 7000x more. The Investigation The deterministic version "worked" — writes succeeded, distribution looked plausible, throughput was decent. Then I added timing instrumentation per TMM and looked at the percentiles: TMM samples min avg max 0 74 121 64855.6 229089 1 34 136 71536.3 236204 2 38 121 88516.9 293259 3 62 3 13.3 25 TMM 3 was writing in 3-25 clicks. Every other TMM was averaging tens of thousands, which is a 5,000-7,000x gap! Something was very wrong. The diagnosis came from a `/probe` endpoint I'd added for unrelated reasons: hit the same subtable name from many connections, time each write, count which TMM responds fast. Probing each of the four "deterministic" names produced: localdb_tmm_0 → owner is TMM 2 localdb_tmm_1 → owner is TMM 2 localdb_tmm_2 → owner is TMM 3 localdb_tmm_3 → owner is TMM 3 Visualizing the result for one of those probes makes the signal unambiguous: Two of the four names hashed to TMM 2, the other two hashed to TMM 3. TMMs 0 and 1 didn't own any of the subtables I'd "assigned" to them. This is the key insight: **the subtable name `localdb_tmm_3` doesn't get owned by TMM 3 just because its name ends in 3.** TMOS hashes the whole name string and assigns ownership based on that hash. The hash is opaque, and it's stable, but it has no relationship to the content of the name. My deterministic scheme was generating four unique names, which guaranteed no key collisions across TMMs — but it didn't guarantee, and couldn't guarantee, that name N landed on TMM N. Why The Original Trick Was Right Going back to the LOCALDB proc library pattern from DevCentral: while { $try < $maxtry } { set name [expr rand()] set before [clock clicks] table set -subtable $name test_$name $name 5 set after [clock clicks] set diff [expr {$after - $before}] if { $diff < $maxdiff } { break } incr try } Generate a random name. Probe it. If it's fast, keep it; if not, throw it away and try another. Each TMM independently does this, and on average needs ~N tries on an N-TMM system to find a name it owns. The probe is the *only* reliable way to know. The randomness is load-bearing. The timing measurement is load-bearing. Neither is decorative. My "elegant" rewrite removed both and produced a system that looked fine but was burning 99% of its potential throughput shipping writes between TMMs. How to Verify A timing histogram per TMM is the diagnostic. The test workflow: Add a `/probe?name=X` endpoint that times a single `table set` against an arbitrary subtable name and reports clicks + the responding TMM Hit it many times from a multi-threaded client Aggregate per-TMM: hits, OWNER count (writes under threshold), NON_OWNER count, min/avg/max clicks The owner of name X will show up as ~all-OWNER with consistently low clicks; everyone else shows ~all-NON_OWNER with high clicks A handful of stray "OWNER" tags on non-owners is just noisy variance in `clock clicks` measurement. The real signal is overwhelming: 50+ OWNER tags vs 0-3 OWNER tags, and average clicks differing by 1000-10000x. Lessons About TMM Subtables A few things worth internalizing if you work with these: Names are global; storage is partitioned Two TMMs writing the same name reach the same logical subtable, but only the owner stores it locally. Non-owners pay an inter-TMM coordination tax on every operation. This is fundamentally a sharding scheme where the shard key is the subtable name and the shard map is hidden from you. Construction can't replace discovery Anywhere a system uses an opaque hash to assign ownership of named resources, you cannot construct a locally-owned name, you can only find one by trying. This pattern shows up well beyond TMOS: Cassandra token ranges, Redis Cluster slots, Kafka partition assignments, consistent-hashing rings in general. Discovery beats construction whenever the mapping function is hidden. O(n) reads in hot paths kill throughput I had a `count` proc that called `table keys -subtable X` and ran `llength` on the result. With per-TMM subtables of ~25k entries, that's 25k strings to enumerate per request. Throughput decayed from 3300/s to 600/s over a 40k-record run, a perfect 1/n curve. Maintaining the count incrementally in a `static::` variable made it O(1) and throughput stayed flat. The fix is obvious in hindsight; the bug is invisible without per-second throughput measurement. Static variables are per-TMM This is great when you want it (per-TMM owned-subtable name, per-TMM counters) and confusing when you don't (you can't share state across TMMs through statics alone). The variables are also persistent across rule reloads in some versions, which means a rule update that adds a new static can leave you with TMMs running the new code but missing the new state. Defensive existence checks at the top of every proc are worthwhile. Sampling debug logs is mandatory at scale Logging every write to `/var/log/ltm` for a million-record load is 1M log lines, hundreds of MB, and enough log I/O to tank throughput on its own. Sample 1-in-N (where N grows with load size), and gate calling-rule logs on the same sample point so the log narrative stays coherent. A `should_log` helper proc shared between the library and its callers keeps this clean. Test harnesses should reset, not reload I initially "reset" between runs by reloading the iRule. `RULE_INIT` re-ran and statics reset, but the *subtable contents* persisted in TMM session memory because they're indexed by name, not by rule. Each rule reload picked a new random name and orphaned the old subtable's entries. Over many runs, memory accumulated. A `/reset` endpoint that walks `table keys` and deletes them is the right abstraction. What "Done" Looked Like After the fix, a 100k-record run on a 4-TMM system: TMM samples min avg max 0 98 3 17.4 71 1 101 4 18.9 88 2 99 3 16.8 77 3 102 4 19.1 91 Throughput stayed flat at ~3000/s for the entire run. Every TMM in the same low-clicks range. No `SLOW` tags in the sampled logs. The before-and-after chart (log scale) makes the impact unmistakable: TMM 3 is interesting on its own. Under the broken design it was already fast (averaging 13.3 clicks) because the deterministic names happened to hash to it, meaning every other TMM was ferrying its writes over to TMM 3. Under the fix, TMM 3 stops being a single hot point and instead does roughly the same work as everyone else, on its own subtable. The fact that TMM 3's "broken" bar isn't dramatically taller is what makes this kind of bug survive a smoke test: writes were succeeding, throughput looked plausible, *one* TMM was even fast. The percentile breakdown is what gave it away. The Validated Test Session Here is the actual end-to-end verification run, command by command, on a 4-TMM lab BIG-IP. This is the workflow that I ended up codifying in the project's `USAGE.md` — it both validates that the fix works and demonstrates each tool's role. Step 1: Verify Every TMM Picked a Unique Subtable After deploying the LOCALDB rule and the calling rule, hit `/whoami` enough times that fresh TCP connections fan out across all TMMs: $ for i in $(seq 1 30); do curl -s http://10.0.2.49/whoami; done | sort -u tmm 0 subtable localdb_tmm_0_865802 total_tmms 4 writes 0 entries 0 tmm 1 subtable localdb_tmm_1_922743 total_tmms 4 writes 0 entries 0 tmm 2 subtable localdb_tmm_2_5946 total_tmms 4 writes 0 entries 0 tmm 3 subtable localdb_tmm_3_441563 total_tmms 4 writes 0 entries 0 Four things to read out of this: Four unique TMMs (0, 1, 2, 3) responded meaning full coverage. With `Connection: close` from curl, each request gets a fresh ephemeral source port and the BIG-IP's DAG re-hashes; 30 requests against 4 TMMs is essentially guaranteed to hit all of them. Four unique subtable names, each with the responding TMM number as a prefix and a random suffix. The TMM-number prefix is just a label for human readability. The random suffix is what `init_table` actually iterates on during timing-probe discovery, throwing away names that hash to other TMMs and keeping the first one whose write completes under the threshold. `total_tmms=4` is consistent on every row. `TMM::cmp_count` is reporting the cluster size correctly. writes=0 entries=0` everywhere. Clean baseline before any load. Step 2: Reset to a Clean Baseline $ python tbl-loader.py reset --host 10.0.2.49 --port 80 Discovering TMM count from 10.0.2.49:80/info ... BIG-IP reports 4 TMMs. Sending 200 /reset requests with 32 workers... Reset summary: TMM hits first_deleted total_deleted ------------------------------------------ 0 50 0 0 1 47 0 0 2 55 0 0 3 48 0 0 All 4 TMMs cleared. Total entries removed (first-hit): 0 200 reset requests, 50 / 47 / 55 / 48 distribution across the four TMMs. That's essentially perfect uniform. Expected mean is 50, observed range is 47-55, which is well within the natural variance of a fair hash. Worth confirming because the same DAG is what'll spread the load run; uneven reset distribution would predict uneven load distribution, which complicates the analysis. `first_deleted=0` everywhere because the previous step's `whoami` had already shown empty subtables. After a load run, this column tells you exactly how many entries each TMM was holding. Step 3: Run the Load $ python tbl-loader.py load --host 10.0.2.49 --port 80 --count 100000 --workers 64 ... completed=100,000/100,000 (100.0%) rate=4376/s coverage=4/4 missing=[] errors=0 Done. completed=100,000 errors=0 elapsed=22.9s rate=4375/s Final distribution: tmm 0: 25,198 writes (25.20%) tmm 1: 24,782 writes (24.78%) tmm 2: 24,914 writes (24.91%) tmm 3: 25,106 writes (25.11%) Three numbers worth lingering on: Sustained 4,375/s throughput, completely flat Earlier in the project, before the O(1) `count` fix, the equivalent run started at 3,300/s and decayed to 600/s by the 40k-record mark, a perfect 1/n curve from the hidden `table keys` + `llength` cost in the calling rule. With `static::LOCALDB_entries` maintained incrementally, the per-write work is genuinely constant and throughput stays where it starts. Distribution within ±0.25% of perfect uniform 25.20% / 24.78% / 24.91% / 25.11% is what fair hashing produces over 100k samples. The DAG is doing its job; nothing is being funneled through one TMM the way the broken-locality version was. Zero errors over 100k fresh TCP connections No TIME_WAIT exhaustion on the client (the ephemeral port range is wide enough), no rate limiting on the BIG-IP, no socket timeouts. Suggests the workload is well within both ends' capacity. The 22.9 second elapsed time works out to ~5 microseconds per write end-to-end, including the full TCP setup/teardown for each request. The actual `table set` is in the tens of clock clicks (single-digit microseconds), so HTTP and TCP overhead dominate, which is the right answer when the iRule work itself is fast and local. Step 4: Verify Per-TMM Locality from the Logs The throughput and distribution numbers tell us writes are happening evenly, but they don't directly prove each write is *local*. For that, pull the sampled timing lines from the BIG-IP's log and run them through the analyzer. Filter to the test window so earlier (broken) runs don't pollute the stats: $ ssh [email protected] "grep '^May 6 16' /var/log/ltm | grep 'sampled'" \ | python3 timing_stats.py Sample rate: 1/1000 Locality threshold: 100 clicks TMM n FAST SLOW min p50 avg p95 p99 max ------------------------------------------------------------------------------ 0 25 25 0 3 5 5.5 10 11 11 1 24 24 0 3 5 6.1 11 18 18 2 24 24 0 2 6 6.1 10 11 11 3 25 25 0 2 6 6.5 12 13 13 ------------------------------------------------------------------------------ Total: 98 samples across 4 TMMs FAST_LOCAL=98 SLOW=0 OK: all TMMs have average write timing below 100 clicks. Per-TMM locality is working. This is the centerpiece of the validation. Reading it line by line: Sample counts 25 / 24 / 24 / 25 samples per TMM matches the 25.20% / 24.78% / 24.91% / 25.11% write distribution from the load output, which is what you'd expect if the BIG-IP is logging 1-in-1000 of all writes uniformly. Timing Single-digit minimums (2-3 clicks). Averages of 5.5-6.5 clicks. p99s of 11-18. Max of 18 across all 98 samples. Compare to the broken run earlier in the project (shown at the top of the article in the investigation section), on the same hardware with the same workload but the wrong `init_table`. That's a **10,000x improvement on three of the four TMMs** between the two runs. The only thing that changed was `init_table` switching from deterministic naming to timing-probe discovery. Tag tally 98 FAST_LOCAL, 0 SLOW. Not a single sampled write missed the locality threshold. The 100-click threshold has plenty of headroom, the actual max was 18, an order of magnitude below. Verdict The script's automated check confirms locality is working. This is the line you'd grep for in CI if you wanted regression coverage. Step 5: Spot-Check Ownership of a Discovered Name The timing report proves writes were fast, but it doesn't prove that the *names* each TMM picked are actually owned by those TMMs (only that their writes were fast for whatever reason). To close that gap, take one of the names from `whoami` and probe it directly: $ python tbl-loader.py probe --host 10.0.2.49 --port 80 --name localdb_tmm_2_5946 --requests 200 ... Results for subtable 'localdb_tmm_2_5946': TMM hits OWNER NON_OWNER min_clicks avg_clicks max_clicks ---------------------------------------------------------------- 0 55 0 55 286 5139.9 19814 1 70 0 70 127 12475.3 52544 2 8 8 0 3 8.6 20 3 67 0 67 238 7126.6 51939 Likely owner of subtable 'localdb_tmm_2_5946': TMM 2 (avg 8.6 clicks, tagged OWNER 8 times) This is unambiguous: TMM 2 wrote in 3-20 clicks, average 8.6 Consistent with the 6.1 average from `timing_stats.py` during the load. Small differences, both well under threshold, both unambiguously local. TMMs 0, 1, 3 took 127-52,544 clicks, averages 5,139 / 12,475 / 7,126 Roughly 600x to 1,500x slower than TMM 2 on the same operation. They're paying the inter-TMM coordination tax because the subtable is owned by TMM 2. Zero stray OWNER tags on non-owning TMMs Earlier probe runs against fresh subtables sometimes had 1-3 stray OWNER tags from non-owners due to `clock clicks` jitter on small subtables. With this subtable now containing ~25k entries, the non-owner penalty is large enough (mins of 127-286 clicks) that no stray write made it under the 100-click threshold. The bigger the subtable, the cleaner the signal. TMM 2 only got 8 hits That's just sampling variance. The DAG hashed inbound connections 55 / 70 / 8 / 67, which over 200 requests is a normal-looking spread. With 1000 requests you'd see ~250 hits per TMM. The 8 hits TMM 2 did get were unanimous on OWNER, which is what matters. A run against any of the other discovered names (`localdb_tmm_0_865802`, `localdb_tmm_1_922743`, `localdb_tmm_3_441563`) produces the same shape of result with the corresponding TMM as owner. What This Validates Step 1 proves every TMM ran `init_table` and picked a unique name. Step 2 proves clean baseline and even DAG distribution. Step 3 proves throughput is sustained and writes spread evenly across TMMs at scale. Step 4 proves every write was fast at the time it happened. Step 5 proves the names each TMM picked are genuinely owned by those TMMs. Together they're a complete proof of the design: the timing-probe discovery in `init_table` correctly identifies a locally-owned subtable name on each TMM, and operations against those names cost ~10 clock clicks instead of ~70,000. The cost gap is the entire reason the per-TMM-subtable pattern exists, and it's now empirically demonstrated end-to-end. This validation run took maybe three minutes of wall time. It's the kind of verification I should have been running before believing the original "deterministic naming" rewrite worked, not after watching it fail under load. Pushing Throughput: Per-Write to Bulk-POST The validated workflow above writes one key per HTTP request. That's the right shape for testing locality (each write is a clean, isolated trial), but it makes TCP connection setup the dominant cost. At ~4,375 writes per second on a 4-TMM box, the iRule is spending most of its time accepting connections, parsing headers, and tearing down sockets, not writing to subtables. The natural next step is to batch many writes into a single HTTP request. A separate `/bulk_load` endpoint accepts a POST body of newline-separated keys (UUIDs in our test case), collects the body via `HTTP::collect`, and walks the lines in a tight loop calling `LOCALDB::set_unique` on each. One TCP connection now writes 15,625 keys instead of one. Per-batch timing comes back in the response so the loader can aggregate it client-side. The throughput result is striking: Same hardware, same iRule logic, same per-TMM locality — the 30× gap is purely TCP setup cost saved. The per-write timing inside the iRule barely changed (3-6 clicks per `LOCALDB::set_unique` either way), but the request-level overhead collapsed because we stopped paying it 1M times. A few things worth noting about this bulk path that aren't obvious: Locality holds inside the loop A `/bulk_load` request that lands on TMM 2 will do all 15,625 of its writes against TMM 2's local subtable. There's no opportunity for a single batch to "leak" writes to other TMMs, because the connection is pinned to one TMM by DAG and the subtable name is fixed by `static::LOCALDB_name`. So the locality verdict from the per-write test carries over without needing re-verification and the loader's per-batch `clicks_per_write` measurement confirms it stays in the 3-6 click range. DAG fan-out still distributes work With 64 fresh POSTs, each gets its own ephemeral source port, so the DAG hashes them across TMMs the same way it did with single-write requests. After enough batches, the per-TMM POST counts converge. In one of the runs, 4 TMMs each took exactly 16 of 64 POSTs. Body size matters for HTTP::collect The `/bulk_load` handler reads `Content-Length` and calls `HTTP::collect $cl` to buffer the entire body before processing. We cap at 16 MiB to protect TMM memory; that's plenty of headroom (~400k UUIDs per batch) but it's a real ceiling worth knowing about. The default of 15,625 UUIDs is ~580 KiB, which is well within bounds. An aside: log volume kills throughput at this rate Our first three bulk-post runs showed throughput drifting downward across consecutive runs...163k/s, then 129k/s, then 122k/s on the same hardware with no other state changes between them. The cause turned out to be the calling rule's logging itself. The `/bulk_load` and `/reset` handlers each had unconditional `log local0.` statements, producing 64 + 200 = 264 syslog writes per test cycle on top of the LOCALDB sample logs. After silencing those handlers (the response bodies already carried the per-batch timing data, so we lost no visibility), runs stabilized at ~133k writes/s ± 4% and survived 60-second sleeps with no warmup penalty. The lesson generalizes: at high write rates, the rule path needs to be quiet, not just "not chatty." Even gated log statements run their gate evaluation on every request, and unconditional ones write to syslog regardless of intent. When the per-write iRule cost is in the single-digit microseconds, *any* per-request work shows up. The rule of thumb that emerged: log statements that fire once per HTTP request are fine for diagnostics (`/probe`, `/whoami`) but should be sampled or removed entirely from the hot path (`/load`, `/bulk_load`, `/reset`). The loader can carry timing data back in response bodies and aggregate it client-side, which is both faster and more useful for analysis. Worth flagging that the absolute throughput numbers here (130-160k writes/s) reflect the test environment: a BIG-IP VE running on an Intel NUC under VMware, sharing the host with the load generator and other VMs. Those are not headroom numbers; they're contention-dominated. A 16-vCPU appliance without that contention should comfortably scale 5-10× from these figures, putting bulk-load throughput into the millions of writes per second on real hardware. The Code The updated `LOCALDB.tcl`, the test harness `subtable_test_updates.tcl`, the Python loader/prober/timing-analyzer, and the USAGE.md are all in the irules-subtable-discovery repo out on Github. Two key bits to study: The `init_table` proc that does the timing-probe discovery, including the fallback path that logs a WARNING and uses a slow name rather than failing silently when discovery exhausts its tries. The 200-try ceiling is sized for 16+ TMMs; on a 4-TMM box you'll typically find a local name in 1-3 tries. The `/probe` endpoint and the loader's `probe` mode. Together they let you take any subtable name and identify which TMM owns it in seconds. Worth keeping in your toolkit; it's the cleanest way I've found to interrogate TMOS's hash assignments. Closing Thoughts The whole episode reinforced something I keep relearning: when a working pattern looks weirdly complicated, the complications are usually load-bearing. The original LOCALDB rule looked over-engineered with its random names and timing probes and retry loops. It was actually exactly as engineered as it needed to be. My "cleaner" rewrite was simpler because I'd quietly assumed something untrue about how TMOS assigns ownership. The truth was readable from a 6-line timing report; I just hadn't generated one yet. If you're going to deviate from a working pattern, the deviation should be the thing you instrument first. Note: the original LocalDB proc library I built this from has been updated by the author in a couple different ways since I shared my work with him. I didn't fold that work in here, but I'll post those updates along with the original when I get permission to do so.
