Creating a tmsh script with iControl REST and using it to restart HTTPD
Problem this snippet solves: TMSH has the ability to create tcl scripts that can be used to run multiple commands and transactions. It is rare that you will want to create on with TMSH but there are a few cases where this may be desirable. One of these is to restart HTTPD, which is difficult to do from iControl REST because the REST API is running over HTTPD and the restart will not be clean. See K13292945. This Python script creates a tmsh script, and then runs it to restart HTTPD. How to use this snippet: Syntax is <program_name.py> host user password Sample output: ./rest_script_example.py 10.155.117.12 admin admin Before httpd (pid 3186) is running... After httpd (pid 3289) is running... Code : #!/usr/bin/python #[email protected] #Makes tmsh script to restart HTTP #Syntax: host username password import json #allow python 2 and python 3 by loading the correct libraries. try: from http.client import BadStatusLine from urllib.parse import urlparse, urlencode from urllib.request import urlopen, Request from urllib.error import HTTPError except ImportError: from httplib import BadStatusLine from urlparse import urlparse from urllib import urlencode from urllib2 import urlopen, Request, HTTPError import ssl import sys import time #Internal calls will not verify certs so disable cert verification. ssl._create_default_https_context = ssl._create_unverified_context #Create request for token based authentication. This is in Bigip 12 and later: url = 'https://'+sys.argv[1]+'/mgmt/shared/authn/login' values = {'username' : sys.argv[2], 'password' : sys.argv[3], 'loginProviderName' : 'tmos'} values = json.dumps(values).encode('utf-8') Request(url,data=values) req = Request(url,data=values) req.add_header('Content-Type' , 'application/json') #Request authentication token. response = urlopen(req) #auth=result will be a json data structure. auth_result = response.read() #print (auth_result) #Json.loads makes an internal python data structure that is easier to extract auth token from json. #Now construct icontrol rest query for device-groups info. auth=json.loads(auth_result) token=(auth['token']['token']) #print(token) #Get current PID of HTTPD url = 'https://'+sys.argv[1]+'/mgmt/tm/sys/service/httpd/stats' req = Request(url) req.add_header('X-F5-Auth-Token',auth['token']['token']) response = urlopen(req,data=None) json_response=(response.read()) python_response=json.loads(json_response) print("Before") print(python_response["apiRawValues"]["apiAnonymous"]) #look for script with name to make sure that the script does not already exist url = 'https://'+sys.argv[1]+'/mgmt/tm/cli/script/example.tcl' #urllib2 raises an exception with an HTTP 404 req = Request(url) req.add_header('X-F5-Auth-Token',auth['token']['token']) try: response = urlopen(req,data=None) except HTTPError as err: if err.code==404: #print (err.code) #print("\nCreate cli script\n") #request create here url = 'https://'+sys.argv[1]+'/mgmt/tm/cli/script' req = Request(url) req.add_header('X-F5-Auth-Token',auth['token']['token']) req.add_header('Content-Type' , 'application/json') values = {"name":"example.tcl", "apiAnonymous": "proc script::init {} {\n}\n\nproc script::run {} {\n tmsh::run util bash -c 'killall -9 httpd' \n tmsh::start sys service httpd\n} \n\nproc script::help {} {\n}\n\nproc script::tabc {} {\n}\n"} values = json.dumps(values) response = urlopen(req,data=values) response_py=(json.load(response)) #print(json.dumps(response_py,sort_keys=True,indent=4)) #Now run script url = 'https://'+sys.argv[1]+'/mgmt/tm/cli/script/example.tcl' req = Request(url) req.add_header('X-F5-Auth-Token',auth['token']['token']) req.add_header('Content-Type' , 'application/json') values = {"kind":"tm:cli:script:runstate","command":"run"} values = json.dumps(values).encode('utf-8') # Killing HTTPD will abort the connection so catch the exception. try: urlopen(req,data=values) except BadStatusLine: pass #Wait for httpd to restart so you can query. time.sleep(5) #Get current PID of HTTPD after restart url = 'https://'+sys.argv[1]+'/mgmt/tm/sys/service/httpd/stats' req = Request(url) req.add_header('X-F5-Auth-Token',auth['token']['token']) response = urlopen(req,data=None) json_response=(response.read()) python_response=json.loads(json_response) print("After") print(python_response["apiRawValues"]["apiAnonymous"]) Tested this on version: 13.0Restsh is now available under an Open Source license!
I am proud to announce that the complete Restsh package is now released under the GNU General Public License version 3 (GPLv3) or later. There are no hidden restrictions — we are not withholding any enterprise features. Restsh will remain actively maintained and further developed by Axians IT Security. What is Restsh? Restsh is a lightweight Bash-based shell environment for working with REST APIs from the command line. It was built for interactive use, for automation in scripts, and for robust execution in CI/CD pipelines. Restsh is a core component of the Axians Automation Framework, enabling automated management of F5 environments via GitLab CI/CD pipelines. Restsh does not replace your shell. Instead it exports a small set of environment variables and provides focused helper functions to call and parse REST APIs. Combine the power of Bash, curl, jq and Mustache templates to build reliable, repeatable workflows and automation. What can I do with it? Almost anything related to REST API automation. Restsh supports the common REST verbs and includes autocompletion for F5 and GitLab APIs. To simplify day-to-day tasks, it ships hundreds of small, focused helper scripts that wrap API endpoints — designed with the Unix principle in mind: do one thing well. These compact scripts can be piped together, filtered, or executed inside loops. For example, exporting all WAF policies from an F5 is a simple one-liner: f5.asm.policy.list -r -f ".items[].fullPath" | XARGS f5.asm.policy.export Modular design Restsh is modular and provides many functions to interact with the REST APIs of F5 BIG-IP, F5 OS-A and GitLab: F5 functions F5 OS-A functions GitLab functions Do I have to sell my soul to get it? Restsh is publicly available and can be downloaded from the official GitHub repository. Support This is the open-source, community-supported edition of Restsh. For enterprise-grade support and SLAs, Axians IT Security GmbH offers commercial support plans. Contact me to discuss options. Documentation Full documentation is available online: https://axiansitsecurity.github.io/Restsh/ Tutorials F5 AS3 management F5 certificate deployment with HashiCorp Vault F5 Declarative Onboarding (DO) F5 Declarative WAF F5 New Declarative API GitLab PipelineBIG-IP Report
Problem this snippet solves: Overview This is a script which will generate a report of the BIG-IP LTM configuration on all your load balancers making it easy to find information and get a comprehensive overview of virtual servers and pools connected to them. This information is used to relay information to NOC and developers to give them insight in where things are located and to be able to plan patching and deploys. I also use it myself as a quick way get information or gather data used as a foundation for RFC's, ie get a list of all external virtual servers without compression profiles. The script has been running on 13 pairs of load balancers, indexing over 1200 virtual servers for several years now and the report is widely used across the company and by many companies and governments across the world. It's easy to setup and use and only requires auditor (read-only) permissions on your devices. Demo/Preview Interactive demo http://loadbalancing.se/bigipreportdemo/ Screen shots The main report: The device overview: Certificate details: How to use this snippet: Installation instructions BigipReport REST This is the only branch we're updating since middle of 2020 and it supports 12.x and upwards. Downloads: https://github.com/net-utilities/BigIPReport/releases Documentation, installation instructions and troubleshooting: https://net-utilities.github.io/BigIPReport/ BIG-IP Report (Legacy) Older version of the report that only runs on Windows and is depending on a Powershell plugin originally written by Joe Pruitt (F5). The documentation for this will stay on loadbalancing.se. BIG-IP Report (only download this if you have v10 devices): https://github.com/net-utilities/BigIPReport/releases/download/v5.8.0/bigipreport-5.4.0-beta.zip iControl Snapin https://github.com/net-utilities/BigIPReport/releases/download/v5.8.0/f5-icontrol.zip Got issues/problems/feedback? Still have issues? Drop a comment below. We usually reply quite fast. Any bugs found, issues detected or ideas contributed makes the report better for everyone, so it's always appreciated. --- Join us on Discord: https://discord.gg/7JJvPMYahA Tested this on versions: 12, 13, 14, 15, 16, 17 (probably works on later versions too)F5 certificate deployment with iControl REST and HashiCorp Vault
Welcome to the first article in a comprehensive series dedicated to administrating F5 BIG-IP systems through the power of the iControl REST API. This series is designed for both beginners and experienced network administrators who want to streamline their F5 management workflows and embrace infrastructure-as-code principles. Throughout this series, we will explore practical, real-world scenarios for managing F5 BIG-IP appliances programmatically. Rather than relying solely on the F5 web interface, we'll leverage the iControl REST API. To make this journey accessible and efficient, I'll be using Restsh, my open-source command-line interface tool. Restsh is purpose-built to simplify interactions with the iControl REST API, abstracting away complexity while maintaining full flexibility. Whether you're new to F5 REST API or looking to enhance your automation skills, these tutorials will provide clear, step-by-step guidance with practical examples you can implement immediately. In this workflow, you’ll create the certificate materials on the F5, have them signed by a trusted Certificate Authority (in this case, HashiCorp Vault), and then deploy the signed certificate back to the F5. This process ensures that the private key never leaves the F5. Initial configuration of Restsh See First Steps for the initial configuration of Restsh. Connect Connect to the F5 with Restsh: restsh Select your F5 1. Create a key pair and signing request (CSR) on the F5 The first step is to create a key pair and a Certificate Signing Request (CSR) on the F5. A key pair consists of a private key (which stays secret on the F5) and a public key. The CSR contains the public key and certificate details that will be sent to a Certificate Authority (CA) for signing. Below are the certificate parameters we’ll use for this example: Name on the F5: test.lab.lan Partition: Common CN: test.lab.lan Subject Alternative Names: test.lab.lan, test2.lab.lan Keytype: EC with SECP384R1 f5.cert.csr.create --cn=test.lab.lan --keytype=ec-private --curvename=secp384r1 -a test.lab.lan -a test2.lab.lan -n test.lab.lan 2. Download the signing request The next step is to download the Certificate Signing Request from the F5 to your local machine. This file contains the public key and certificate information that needs to be signed by the Certificate Authority. We’ll save it to a temporary directory for processing. f5.cert.csr.get /Common/test.lab.lan > "$RESTSH_TMP/test.lab.lan.csr" You can inspect the details of the CSR to verify it contains the correct information (domain names, key type, etc.). The cert.* commands are convenient wrapper scripts for the OpenSSL command-line tool, making certificate operations easier. cert.csr.show "$RESTSH_TMP/test.lab.lan.csr" You can also view the public key contained in the CSR: cert.csr.pubkey "$RESTSH_TMP/test.lab.lan.csr" 3. Sign the CSR with HashiCorp Vault Now we use HashiCorp Vault to sign the CSR. Vault acts as your Certificate Authority (CA) and will create a signed certificate based on your CSR. First, you authenticate with Vault using vault login, then you submit the CSR for signing. The vault write command sends your CSR to the Vault PKI backend, which signs it and returns a complete certificate. In this example, we’re using the signing backend/role pki/sign/lab-ca (adjust this to match your Vault configuration). vault login vault write -field=certificate pki/sign/lab-ca csr=@"$RESTSH_TMP/test.lab.lan.csr" > "$RESTSH_TMP/test.lab.lan.crt" After signing, you can verify the certificate details to ensure it was signed correctly and contains the expected information (domain names, validity period, key type, etc.): cert.x509.show "$RESTSH_TMP/test.lab.lan.crt" You can also view the public key in the signed certificate to confirm it matches the public key from the CSR: cert.x509.pubkey "$RESTSH_TMP/test.lab.lan.crt" 4. Upload the signed certificate to the F5 Finally, we upload the signed certificate back to the F5. This makes the certificate available for use in SSL profiles. We use the same certificate name that we used when creating the CSR, so the F5 knows to pair it with the private key we created in step 1. f5.cert.import /Common/test.lab.lan "$RESTSH_TMP/test.lab.lan.crt" You can view the imported certificate with following command: f5.cert.get /Common/test.lab.lan 5. Cleanup Cleanup is optional the files in the temporary directory will be automatically removed after Restsh finishes. However, if you want to manually remove the CSR and certificate files from your local machine, you can do so with the following command: rm "$RESTSH_TMP/test.lab.lan.csr" "$RESTSH_TMP/test.lab.lan.crt" Optionally sync the F5 cluster to ensure the new certificate is available on all cluster members. "failover" is here the name of the Sync-Failover device group. f5.cluster.config-sync failover Summary In this tutorial, we walked through the process of deploying a new certificate on an F5 device using Restsh and HashiCorp Vault. We created a key pair and CSR on the F5, signed the CSR with Vault, and then imported the signed certificate back to the F5. This workflow allows you to securely manage certificates without exposing private keys, while leveraging the power of Restsh for automation and integration with your existing infrastructure.Update an ASM Policy Template via REST-API - the reverse engineering way
I always want to automate as many tasks as possible. I have already a pipeline to import ASM policy templates. Today I had the demand to update this base policies. Simply overwriting the template with the import tasks does not work. I got the error message "The policy template ax-f5-waf-jump-start-template already exists.". Ok, I need an overwrite tasks. Searching around does not provide me a solution, not even a solution that does not work. Simply nothing, my google-foo have deserted me. Quick chat with an AI, gives me a solution that was hallucinated. The AI answer would be funny if it weren't so sad. I had no hope that AI could solve this problem for me and it was confirmed, again. I was configuring Linux systems before the internet was widely available. Let's dig us in the internals of the F5 REST API implementation and solve the problem on my own. I took a valid payload and removed a required parameter, "name" in this case. The error response changes, this is always a good signal in this stage of experimenting. The error response was "Failed Required Fields: Must have at least 1 of (title, name, policyTemplate)". There is also a valid field named "policyTemplate". My first thought: This could be a reference for an existing template to update. I added the "policyTemplate" parameter and assigned it an existing template id. The error message has changed again. It now throws "Can't use string (\"ox91NUGR6mFXBDG4FnQSpQ\") as a HASH ref while \"strict refs\" in use at /usr/local/share/perl5/F5/ASMConfig/Entity/Base.pm line 888.". An perl error that is readable and the perl file is in plain text available. Looking at the file at line 888: The Perl code looks for an "id" field as property of the "policyTemplate" parameter. Changing the payload again and added the id property. And wow that was easy, it works and the template was updated. Final the payload for people who do not want to do reverse engineering. Update POST following payload to /mgmt/tm/asm/tasks/import-policy-template to update an ASM policy template: { "filename": "<username>~<filename>", "policyTemplate": { "id": "ox91NUGR6mFXBDG4FnQSpQ" } } Create POST following payload /mgmt/tm/asm/tasks/import-policy-template to create an ASM policy template: { "name": "<name>", "filename": "<username>~<filename>" } Hint: You must upload the template before to /var/config/rest/downloads/<username>~<filename>". Conclusion Documentation is sometimes overrated if you can read Perl. Missed I the API documentation for this endpoint and it was just a exercise for me?
FTP Session Logging
Problem this snippet solves: This iRule logs FTP connections and username information. By default connection mapping from client through BIG-IP to server is logged as well as the username entered by the client. Optionally you can log the entire FTP session by uncommenting the log message in CLIENT_DATA. Code : # This iRule logs FTP connections and username information. # By default connection mapping from client through BIG-IP to server is logged # as well as the username entered by the client. Optionally you can log the # entire FTP session by uncommenting the log message in CLIENT_DATA. when CLIENT_ACCEPTED { set vip [IP::local_addr]:[TCP::local_port] set user "unknown" } when CLIENT_DATA { # uncomment for full session logging #log local0. "[IP::client_addr]:[TCP::client_port]: collected payload ([TCP::payload length]): [TCP::payload]" # check if payload contains the string we want to replace if { [TCP::payload] contains "USER" } { # use a regular expression to save the user name ## regex modified by arkashik regexp "USER \(\[a-zA-Z0-9_-]+)" [TCP::payload] all user # log connection mapping from client through BIG-IP to server log local0. "FTP connection from $client. Mapped to $inside -> $node, user $user" TCP::release TCP::collect } else { TCP::release TCP::collect } } when SERVER_CONNECTED { set client "[IP::client_addr]:[TCP::client_port]" set node "[IP::server_addr]:[TCP::server_port]" set inside "[serverside {IP::local_addr}]:[serverside {TCP::local_port}]" TCP::collect } when SERVER_DATA { TCP::release clientside { TCP::collect } }F5 Per-App AS3 Part 2 How to see if there are manual changes!
Code version: The code was tested on 17.1.5.3 AS3: 3.55 For more about AS3 and per-app AS3 see my previous code share Part 1 article: https://community.f5.com/kb/codeshare/f5-per-app-as3-part-1-how-share-tenant-specific-object/345072 First we will send Per-App AS3 declaration as shown below. { "id": "per-app-declarationn", "schemaVersion": "3.55.0", "controls": { "class": "Controls", "logLevel": "debug", "trace": true, "traceResponse": true }, "A2": { "class": "Application", "service": { "class": "Service_HTTP", "virtualAddresses": [ "10.0.3.10" ], "pool": "web2_pool" }, "web2_pool": { "class": "Pool", "monitors": [ "http" ], "members": [{ "servicePort": 80, "serverAddresses": [ "192.7.21.10", "192.7.21.11" ] }] } } } Then we will change for example virtual server ip from 10.0.3.10 to 10.0.3.11 and we will send the same declaration but with "dryRun" set to true as this will cause AS3 to validate the config but not to execute it and with trace and traceResponse we will get the difference 😎 { "id": "per-app-declarationn", "schemaVersion": "3.55.0", "controls": { "class": "Controls", "logLevel": "debug", "trace": true, "dryRun": true, "traceResponse": true }, "A2": { "class": "Application", "service": { "class": "Service_HTTP", "virtualAddresses": [ "10.0.3.10" ], "pool": "web2_pool" }, "web2_pool": { "class": "Pool", "monitors": [ "http" ], "members": [{ "servicePort": 80, "serverAddresses": [ "192.7.21.10", "192.7.21.11" ] }] } } } Now we see that the IP has been changed from 10.0.3.10 to 10.0.3.11 and here we go now we have the difference ! This can be added in CI/CD to always first do "dry-run" using the original declaration to see if there are changes before executing the new AS3 declaration that could be for example changing the IP address to 10.0.3.12 but using the official way. Look at the Json reply "diff" section that is seen thanks to trace and traceResponse options and an automation can just check this section and stop the new deployment if the manual changes need to be reviewed first. For AS3 basic declaration not Per-App actually the "dry-run" is a different. F5 likes changing the naming like Local Traffic policies to Endpoint Policies or naming of TLS profiles between GU/TMSH and AS3 😅 { "$schema": "https://raw.githubusercontent.com/F5Networks/f5-appsvcs-extension/refs/heads/main/schema/3.55.0/as3-schema.json", "class": "AS3", "action": "dry-run", "logLevel": "debug", "trace": true, "traceResponse": true, "persist": true, "declaration": { "class": "ADC", "schemaVersion": "3.55.0", "id": "BIG-IP-Example-Tenant", "Example": { "class": "Tenant", "Shared": { "class": "Application", "template": "shared", "Example_Response": { "remark": "Used for F5 response", "class": "iRule", "iRule": { "base64": "d2hlbiBIVFRQX1JFUVVFU1Qgew0KICAgSFRUUDo6cmVzcG9uZCAyMDAgY29udGVudCB7DQogICAgICA8aHRtbD4NCiAgICAgICAgIDxoZWFkPg0KICAgICAgICAgICAgPHRpdGxlPkFwb2xvZ3kgUGFnZTwvdGl0bGU+DQogICAgICAgICA8L2hlYWQ+DQogICAgICAgICA8Ym9keT4NCiAgICAgICAgICAgIFdlIGFyZSBzb3JyeSwgYnV0IHRoZSBzaXRlIHlvdSBhcmUgbG9va2luZyBmb3IgaXMgdGVtcG9yYXJpbHkgb3V0IG9mIHNlcnZpY2U8YnI+DQogICAgICAgICAgICBJZiB5b3UgZmVlbCB5b3UgaGF2ZSByZWFjaGVkIHRoaXMgcGFnZSBpbiBlcnJvciwgcGxlYXNlIHRyeSBhZ2Fpbi4NCiAgICAgICAgIDwvYm9keT4NCiAgICAgIDwvaHRtbD4NCiAgIH0NCn0=" } } } } } } This will not show if someone has manually added a vlan for example as only changes on the apps that were deployed with AS3 will be seen. For those you will get error like the one below when you try to delete the partition. "" 0107082a:3: All objects must be removed from a partition "" https://my.f5.com/manage/s/article/K02718312 https://my.f5.com/manage/s/article/K000138638 Github link: https://github.com/Nikoolayy1/AS3-Per-App-Manual-Changes/tree/mainF5 Per-App AS3 Part 1 How Share Tenant specific object
Code version: The code was tested on 17.1.5.3 AS3: 3.55 For testing F5 AS3 you can use combination of Postman as Visual Studio has option to post AS3 declarations but not Per-App specific as that will be needed for executing my code. Still I recommend using Visual Studio for general AS3 or Fast Templates and you can use the reference links for how to use Visual Studio F5 extensions and maybe F5 will update the extension in the future to work for Per-App AS3 with options for adding tenant and as variables 🤔 For FAST nice example is at https://github.com/Nikoolayy1/fast-template-examples The same way the /Shared objects can be configured under /Common can be done under a specific tenant and this is really useful if Per-App AS3 is enabled as each AS3 app will use separate declaration and all of them can use a customer iRule defined under /<Tenant>/Shared. References for AS3 Declarative API: https://www.youtube.com/watch?v=rrjrcnwAe1g&t=433s AS3 Foundations: Beyond Imperatives - What the Heck is AS3? | DevCentral https://www.youtube.com/watch?v=OHSBpOtQg_0&list=PLrn3yqXftAPUPdSP_kbCX6BDicVprsnfI Per-App AS3 is enabled by default for the AS3 3.50.x and above but if needed enable it. POST Endpoint: /mgmt/shared/appsvcs/settings { "perAppDeploymentAllowed": true } POST Endpoint for Creating the Tenant and the Shared irule that is base64 endpoint: /mgmt/shared/appsvcs/declare { "schemaVersion": "3.45.0", "Shared": { "class": "Application", "template": "shared", "test": { "class": "iRule", "remark": "yes", "iRule": { "base64": "d2hlbiBIVFRQX1JFUVVFU1Qgew0KICAgSFRUUDo6cmVzcG9uZCAyMDAgY29udGVudCB7DQogICAgICA8aHRtbD4NCiAgICAgICAgIDxoZWFkPg0KICAgICAgICAgICAgPHRpdGxlPkFwb2xvZ3kgUGFnZTwvdGl0bGU+DQogICAgICAgICA8L2hlYWQ+DQogICAgICAgICA8Ym9keT4NCiAgICAgICAgICAgIFdlIGFyZSBzb3JyeSwgYnV0IHRoZSBzaXRlIHlvdSBhcmUgbG9va2luZyBmb3IgaXMgdGVtcG9yYXJpbHkgb3V0IG9mIHNlcnZpY2U8YnI+DQogICAgICAgICAgICBJZiB5b3UgZmVlbCB5b3UgaGF2ZSByZWFjaGVkIHRoaXMgcGFnZSBpbiBlcnJvciwgcGxlYXNlIHRyeSBhZ2Fpbi4NCiAgICAgICAgIDwvYm9keT4NCiAgICAgIDwvaHRtbD4NCiAgIH0NCn0=" } } } } POST Endpoint for sending Per-App AP that references the Tenant shared iRule I: /mgmt/shared/appsvcs/declare/Example/applications { "schemaVersion": "3.45.0", "A1": { "class": "Application", "service": { "class": "Service_HTTP", "virtualAddresses": [ "10.0.1.10" ], "pool": "web_pool", "iRules": [ { "use": "/Test/Shared/test" }, "insert_xff" ] }, "web_pool": { "class": "Pool", "monitors": [ "http" ], "members": [ { "servicePort": 80, "serverAddresses": [ "192.0.1.10", "192.0.1.12" ] } ] }, "insert_xff": { "class": "iRule", "iRule": "when HTTP_REQUEST { HTTP::header insert X-Forwarded-For [IP::client_addr] }" } } } Github link: Editing AS3-Per-App-Shared-Tenant-Objects/README.md at main · Nikoolayy1/AS3-Per-App-Shared-Tenant-Objects Note! To see the Tenant config use GET Endpoint /mgmt/shared/appsvcs/declare/Example or for the app /mgmt/shared/appsvcs/declare/Example/A1 For only the Tenant shared config use GET /mgmt/shared/appsvcs/declare/Example/applications/Shared By placing the pools in the /Shared under the Tenant all Per-AS3 declarations for Tenant can use the pool like for the iRule example I showed and also the pools can share the node IP addresses like the "shareNodes" option but only for the tenant. https://my.f5.com/manage/s/article/K88250015F5 Container Ingress Services (CIS) and using k8s traffic policies to send traffic directly to pods
This article will take a look how you can use health monitors on the BIG-IP to solve the issue with constant AS3 REST-API pool member changes or when there is a sidecar service mesh like Istio (F5 has version called Aspen mesh of the istio mesh) or Linkerd mesh. I also have described some possible enchantments for CIS/AS3, Nginx Ingress Controller or Gateway Fabric that will be nice to have in the future. Intro Install Nginx Ingress Open source and CIS F5 CIS without Ingress/Gateway F5 CIS with Ingress F5 CIS with Gateway fabric Summary 1. Intro F5 CIS allows integration between F5 and k8s kubernetes or openshift clusters. F5 CIS has two modes and that are NodePort and ClusterIP and this is well documented at https://clouddocs.f5.com/containers/latest/userguide/config-options.html . There is also a mode called auto that I prefer as based on k8s service type NodePort or ClusterIP it knows how to configure the pool members. CIS in ClusterIP mode generally is much better as you bypass the kube-proxy as send traffic directly to pods but there could be issues if k8s pods are constantly being scaled up or down as CIS uses AS3 REST-API to talk and configure the F5 BIG-IP. I also have seen some issues where a bug or a config error that is not well validated can bring the entire CIS to BIG-IP control channel down as you then see 422 errors in the F5 logs and on CIS logs. By using NodePort and "externaltrafficpolicy: local" and if there is an ingress also "internaltrafficpolicy: local" you can also bypass the kubernetes proxy and send traffic directly to the pods and BIG-IP health monitoring will mark the nodes that don't have pods as down as the traffic policies prevent nodes that do not have the web application pods to send the traffic to other nodes. 2..Install Nginx Ingress Open source and CIS As I already have the k8s version of nginx and F5 CIS I need 3 different classes of ingress. k8s nginx is end of life https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/ , so my example also shows how you can have in parallel the two nginx versions the k8s nginx and F5 nginx. There is a new option to use The Operator Lifecycle Manager (OLM) that when installed will install the components and this is even better way than helm (you can install OLM with helm and this is even newer way to manage nginx ingress!) but I found it still in early stage for k8s while for Openshift it is much more advanced. I have installed Nginx in a daemonset not deployment and I will mention why later on and I have added a listener config for the F5 TransportServer even if later it is seen why at the moment it is not usable. helm install -f values.yaml ginx-ingress oci://ghcr.io/nginx/charts/nginx-ingress \ --version 2.4.1 \ --namespace f5-nginx \ --set controller.kind=daemonset \ --set controller.image.tag=5.3.1 \ --set controller.ingressClass.name=nginx-nginxinc \ --set controller.ingressClass.create=true \ --set controller.ingressClass.setAsDefaultIngress=false cat values.yaml controller: enableCustomResources: true globalConfiguration: create: true spec: listeners: - name: nginx-tcp port: 88 protocol: TCP kubectl get ingressclasses NAME CONTROLLER PARAMETERS AGE f5 f5.com/cntr-ingress-svcs <none> 8d nginx k8s.io/ingress-nginx <none> 40d nginx-nginxinc nginx.org/ingress-controller <none> 32s niki@master-1:~$ kubectl get pods -o wide -n f5-nginx NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ingress-controller-2zbdr 1/1 Running 0 62s 10.10.133.234 worker-2 <none> <none> nginx-ingress-controller-rrrc9 1/1 Running 0 62s 10.10.226.87 worker-1 <none> <none> niki@master-1:~$ The CIS config is shown below. I have used "pool_member_type" auto as this allows Cluster-IP or NodePort services to be used at the same time. helm install -f values.yaml f5-cis f5-stable/f5-bigip-ctlr cat values.yaml bigip_login_secret: f5-bigip-ctlr-login rbac: create: true serviceAccount: create: true name: namespace: f5-cis args: bigip_url: X.X.X.X bigip_partition: kubernetes log_level: DEBUG pool_member_type: auto insecure: true as3_validation: true custom_resource_mode: true log-as3-response: true load-balancer-class: f5 manage-load-balancer-class-only: true namespaces: [default, test, linkerd-viz, ingress-nginx, f5-nginx] # verify-interval: 35 image: user: f5networks repo: k8s-bigip-ctlr pullPolicy: Always nodeSelector: {} tolerations: [] livenessProbe: {} readinessProbe: {} resources: {} version: latest 3. F5 CIS without Ingress/Gateway Without Ingress actually the F5's configuration is much simpler as you just need to create nodeport service and the VirtualServer CR. As you see below the health monitor marks the control node and the worker node that do not have pod from "hello-world-app-new-node" as shown in the F5 picture below. Sending traffic without Ingresses or Gateways removes one extra hop and sub-optimal traffic patterns as when the Ingress or Gateway is in deployment mode for example there could be 20 nodes and only 2 ingress/gateway pods on 1 node each. Traffic will need to go to only those 2 nodes to enter the cluster. apiVersion: v1 kind: Service metadata: name: hello-world-app-new-node labels: app: hello-world-app-new-node spec: externalTrafficPolicy: Local ports: - name: http protocol: TCP port: 8080 targetPort: 8080 selector: app: hello-world-app-new type: NodePort --- apiVersion: "cis.f5.com/v1" kind: VirtualServer metadata: name: vs-hello-new namespace: default labels: f5cr: "true" spec: virtualServerAddress: "192.168.1.71" virtualServerHTTPPort: 80 host: www.example.com hostGroup: "new" snat: auto pools: - monitor: interval: 10 recv: "" send: "GET /" timeout: 31 type: http path: / service: hello-world-app-new-node servicePort: 8080 For Istio and Linkerd Integration an irule could be needed to send custom ALPN extensions to the backend pods that now have a sidecar. I suggest seeing my article at "the Medium" for more information see https://medium.com/@nikoolayy1/connecting-kubernetes-k8s-cluster-to-external-router-using-bgp-with-calico-cni-and-nginx-ingress-2c45ebe493a1 Keep in mind that for the new options with Ambient mesh (sidecarless) the CIS without Ingress will not work as F5 does not speak HBONE (or HTTP-Based Overlay Network Environment) protocol that is send in the HTTP Connect tunnel to inform the zTunnel (layer 3/4 proxy that starts or terminates the mtls) about the real source identity (SPIFFE and SPIRE) that may not be the same as the one in CN/SAN client SSL cert. Maybe in the future there could be an option based on a CRD to provide the IP address of an external device like F5 and the zTunnel proxy to terminate the TLS/SSL (the waypoint layer 7 proxy usually Envoy is not needed in this case as F5 will do the HTTP processing) and send traffic to the pod but for now I see no way to make F5 work directly with Ambient mesh. If the ztunnel takes the identity from the client cert CN/SAN F5 will not have to even speak HBONE. 4. F5 CIS with Ingress Why we may need an ingress just as a gateway into the k8s you may ask? Nowadays many times a service mesh like linkerd or istio or F5 aspen mesh is used and the pods talk to each other with mTLS handled by the sidecars and an Ingress as shown in https://linkerd.io/2-edge/tasks/using-ingress/ is an easy way for the client-side to be https while the server side to be the service mesh mtls, Even ambient mesh works with Ingresses as it captures traffic after them. It is possible from my tests F5 to talk to a linkerd injected pods for example but it is hard! I have described this in more detail at https://medium.com/@nikoolayy1/connecting-kubernetes-k8s-cluster-to-external-router-using-bgp-with-calico-cni-and-nginx-ingress-2c45ebe493a1 Unfortunately when there is an ingress things as much more complex! F5 has Integration called "IngressLink" but as I recently found out it is when BIG-IP is only for Layer 3/4 Load Balancing and the Nginx Ingress Controller will actually do the decryption and AppProtect WAF will be on the Nginx as well F5 CIS IngressLink attaching WAF policy on the big-ip through the CRD ? | DevCentral Wish F5 to make an integration like "IngressLink" but the reverse where each node will have nginx ingress as this can be done with demon set and not deployment on k8s and Nginx Ingress will be the layer 3/4, as the Nginx VirtualServer CRD support this and to just allow F5 in the k8s cluster. Below is how currently this can be done. I have created a Transportserver but is not used as it does not at the momemt support the option "use-cluster-ip" set to true so that Nginx does not bypass the service and to go directly to the endpoints as this will cause nodes that have nginx ingress pod but no application pod to send the traffic to other nodes and we do not want that as add one more layer of load balancing latency and performance impact. The gateway is shared as you can have a different gateway per namespace or shared like the Ingress. apiVersion: v1 kind: Service metadata: name: hello-world-app-new-cluster labels: app: hello-world-app-new-cluster spec: internalTrafficPolicy: Local ports: - name: http protocol: TCP port: 8080 targetPort: 8080 selector: app: hello-world-app-new type: ClusterIP --- apiVersion: k8s.nginx.org/v1 kind: TransportServer metadata: name: nginx-tcp annotations: nginx.org/use-cluster-ip: "true" spec: listener: name: nginx-tcp protocol: TCP upstreams: - name: nginx-tcp service: hello-world-app-new-cluster port: 8080 action: pass: nginx-tcp --- apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: nginx-http spec: host: "app.example.com" upstreams: - name: webapp service: hello-world-app-new-cluster port: 8080 use-cluster-ip: true routes: - path: / action: pass: webapp The second part of the configuration is to expose the Ingress to BIG-IP using CIS. --- apiVersion: v1 kind: Service metadata: name: f5-nginx-ingress-controller namespace: f5-nginx labels: app.kubernetes.io/name: nginx-ingress spec: externalTrafficPolicy: Local type: NodePort selector: app.kubernetes.io/name: nginx-ingress ports: - name: http protocol: TCP port: 80 targetPort: http --- apiVersion: "cis.f5.com/v1" kind: VirtualServer metadata: name: vs-hello-ingress namespace: f5-nginx labels: f5cr: "true" spec: virtualServerAddress: "192.168.1.81" virtualServerHTTPPort: 80 snat: auto pools: - monitor: interval: 10 recv: "200" send: "GET / HTTP/1.1\r\nHost:app.example.com\r\nConnection: close\r\n\r\n" timeout: 31 type: http path: / service: f5-nginx-ingress-controller servicePort: 80 Only the nodes that have a pod will answer the health monitor. Hopefully F5 can make some Integration and CRD that makes this configuration simpler like the "IngressLink" and to add the option "use-cluster-ip" to the Transport server as Nginx does not need to see the HTTP traffic at all. This is on my wish list for this year 😁 Also if AS3 could reference existing group of nodes and just with different ports this could help CIS will need to push AS3 declaration of nodes just one time and then the different VirtualServers could reference it but with different ports and this will make the AS3 REST-API traffic much smaller. 5. F5 CIS with Gateway fabric This does not at the moment work as gateway-fabric unfortunately does not support "use-cluster-ip" option. The idea is to deploy the gateway fabric in daemonset and to inject it with a sidecar or even without one this will work with ambient meshes. As k8s world is moving away from an Ingress this will be a good option. Gateway fabric natively supports TCP , UDP traffic and even TLS traffic that is not HTTPS and by exposing the gateway fabric with a Cluster-IP or Node-Port service then with different hostnames the Gateway fabric will select to correct route to send the traffic to! helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway -f values-gateway.yaml cat values-gateway.yaml nginx: # Run the data plane per-node kind: daemonSet # How the data plane gets exposed when you create a Gateway service: type: NodePort # or NodePort # (optional) if you’re using Gateway API experimental channel features: nginxGateway: gwAPIExperimentalFeatures: enable: true apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: shared-gw namespace: nginx-gateway spec: gatewayClassName: nginx listeners: - name: https port: 443 protocol: HTTPS tls: mode: Terminate certificateRefs: - kind: Secret name: wildcard-tls allowedRoutes: namespaces: from: ALL --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: app-route namespace: app spec: parentRefs: - name: shared-gw namespace: nginx-gateway hostnames: - app.example.com rules: - backendRefs: - name: app-svc port: 8080 F5 Nginx Fabric mesh is evolving really fast from what I see , so hopefully we see the features I mentioned soon and always you can open a github case. The documentation is at https://docs.nginx.com/nginx-gateway-fabric and as this use k8s CRD the full options can be seen at TLS - Kubernetes Gateway API 6. Summary With the release of TMOS 21 F5 now supports much more health monitors and pool members, so this way of deploying CIS with NodePort services may offer benefits with TMOS 21.1 that will be the stable version as shown in https://techdocs.f5.com/en-us/bigip-21-0-0/big-ip-release-notes/big-ip-new-features.html With auto mode some services can still be directly exposed to BIG-IP as the CIS config changes are usually faster to remove a pool member pod than BIG-IP health monitors to mark a node as down. The new version of CIS that will be CIS advanced may take of the concerns of hitting a bug or not well validated configuration that could bring the control channel down and TMOS 21.1 may also handle AS3 config changes better with less cpu/memory issue, so there could be no need in the future of using trafficpolicies and NodePort mode and k8s services of this type. For ambient mesh my example with Ingress and Gateway seems the only option for direct communication at the moment. We will see what the future holds!F5 Distributed Cloud (XC) Custom Routes: Capabilities, Limitations, and Key Design Considerations
This article explores how Custom Routes work in F5 Distributed Cloud (XC), why they differ architecturally from standard Load Balancer routes, and what to watch out for in real-world deployments, covering backend abstraction, Endpoint/Cluster dependencies, and critical TLS trust and Root CA requirements.
