APM Optimisation Script
Problem this snippet solves: With the current Covid-19 lockdown, many workers are now working from home which is putting stress on existing APM VPN devices. This script looks through the config and suggests some changes to be made to reduce CPU usage, based on https://support.f5.com/csp/article/K46161759 Matthieu Dierick has created a YouTube video showing how to use this at https://youtu.be/F0Z1AnM3L54 Let me know if you have any questions or requirements. Source code is held at https://github.com/pwhitef5/apm-vpn-optimisation/tree/master How to use this snippet: Copy the file to the /var/tmp directory as apm-optimisation Give it permissions with `chmod +x /var/tmp/apm-optimisation` Run with `/var/tmp/apm-optimisation`. Output is to stdout Example: [root@apm-1:Active:Standalone] ~ # ./apm-optimisation APM Optimisation Visibility CPU Usage -------------------------------- Current Average Maximum 52% 30% 93% -------------------------------- Compression -------------------------------- Licensed Hardware unlimited None -------------------------------- --- Partition /Common --- Connectivity Profile Compression -------------------------------- Profile Name Status -------------------------------- myConnectivity Disabled myConnectivity2 Disabled -------------------------------- Network Access Profile Compression ----------------------------------------------------------------------------------------------------------- Name | Compression | Split-Tunneling | Client Traffic Classifier | DTLS ----------------------------------------------------------------------------------------------------------- networkAccess | Enabled | Enabled | Disabled | Enabled networkAccess2 | Disabled | Enabled | Disabled | Disabled ----------------------------------------------------------------------------------------------------------- --- Optimisation Suggestions --- - CPU rate is LOW. Go down the Winchester and wait for it all to blow over - Hardware Compression is not included so consider turning off the feature ------- Partition /Common ------- - To turn off compression in the connectivity profile, run the command 'tmsh modify apm profile connectivity /Common/myConnectivity compression disabled' - To turn off compression in the NA profile, run the command 'tmsh modify apm resource network-access /Common/networkAccess compression none' - To turn on Client Traffic Classifier, run the commands below: tmsh create apm resource client-rate-class /Common/rate_class_2M { rate 2000000 } tmsh create apm resource client-rate-class /Common/rate_class_1M { rate 1000000 } tmsh create apm resource client-traffic-classifier /Common/client-traffic-classifier-1 { entries add { entry { client-rate-class rate_class_1M dst-ip any dst-mask any dst-port https src-ip any src-mask any } } } tmsh modify apm resource network-access /Common/networkAccess client-traffic-classifier client-traffic-classifier-1 - Network Access profile /Common/networkAccess is using SNAT automap. Consider using a SNAT pool - To turn on Client Traffic Classifier, run the commands below: tmsh create apm resource client-rate-class /Common/rate_class_2M { rate 2000000 } tmsh create apm resource client-rate-class /Common/rate_class_1M { rate 1000000 } tmsh create apm resource client-traffic-classifier /Common/client-traffic-classifier-1 { entries add { entry { client-rate-class rate_class_1M dst-ip any dst-mask any dst-port https src-ip any src-mask any } } } tmsh modify apm resource network-access /Common/networkAccess2 client-traffic-classifier client-traffic-classifier-1 - To turn on DTLS, create a duplicate virtual server listening on UDP and enabled DTLS in the Network Access List Network Settings ( see https://devcentral.f5.com/s/articles/APM-DTLS-Virtual-Server-iApp ) - Network Access profile /Common/networkAccess2 is using SNAT automap. Consider using a SNAT pool ----------------------------------------------------------------------------------------------------------- Code : #!/bin/bash # Version 5 8/4/2020 P.White # This is a script to check your APM system and give suggestions to reduce CPU usage # Taken from suggestions at https://support.f5.com/csp/article/K46161759 # v2 - small typo fix line 119 create changed to modify # v3 - updated classifier to only include https as it was causing an error # v4 - loops through admin partitions and prints out for each # v5 - added DTLS check and suggestion suggestions="--- Optimisation Suggestions ---\n" getLicensedCompression () { # Show the licensed compression comp=`tmsh -q show sys license detail|grep perf_http_compression|awk '{print $2}'|sed 's/\[\(.*\)\]/\1/g'` if [ x$comp != "x" ];then echo -n "$comp" else echo -n "Error!" fi } getHardwareCompression () { # Show hardware compression hcomp=`tmsh -q show sys license detail|grep "HTTP Hardware Compression"` if [ x$hcomp = "x" ];then # Hardware compression is not enabled echo -n "None" else echo -n "$hcomp" fi } clear echo "APM Optimisation Visibility" # CPU usage cur=`tmsh -q show sys cpu |grep "Utilization"|awk '{print $2}'` avg=`tmsh -q show sys cpu |grep "Utilization"|awk '{print $3}'` max=`tmsh -q show sys cpu |grep "Utilization"|awk '{print $4}'` if [ $avg -gt 90 ];then suggestions+=" - CPU rate is VERY HIGH! Turn off compression, implement split tunneling and consider more processing\n" elif [ $avg -gt 60 ];then suggestions+=" - CPU rate is HIGH! Turn off compression and consider split tunneling for non-internal traffic\n" elif [ $avg -gt 40 ];then suggestions+=" - CPU rate is MEDIUM. Consider turning off compression where required\n" else suggestions+=" - CPU rate is LOW. Go down the Winchester and wait for it all to blow over\n" fi echo echo "CPU Usage" echo "--------------------------------" echo -e "Current\tAverage\tMaximum" echo -e "$cur%\t$avg%\t$max%" echo "--------------------------------" echo # Compression clic=`getLicensedCompression` chw=`getHardwareCompression` if [ $chw = "None" ];then suggestions+=" - Hardware Compression is not included so consider turning off the feature\n" fi echo "Compression" echo "--------------------------------" echo -e "Licensed\tHardware" echo -e "$clic\t$chw" echo "--------------------------------" # loop through adminstrative partitions for partition in `tmsh -q list auth partition one-line|awk '{print $3}'`;do suggestions+="\n------- Partition /$partition -------\n" echo " --- Partition /$partition ---" echo echo "Connectivity Profile Compression" echo "--------------------------------" echo -e "Profile Name\t\tStatus" echo "--------------------------------" for profile in `tmsh -q -c "cd /$partition;list apm profile connectivity one-line"|awk '{print $4}'`;do if [ $profile = "connectivity" ];then continue fi if [ `tmsh -q -c "cd /$partition;list apm profile connectivity $profile one-line"|grep "compress-gzip-level 0"|wc -l` -gt 0 ];then echo -e "$profile\t\tDisabled" else suggestions+=" - To turn off compression in the connectivity profile, run the command 'tmsh modify apm profile connectivity /$partition/$profile compress-gzip-level 0'\n" echo -e "$profile\t\tEnabled" fi done echo "--------------------------------" echo echo "Network Access Profile Compression" echo "-----------------------------------------------------------------------------------------------------------" echo -e " Name\t\t\t| Compression\t| Split-Tunneling\t| Client Traffic Classifier\t| DTLS" echo "-----------------------------------------------------------------------------------------------------------" for profile in `tmsh -q -c "cd /$partition;list apm resource network-access one-line"|awk '{print $4}'`;do # Compression if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "compression gzip"|wc -l` -gt 0 ];then echo -en "$profile\t\t| Enabled" suggestions+=" - To turn off compression in the NA profile, run the command 'tmsh modify apm resource network-access /$partition/$profile compression none'\n" else echo -en "$profile\t\t| Disabled" fi if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "split-tunneling true"|wc -l` -gt 0 ];then echo -en "\t| Enabled" else echo -en "\t| Disabled" suggestions+=" - To turn on split-tunneling, run the command 'tmsh modify apm resource network-access /$partition/$profile split-tunneling true'\n" suggestions+=" - To configure split-tunneling exclude traffic by DNS name, run the command 'tmsh modify apm resource network-access /$partition/$profile address-space-exclude-dns-name add { office.com microsoftonline.com google.com gmail.com facebook.com }'\n" suggestions+=" - To configure split-tunneling exclude traffic by IP address, run the command 'tmsh modify apm resource network-access /$partition/$profile address-space-include-subnet add { { subnet 10.0.0.0/8 } { subnet 172.16.0.0/16 } { subnet 192.168.0.0/16 } }'\n" fi if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "client-traffic-classifier "|wc -l` -gt 0 ];then echo -en "\t\t| Enabled" else echo -en "\t\t| Disabled" suggestions+=" - To turn on Client Traffic Classifier, run the commands below:\n" suggestions+="tmsh create apm resource client-rate-class /$partition/rate_class_2M { rate 2000000 }\n" suggestions+="tmsh create apm resource client-rate-class /$partition/rate_class_1M { rate 1000000 }\n" suggestions+="tmsh create apm resource client-traffic-classifier /$partition/client-traffic-classifier-1 { entries add { entry { client-rate-class rate_class_1M dst-ip any dst-mask any dst-port https src-ip any src-mask any } } }\n" suggestions+="tmsh modify apm resource network-access /$partition/$profile client-traffic-classifier client-traffic-classifier-1\n" fi if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "dtls true"|wc -l` -gt 0 ];then echo -en "\t\t\t| Enabled" else echo -en "\t\t\t| Disabled" suggestions+=" - To turn on DTLS, create a duplicate virtual server listening on UDP and enabled DTLS in the Network Access List Network Settings ( see https://devcentral.f5.com/s/articles/APM-DTLS-Virtual-Server-iApp )\n" fi # Check for SNAT automap if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line all-properties"|grep "snat automap"|wc -l` -gt 0 ];then suggestions+=" - Network Access profile /$partition/$profile is using SNAT automap. Consider using a SNAT pool\n" fi echo "" done echo "-----------------------------------------------------------------------------------------------------------" # Check VSs for mirroring for vs in `tmsh list ltm virtual one-line|awk '{print $3}'`;do if [ `tmsh -q -c "cd /$partition;list ltm virtual $vs mirror"|grep "mirror enabled"|wc -l` -gt 0 ];then echo echo "WARNING! Virtual Server /$partition/$vs has mirroring enabled\n" echo suggestions+="Consider disabling Connection Mirroring for virtual server /$partition/$vs with the command 'tmsh modify ltm virtual /$partition/$vs mirror disabled'\n" fi done done echo echo -e "$suggestions" echo "-----------------------------------------------------------------------------------------------------------" Tested this on version: 13.0Knowledge sharing: Velos and rSeries (F5OS) basic troubleshooting, logs and commands
This another part of my Knowledge sharing articles, where I will take a deeper look into Velos and rSeries investigation of issues, logs and command. 1. Velos HA controller and blade issues. As the Velos system is the one with two controllers in active/standby mode only with Velos it could be needed to check if there is an issue with the controller's HA. As the controller's HA order can be different for the system and the different partitions to check the HA for the system use the /var/log_controller/cc-confd file or for a partition HA issue look at the partition velos log at /var/F5/partition<ID>/log/velos.log . Also you can enable HA debug for the controllers with " system dbvars config debug confd ha-state-machine true ". Overview of HA: https://support.f5.com/csp/article/K19204400 Controller HA: https://support.f5.com/csp/article/K21130014 Partition HA: https://support.f5.com/csp/article/K58515297 List of Velos/rSeries services: Overview of F5 VELOS chassis controller services Overview of F5 VELOS partition services Overview of F5 rSeries system services 2. Entering into F5OS objects. The rSeries and Velos tenants are like vCMP quests with VIPRION and sometimes if there are access issues with them it could be needed to open their console. For this the "virtctl" command can be used and as an example " /usr/share/omd/kubevirt/virtctl console<tenant_name>-<tenant_instance_ID> ". Also as velos uses blades and partitions it could be needed to ssh to a blade with " ssh slot<number> " or to enter a partition with " docker exec -it partition<ID>_cli su admin " as sometimes for example to see the GUI logs entering the GUI container for the partition could be needed but F5 support will for this in most cases and maybe this will be the way to enter the BIG-IP NEXT CLI. Overview of VELOS system architecture: https://support.f5.com/csp/article/K73364432 Overview of rSeries system architecture: https://support.f5.com/csp/article/K49918625 rSeries tanant access: https://support.f5.com/csp/article/K33373310 Velos blade and tenant access: https://support.f5.com/csp/article/K65442484 Velos partition access: https://support.f5.com/csp/article/K11206563 3. Usefull commands and logs. For Velos/rSeries as this is a system with a cluster the "show cluster" command is usefull to see any issues (look fo "cluster is NOT ready."). Also the velos.log for the controller and partitions is a great place to start and debug level can be enabled for it under " SYSTEM SETTINGS Log Settings " as this is also the place for rSeries logging to be set to debug. Also the /var/log/openshift.log is good be checked with velos if there are cluster issues or or ks3.log in rSeries. Also the confd logs are like mcpd logs, so they are really usefull for Velos or rSeries. Other nice commands are docker ps, oc get pod --all-namespaces -o wide, kubectl get pod --all-namespaces -o wide but the support will ask for them in most cases. Velos cluster status: https://support.f5.com/csp/article/K27427444 Velos debug: https://support.f5.com/csp/article/K51486849 Velos openshift example issue: https://support.f5.com/csp/article/K01030619 Monitoring Velos: https://clouddocs.f5.com/training/community/velos-training/html/monitoring_velos.html Monitoring rSeries: https://clouddocs.f5.com/training/community/rseries-training/html/monitoring_rseries.html 4. Velos and rSeries tcpdumps packet captures, file utility and qkview files. For Velos qkviews ca be created for controller or partition as they are seperate qkviews. Tcpdumps for client traffic are done a tcpdump utility from the F5OS (su - admin) and a tcpdump in the Linux kernel is just for the managment ip addresses of the appliance , controller (floating or local) , partition or tenant. The file utility allows for file transfers to remote servers or even downloading any log from the Velos/rSeries to your computer as this was not possible before with iSeries or Viprion. Also the file utility starts outbound session to the remote servers so this an extra security as no inbound sessions need to be allowed on the firewall/web proxy and it can be even triggered by API call and I may make a codeshare article for this. Velos tcpdump utility: https://support.f5.com/csp/article/K12313135 rSeries tcpdump utility: https://support.f5.com/csp/article/K80685750 Qkview Velos: https://support.f5.com/csp/article/K02521182 Qkview Velos CLI location: https://support.f5.com/csp/article/K79603072 Qkview rSeries: https://support.f5.com/csp/article/K04756153 SCP: https://support.f5.com/csp/article/K34776373 For rSeries 2000/4000 tcpdump is different as SR-IOV not FPGA (rSeries Networking (f5.com)) is used to attach interfaces directly to the tenant VM: Article Detail (f5.com) 5. A final fast check could be to use ''kubectl get pods -o wide--all-namespaces'' (with Velos also ''oc get pods -o wide --all-namespaces'' should also work) to see that all pods are ok and running. Also ''docker ps'' or '' docker ps --format 'table {{.Names}}\t{{.RunningFor}}\t{{.Status}}' '' are usefull to see a container that could be going down and up and this can be correlated with issues seen with "show cluster" command. 6. The new F5OS has much better hardware diagnostics than the old devices, so no more the need to do EUD tests as all system hardware components and their health can be viewed from the GUI or CLI and also this is shown in F5 ihealth! https://techdocs.f5.com/en-us/velos-1-5-0/velos-systems-administration-configuration/title-system-settings.html 7. For Velos and rSeries always keep the software up to date as for example I will give with the Velos 1.5.1 the cluster rebuild because of the openshift ssl cert being 1 year is much simpler or the F5 rSeries and the Cisco Nexus issues or the corrupt Qkview generation when the GUI not the CLI is used (the velos cluster rebuild with touch /var/omd/CLUSTER_REINSTALL can solve many issues but it will cause some timeout): http://cdn.f5.com/product/bugtracker/ID1135853.html https://my.f5.com/manage/s/article/K000092905 https://support.f5.com/csp/article/K79603072 In the future ''docker'' commands could be not available but then just use "crictl" as this replaces the docker init system for kubernetes.ASM/WAF Management Automation - TMOS
This post is to go over some of my thoughts on ASM/WAF management, and some custom solutions I've made to make it easier and more accurate. This will be highly technical and will apply to most use-cases. Written based on TMOS v15 software, but will be mostly applicable to all recent TMOS versions. Automatic Learning Informational Automatic Learning will automatically accept most Learning Suggestions that hit 100% confidence Fully Automatic Learning will automatically accept all Learning Suggestions that hit 100% confidence If your application open to the public, it will likely get scanned a lot. Some of the Learning Suggestions will be to loosen your policies based on scanner traffic it sees, like this one that would've accepted a SQL injection attack suggestion from traffic that's clearly from a Veracode scanner Tips Reduce scanner traffic that hits ASM policy: Apply an iRule to your VIPs that have ASM policies that drops external IP-based requests. A lot of scanner traffic hits your VIPs by IP address instead of by hostname, and iRules process before ASM policies. So dropping IP-based traffic via iRule will drastically reduce the amount of bad ASM Learning Suggestions you get I use the attached iRule "irule_all_asm_VIPs" on all ASM VIPs. It has a line to increment the "triggered" field in a Statistics profile named "asm_irule_triggered," so you'll have to create the Statistics profile with the "triggered" field and attach it to the VIP before it will let you attach the iRule to the VIP. This makes sure you don't lose track of how many bad requests are hitting the device If ASM VIP has an existing iRule, irule_all_asm_VIPs must be attached first in order. See below for an automated way to attach this iRule to all ASM VIPs. Ongoing Maintenance Tasks: Enforce Ready Signatures ASM policies have a configured staging period (AKA Enforcement Readiness Period) for all new and modified Attack Signatures. Once this staging period is complete, they must be enforced manually Under Security > Application Security > Policy Building > Traffic Learning, there's an Enforcement Readiness Summary section where you can see ASM signatures that are Ready To Be Enforced. You can also Enforce Ready Entities on this page See here for a script I made that can be pushed from the Big-IQ to Enforce Ready Entities on all ASM Policies: https://community.f5.com/discussions/technicalforum/automate-asm-ready-to-be-enforced-attack-signatures/245055 If using Automatic Learning (not Fully Automatic Learning), periodically review Learning Suggestions at 100% confidence Manual Learning Informational Manual Learning doesn’t accept any Learning Suggestions automatically Without the automation explained in this post or a full WAF team, it can be hard to manage many policies manually Tips Reduce scanner traffic that hits ASM policy. <see above> Ongoing Maintenance Enforce Ready Signatures <see above> Periodically Review Learning Suggestions at 100% confidence <see below for automation> Big-IQ Script Automation Enforce Ready Entities <see above> https://community.f5.com/discussions/technicalforum/automate-asm-ready-to-be-enforced-attack-signatures/245055 Review Learning Suggestions Periodic reviews of Learning Suggestions can be tedious because there isn't a main overview page that tells you how many Learning Suggestions each policy has at 100% confidence. So without automation, you have to click through each policy one-by-one to see which policies need to be reviewed. I made the attached "Email Ready Learning Suggestions" script, that builds an HTML table line-by-line showing how many Learning Suggestions each policy has at 100% confidence, then uses the sendmail command to email it. There are 2 initial one-time setup commands at the top of the script you'll have to check first to make sure your F5 can send emails, and you'll have to replace email@example.com with your own email address in at the bottom. If it doesn't work after that, it's likely that your F5 SMTP traffic isn't allowed through a firewall Attach iRule and Statistics Profile to all ASM VIPs I made the attached "Attach iRule and Statistics Profile to all ASM VIPs" script that can be pushed from the Big-IQ to make a list of all VIPs with ASM policies, then attach the iRule if the VIP doesn't already have it. It doesn't remove existing iRules, and it puts this one first in the list if there are existing iRules Obviously for this to work, your F5 needs to already have the "irule_all_asm_VIPs" iRule and "asm_irule_triggered" Statistics profile created. The Statistics profile needs a field called "triggered" Cron Job Automation Reset Log Statistics Profile Count, and Reset I use these 3 commands in a daily cron job that runs at midnight to send a syslog entry with the number of times an IP-Based request got dropped by the iRule, then reset the counter EXECUTIONS=$(tmsh show ltm profile statistics asm_irule_triggered | grep triggered | tail -n1 | sed s/"triggered"//) logger -p local0.notice "Daily number of IP-based requests that hit ASM iRule: $EXECUTIONS" tmsh reset-stats ltm profile statistics asm_irule_triggered
GTM return LDNS IP to client
Problem this snippet solves: We do a lot of our load balancing based on topology rules, so it's often very useful to know where the DNS request is actually coming from rather than just the client's IP and the DNS servers they have configured. Especially if they're behind an ADSL router doing NAT or some other similar set up. This rule simply returns the IP address of the LDNS that eventually made the query to the GTM device in the response to a lookup for the WideIP using the rule, as well as logging the response and perceived location. Code : rule "DNS_debug" partition "Common" { when DNS_REQUEST { host [IP::client_addr] log local0.err "Debug address : [IP::client_addr] [whereis [IP::client_addr]]" } }
Create DNS monitor without external dig or nslookup
Problem this snippet solves: NOTE: This was written in 10.x and early 11.x days. Since then, a native DNS monitor has been added to the product. This example will create a udp monitor with proper send and receive strings to query a DNS server. Much of the heavy lifting was pulled from an iApp template that ships with v11. I have tested this out on as far back as 10.2.1 base. Installation One method to get this into your config would be: In the code block below, click the icon in the top right corner '<>' view source. Ctrl-A, Ctrl-C to select and copy all. Then in tmsh, type 'edit /cli script dns.monitor' If your editor is configured to use vi (the default setting) Type :set paste, then c, then Shift + G. This will clear out the default skeleton. You will now be in insert mode. Press Shift + insert. This will paste in the script. Now save the script by pressing Esc, then typing :wq < enter > and follow the prompts. How to use this snippet: Example root@bigip(Active)(/Common)(tmos)# run cli script dns.monitor xxx Requires 5 arguments: := Name you want to give the monitor := A, NS, PTR, SOA, CNAME := What you are asking the DNS server (send string) := Why you expect for a "healthy" response (recv string) := The frequency you want the monitor to fire e.g. dns.monitor my_dns A www.example.com 192.0.2.55 5 root@bigip(Active)(/Common)(tmos)# run cli script dns.monitor my_dns A www.example.com 192.0.2.55 5 root@bigip(Active)(/Common)(tmos)# list ltm monitor udp my_dns ltm monitor udp my_dns { debug no defaults-from udp destination *:* interval 5 recv \\x00\\x01.*\\x04\\xc0\\x00\\x02\\x37 send \\x96\\x87\\x01\\x00\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x03www\\x07example\\x03com\\x00\\x00\\x01\\x00\\x01 time-until-up 0 timeout 16 } root@bigip(Active)(/Common)(tmos)# Code : create script dns.monitor { proc script::init {} { } proc script::run {} { if {$tmsh::argc == 6} { # create monitor set monitor_name [lindex $tmsh::argv 1] set monitor_type [string toupper [ lindex $tmsh::argv 2 ]] switch $monitor_type { A { set monitor_send [generate_monitor_send_string_a_record [lindex $tmsh::argv 3]] set monitor_recv [generate_monitor_recv_string_a_record [lindex $tmsh::argv 4]] } NS { set monitor_send [generate_monitor_send_string_ns_record [lindex $tmsh::argv 3]] set monitor_recv [generate_monitor_recv_string_ns_record [lindex $tmsh::argv 4] [lindex $tmsh::argv 3]] } PTR { set monitor_send [generate_monitor_send_string_ptr_record [lindex $tmsh::argv 3]] set monitor_recv [generate_monitor_recv_string_ptr_record [lindex $tmsh::argv 4]] } SOA { set monitor_send [generate_monitor_send_string_soa_record [lindex $tmsh::argv 3]] set monitor_recv [generate_monitor_recv_string_soa_record [lindex $tmsh::argv 4] [lindex $tmsh::argv 3]] } CNAME { set monitor_send [generate_monitor_send_string_cname_record [lindex $tmsh::argv 3]] set monitor_recv [generate_monitor_recv_string_cname_record [lindex $tmsh::argv 4] [lindex $tmsh::argv 3]] } default { usage } } set monitor_interval [lindex $tmsh::argv 5] set monitor_timeout [ expr { [ expr {$monitor_interval * 3} ] + 1 } ] # doctor the send and receive strings so that they go through as intended set monitor_send [doctor_hex_string $monitor_send] set monitor_recv [doctor_hex_string $monitor_recv] tmsh_create "ltm monitor udp $monitor_name defaults-from udp interval $monitor_interval timeout $monitor_timeout send \"$monitor_send\" recv \"$monitor_recv\"" } else { usage } } proc script::help {} { tmsh::add_help "Create a DNS monitor \n\ " } proc script::tabc {} { } proc tmsh_create { arguments } { regsub -all {\[} $arguments "\\\[" arguments regsub -all {\]} $arguments "\\\]" arguments tmsh::create "$arguments" } proc number_to_hex_string { number } { if { $number < 0 } { set number [ expr {0 - $number} ] } set hex_string [format "\\x%02x" $number] return $hex_string } proc hostname_or_ip_address_to_dns_name { hostname } { set fields [split $hostname .] set dns_name "" foreach field $fields { set len [string length $field] set str_len [number_to_hex_string $len] append dns_name $str_len append dns_name $field } append dns_name "\\x00" return $dns_name } proc hostname_or_ip_address_to_reverse_dns_name { hostname } { set complete_hostname [format "arpa.in-addr.%s" $hostname] set fields [split $complete_hostname .] set dns_name "" foreach field $fields { # if [string is integer $field] { set len [string length $field] set str_len [number_to_hex_string $len] set dns_name [format "%s%s%s" $str_len $field $dns_name] # } } append dns_name "\\x00" return $dns_name } proc get_id { } { set value1 [expr { int(256 * rand()) }] set value2 [expr { int(256 * rand()) }] set id [number_to_hex_string $value1] append id [number_to_hex_string $value2] return $id } proc generate_dns_packet_payload { question } { set payload [get_id] append payload "\\x01\\x00" append payload "\\x00\\x01" append payload "\\x00\\x00" append payload "\\x00\\x00" append payload "\\x00\\x00" append payload $question return $payload } proc ip_addr_to_a_resp_addr { address } { set fields [split $address .] set num_fields 0 set field_encoding "" foreach field $fields { if {[string is integer $field]} { set num_fields [expr { $num_fields + 1 }] set part [number_to_hex_string $field] append field_encoding $part } } set dns_name [number_to_hex_string $num_fields] append dns_name $field_encoding return $dns_name } proc generate_send_payload { input_string record_type_code reverse } { if { $reverse == 0 } { set dns_name [hostname_or_ip_address_to_dns_name $input_string] } else { set dns_name [hostname_or_ip_address_to_reverse_dns_name $input_string] } set internet_address_qclass "\\x00\\x01" set question $dns_name append question $record_type_code append question $internet_address_qclass set send [generate_dns_packet_payload $question] return $send } proc generate_monitor_send_string_a_record { input_string } { set send [generate_send_payload $input_string "\\x00\\x01" 0] return $send } proc generate_monitor_send_string_ns_record { input_string } { set send [generate_send_payload $input_string "\\x00\\x02" 0] return $send } proc generate_monitor_send_string_ptr_record { input_string } { set send [generate_send_payload $input_string "\\x00\\x0c" 1] return $send } proc generate_monitor_send_string_soa_record { input_string } { set send [generate_send_payload $input_string "\\x00\\x06" 0] return $send } proc generate_monitor_send_string_cname_record { input_string } { set send [generate_send_payload $input_string "\\x00\\x05" 0] return $send } proc generate_monitor_recv_string_a_record { input_string } { set a_resp_addr [ip_addr_to_a_resp_addr $input_string] set recv [format "\\x00\\x01.*%s" $a_resp_addr] return $recv } proc generate_monitor_recv_string_ns_record { input_string domain_name } { regsub -all {\.} $domain_name \. domain_name set re [format "^(.*)(\.%s)$" $domain_name] set found [regexp $re $input_string matched new_hostname rest] if { $found != 0 } { set recv_temp [hostname_or_ip_address_to_dns_name $new_hostname] set re {^(.*\\)(x00)$} set found [regexp -nocase $re $recv_temp matched recv rest] if { $found != 0 } { set recv [format "\\x00\\x02.*%sxc0" $recv] } else { set recv [format "\\x00\\x02.*%s" $recv_temp] } } else { set recv [hostname_or_ip_address_to_dns_name $input_string] set recv [format "\\x00\\x02.*%s" $recv] } return $recv } proc generate_monitor_recv_string_ptr_record { input_string } { set dns_name [hostname_or_ip_address_to_dns_name $input_string] set recv [format "\\x00\\x0c.*%s" $dns_name] return $recv } proc generate_monitor_recv_string_soa_record { input_string domain_name } { regsub -all {\.} $domain_name \. domain_name set re [format "^(.*)(\.%s)$" $domain_name] set found [regexp -nocase $re $input_string matched new_hostname rest] if { $found != 0 } { set recv_temp [hostname_or_ip_address_to_dns_name $new_hostname] set re {^(.*)(\\x00)$} set found [regexp -nocase $re $recv_temp matched recv rest] if { $found != 0 } { set recv [format "\\x00\\x06.*%s(\\xc0|\\x05)" $recv] } else { set recv [format "\\x00\\x06.*%s" $recv_temp] } } else { set recv [hostname_or_ip_address_to_dns_name $input_string] set recv [format "\\x00\\x06.*%s" $recv] } return $recv } proc generate_monitor_recv_string_cname_record { input_string hostname } { regsub -all {\.} $hostname \. hostname #puts "hostname = $hostname" set re {^([^\.]+)\.(.+)$} #puts "re = $re" set found [regexp -nocase $re $hostname matched first domain_name] #puts "found = $found" if { $found == 0 } { set domain_name $hostname } regsub -all {\.} $domain_name \. domain_name #puts "domain_name = $domain_name" #puts "input_string = $input_string" set re [format "^(.*)(\.%s)$" $domain_name] #puts "re = $re" set found [regexp -nocase $re $input_string matched new_hostname rest] if { $found != 0 } { set recv_temp [hostname_or_ip_address_to_dns_name $new_hostname] set re {^(.*\\)(x00)$} set found [regexp -nocase $re $recv_temp matched recv rest] if { $found != 0 } { set re {^(.*)(\\x00)$} set found [regexp -nocase $re $recv_temp matched recv rest] set recv [format "\\x00\\x05.*%s(\\xc0|\\x05)" $recv] } else { set recv [format "\\x00\\x05.*%s" $recv_temp] } } else { set recv [hostname_or_ip_address_to_dns_name $input_string] set recv [format "\\x00\\x05.*%s" $recv] } return $recv } proc doctor_hex_string { input } { set pieces [split $input \\ ] set output "" set first_time 1 foreach piece $pieces { if { $first_time == 1 } { set first_time 0 } else { append output {\\} } append output $piece } return $output } proc usage { } { puts "Requires 5 arguments: \n\ := Name you want to give the monitor \n\ := A, NS, PTR, SOA, CNAME \n\ := What you are asking the DNS server (send string) \n\ := Why you expect for a \"healthy\" response (recv string) \n\ := The frequency you want the monitor to fire \n\n\ e.g. [lindex $tmsh::argv 0] A www.example.com 192.0.2.55 5" exit } }Generate private key w/ CSR via iControl REST
Problem this snippet solves: Generate a private key w/ CSR How to use this snippet: To create a private key with a CSR via iControl REST: POST URL:https://10.1.1.165/mgmt/tm/sys/crypto/key Use the data below as your payload. For the name field, it must end in .key or you will get a false 404! Code : { "name":"www.testing.com.key", "commonName":"www.testing.com", "keySize":"4096", "keyType":"rsa-private", "options":[{"gen-csr":"www.testing.com"}], "organization":"Let It Snow Corp.", "ou":"Ice Engineering", "city":"Calhoun", "state":"AZ", "admin-email-address":"jerry@letit.snow", "email-address":"beth@letit.snow", "subject-alternative-name":"DNS:www.testing.com", "challenge-password":"myP4ssword" } Tested this on version: 13.0
FQDN nodes in non-default route domains
Problem this snippet solves: Currently there is no support for FQDN nodes in non-default route domains as per Article. With recent increase in cloud deployment, most of the time there is a requirement from F5 to load balance to servers or ELB in the cloud which has FQDN names as those are having dyanamic IP addresses. If you are using route domains in your BIGIP environment then this becomes a challenge. Below iRule script can be used in those scenerios to use F5 to send traffic to FQDN nodes in non-default route domains. How to use this snippet: If FQDN needs to be resolved by your internal DNS, create performance layer 4 VIP (dns_53) load balancing your DNS servers. DNS server can be used directly in the iRule itself, but it would be better to use a VIP to have redundancy. Create iRule and apply to the VIP Code : when CLIENT_ACCEPTED { set host [RESOLV::lookup @dns_53 "server.example.com"] set ip [getfield $host " " 1] node [lindex $ip 0]%<Rd> 443 } Tested this on version: 14.1BIG-IP Report
Problem this snippet solves: Overview This is a script which will generate a report of the BIG-IP LTM configuration on all your load balancers making it easy to find information and get a comprehensive overview of virtual servers and pools connected to them. This information is used to relay information to NOC and developers to give them insight in where things are located and to be able to plan patching and deploys. I also use it myself as a quick way get information or gather data used as a foundation for RFC's, ie get a list of all external virtual servers without compression profiles. The script has been running on 13 pairs of load balancers, indexing over 1200 virtual servers for several years now and the report is widely used across the company and by many companies and governments across the world. It's easy to setup and use and only requires auditor (read-only) permissions on your devices. Demo/Preview Interactive demo http://loadbalancing.se/bigipreportdemo/ Screen shots The main report: The device overview: Certificate details: How to use this snippet: Installation instructions BigipReport REST This is the only branch we're updating since middle of 2020 and it supports 12.x and upwards (maybe even 11.6). Downloads (two latest versions): https://loadbalancing.se/downloads/bigipreport-v5.7.11.zip https://loadbalancing.se/downloads/bigipreport-v5.7.10.zip Documentation, installation instructions and troubleshooting:https://loadbalancing.se/bigipreport-rest/ Docker support https://loadbalancing.se/2021/01/05/running-bigipreport-on-docker/ Kubernetes support https://loadbalancing.se/2021/04/16/bigipreport-on-kubernetes/ BIG-IP Report (Legacy) Older version of the report that only runs on Windows and is depending on a Powershell plugin originally written by Joe Pruitt (F5) BIG-IP Report (only download this if you have v10 devices): https://loadbalancing.se/downloads/bigipreport-5.4.0-beta.zip iControl Snapin https://loadbalancing.se/downloads/f5-icontrol.zip Documentation and Installation Instructions https://loadbalancing.se/bigip-report/ Upgrade instructions Protect the report using APM and active directory Written by DevCentral member Shann_P: https://loadbalancing.se/2018/04/08/protecting-bigip-report-behind-an-apm-by-shannon-poole/ Got issues/problems/feedback? Still have issues? Drop a comment below. We usually reply quite fast. Any bugs found, issues detected or ideas contributed makes the report better for everyone, so it's always appreciated. --- Join us on Discord: https://discord.gg/7JJvPMYahA Code : BigIP Report Tested this on version: 12, 13, 14, 15, 16
BIG-IP Wide-IP to F5XC DNSLB converter
This is a conceptual sample script that converts BIG-IP Wide-IP records to F5XC DNSLB records. This bash script can be run using a cron job to check for configuration changes and synchronize them to F5XC. We used the F5XC API to post and update the configuration. You need to get an APIToken from your F5XC tenant and change the value on the POST commands on the script below. Note: Since this is not a full-blown converter script, it is limited to handling only a single Wide-IP pool member. You need to configure a GTM pool to include the IP addresses that need to be load balanced. Check the main article for more details. #!/bin/bash # Get list of wide IPs wideip_output=$(tmsh list gtm wideip all-properties one-line) # Get list of Pool pool_output=$(tmsh list gtm pool a one-line all-properties) # Declare associative arrays declare -A wideip_list declare -A current_wideip_info declare -A zone_array declare -A subdomain_info declare -A a_record_per_zone declare -A pool_list declare -A membersip_array # Unset variables function unset_arrays { unset current_wideip_info name subdomain domain type aliases description status failure_rcode last_resort_pool load_balancing_decision_log metadata minimal_response partition persist_cidr_ipv4 persist_cidr_ipv6 persistence pool_lb_mode pools pool_cname topology_edns0 ttl_persistence poolnames poolnames_array zone_array subdomain_info a_record_per_zone dnslb_name pool_list membersip_array } # Print wide IP details function print_wideip { for wideip in "${!wideip_list[@]}"; do echo "Wide IP: $wideip, Details: ${wideip_list[$wideip]}" done } # Create Zone function create_zone { curl -X POST -H "Authorization: APIToken XXXXX" -H "Accept: application/json" -H "Access-Control-Allow-Origin: *" -H "x-volterra-apigw-tenant: cag-waap2023" -H "Content-Type: application/json" -d "{\"metadata\":{\"name\":\"$zone\",\"namespace\":\"system\"},\"spec\":{\"primary\":{\"allow_http_lb_managed_records\":true},\"default_soa_parameters\":{},\"dnssec_mode\":{},\"rr_set_group\":[],\"soa_parameters\":{\"refresh\":3600,\"expire\":0,\"retry\":60,\"negative_ttl\":0,\"ttl\":0}}}" https://cag-waap2023.console.ves.volterra.io/api/config/dns/namespaces/system/dns_zones } # Create DNSLB function create_dnslb { curl -X POST -H "Authorization: APIToken XXXXX" -H "Accept: application/json" -H "Access-Control-Allow-Origin: *" -H "x-volterra-apigw-tenant: cag-waap2023" -H "Content-Type: application/json" -d "{\"metadata\":{\"name\":\"$dnslbname\",\"namespace\":\"system\",\"labels\":{},\"annotations\":{},\"disable\":false},\"spec\":{\"record_type\":\"A\",\"rule_list\":{\"rules\":[{\"geo_location_set\":{\"tenant\":\"cag-waap2023-gwjvytud\",\"namespace\":\"system\",\"name\":\"geo-1\",\"kind\":\"geo_location_set\"},\"pool\":{\"tenant\":\"cag-waap2023-gwjvytud\",\"namespace\":\"system\",\"name\":\"$xcdnslbpoolname\",\"kind\":\"dns_lb_pool\"},\"score\":100}]},\"response_cache\":{\"disable\":{}}}}" https://cag-waap2023.console.ves.volterra.io/api/config/dns/namespaces/system/dns_load_balancers } # Loop through each line of output while IFS= read -r line; do pool_name=$(awk '{print $4}' <<< "$line") dnslbpool_name=$(echo "$pool_name" | sed 's/[^a-zA-Z0-9]/-/g; s/.*/\L&/') pool_type=$(awk '{print $3}' <<< "$line") lbmode=$(grep -o 'load-balancing-mode [^ ]*' <<< "$line" | awk '{print $2}') # Convert load_balancing_mode to lowercase if it is "ROUND_ROBIN" if [[ "$lbmode" == "round-robin" ]]; then lbmode="ROUND_ROBIN" elif [[ "$lbmode" == "static-persistence" ]]; then lbmode="STATIC_PERSIST" elif [[ "$lbmode" == "global-availability" ]]; then lbmode="PRIORITY" elif [[ "$lbmode" == "ratio" ]]; then lbmode="RATIO_MEMBER" fi # Extract members block using awk #members=$(awk -F 'members {| }' '{print $2}' <<< "$line") members=$(echo "$line" | grep -o -P '(?<=members \{ ).*?(?=\} \})') membernames=$(echo "$members" | grep -oP '\S+(?=\s*{)') # Temporary array to hold member IP addresses declare -a temp_members_array temp_members_array=($(awk -F ':' '{print $2}' <<< "$membernames")) monitor=$(awk -F 'monitor ' '{print $2}' <<< "$line" | awk '{print $1}') ttl=$(awk '{print $2}' <<< "$(grep -o 'ttl [^ ]*' <<< "$line")") # Assign values to the associative array membersip_array["$dnslbpool_name"]="${temp_members_array[@]}" # Store extracted values in the array pool_list["$dnslbpool_name"]="pool_type: $pool_type, lbmode: $lbmode, monitor: $monitor, members: ${membersip_array["$dnslbpool_name"]}, ttl: $ttl" done <<< "$pool_output" # Loop through each pool in the pool_list for dnslbpool_name in "${!pool_list[@]}"; do # Extract only the TTL value from the string ttl=$(awk -F 'ttl: ' '{print $2}' <<< "${pool_list[$dnslbpool_name]}") lbmode=$(awk -F 'lbmode: ' '{print $2}' <<< "${pool_list[$dnslbpool_name]}" | awk -F ',' '{print $1}') members=$(awk -F 'members: ' '{print $2}' <<< "${pool_list[$dnslbpool_name]}" | awk -F ',' '{print $1}') pool_type=$(awk -F 'pool_type: ' '{print $2}' <<< "${pool_list[$dnslbpool_name]}" | awk -F ',' '{print $1}') # Check if pool_type is "a" if [[ "$pool_type" == "a" ]]; then # Initialize an empty string to store the JSON strings members_string="" # Loop through each record in the current zone for ip in ${membersip_array["$dnslbpool_name"]}; do # Create JSON string for each member and append to the existing string members_string+="{\"ip_endpoint\":\"$ip\",\"ratio\":10,\"priority\":1}," done # Remove the trailing comma from the JSON string members_string="${members_string%,}" # Create DNSLB Pools curl -X POST \ -H "Authorization: APIToken Rs0aGJm/lda/JmbE00c9lFXWw4I=" \ -H "Accept: application/json" \ -H "Access-Control-Allow-Origin: *" \ -H "x-volterra-apigw-tenant: cag-waap2023" \ -H "Content-Type: application/json" \ -d "{\"metadata\":{\"name\":\"$dnslbpool_name\",\"namespace\":\"system\"},\"spec\":{\"a_pool\":{\"members\":[$members_string],\"disable_health_check\":null,\"max_answers\":1},\"ttl\":\"$ttl\",\"load_balancing_mode\":\"$lbmode\"}}" \ "https://cag-waap2023.console.ves.volterra.io/api/config/dns/namespaces/system/dns_lb_pools" fi done # Unset variables to free up memory unset pool_list membersip_array # Loop through each line of output while IFS= read -r line; do # Extracting specific details using awk and sed based on the current line name=$(echo "$line" | awk '{print $4}') dnslb_name=$(echo "$name" | sed 's/\./-/g') subdomain=$(echo "$name" | cut -d'.' -f1) domain=$(echo "$name" | sed 's/^[^.]*\.//') type=$(echo "$line" | awk '{print $3}') aliases=$(echo "$line" | grep -o 'aliases [^}]*' | awk '{print $2}') description=$(echo "$line" | grep -o 'description [^ ]*' | sed 's/description //') status=$(echo "$line" | awk '{print $12}') failure_rcode=$(echo "$line" | grep -o 'failure-rcode [^ ]*' | sed 's/failure-rcode //') last_resort_pool=$(echo "$line" | grep -o 'last-resort-pool [^ ]*' | sed 's/last-resort-pool //') load_balancing_decision_log=$(echo "$line" | grep -o 'load-balancing-decision-log-verbosity [^ ]*' | sed 's/load-balancing-decision-log-verbosity //') metadata=$(echo "$line" | grep -o 'metadata [^ ]*' | sed 's/metadata //') minimal_response=$(echo "$line" | grep -o 'minimal-response [^ ]*' | sed 's/minimal-response //') partition=$(echo "$line" | grep -o 'partition [^ ]*' | sed 's/partition //') persist_cidr_ipv4=$(echo "$line" | grep -o 'persist-cidr-ipv4 [^ ]*' | sed 's/persist-cidr-ipv4 //') persist_cidr_ipv6=$(echo "$line" | grep -o 'persist-cidr-ipv6 [^ ]*' | sed 's/persist-cidr-ipv6 //') persistence=$(echo "$line" | grep -o ' persistence [^ ]*' | sed 's/persistence //') pool_lb_mode=$(echo "$line" | grep -o 'pool-lb-mode [^ ]*' | sed 's/pool-lb-mode //') pools=$(echo "$line" | grep -o -P '(?<=pools \{ ).*?(?=\} \})') pool_cname=$(echo "$line" | grep -o 'pools-cname [^ ]*' | sed 's/pools-cname //') topology_edns0=$(echo "$line" | grep -o 'topology-prefer-edns0-client-subnet [^ ]*' | sed 's/topology-prefer-edns0-client-subnet //') ttl_persistence=$(echo "$line" | grep -o 'ttl-persistence [^ ]*' | sed 's/ttl-persistence //') # Use grep to find strings before "{" poolnames=$(echo "$pools" | grep -oP '\S+(?=\s*{)' | sed 's/[^a-zA-Z0-9]/-/g; s/.*/\L&/') # Convert matches to an array readarray -t poolnames_array <<< "$poolnames" # Store extracted values in the associative array current_wideip_info=([Type]="$type" [Subdomain]="$subdomain" [Domain]="$domain" [Status]="$status" [DNSLB]="$dnslb_name" [Pools]="${poolnames_array[@]}" [Pool_LB_Mode]="$pool_lb_mode") # Assign wideip_info to wideip_list wideip_list["$name"]="${current_wideip_info[@]}" # Add subdomains to zone_array if [ -n "${zone_array[$domain]}" ]; then zone_array["$domain"]="${zone_array[$domain]},$subdomain" else zone_array["$domain"]=$subdomain fi # Store subdomain information in subdomain_info array subdomain_info["$subdomain"]="${current_wideip_info[@]}" # Store subdomain type "a" and add it to the array for that zone if [ "$type" == "a" ]; then a_record_per_zone[$domain]="${a_record_per_zone[$domain]}${a_record_per_zone[$domain]:+,}$subdomain" fi done <<< "$wideip_output" for zone in "${!zone_array[@]}"; do create_zone done # Loop through each domain in a_record_per_zone and echo its A record subdomains for domain in "${!a_record_per_zone[@]}"; do echo "Domain: $domain" echo "A Record Subdomains: ${a_record_per_zone[$domain]}" echo "--------------------------" # Initialize an empty string to store the JSON strings a_records_string="" # Loop through each record in the current zone for record in ${a_record_per_zone[$domain]//,/ }; do # Create JSON string for each A record and append to the existing string xcdnslbpoolname=$(echo ${wideip_list[$record.$domain]} | awk '{for (i=6; i<=(NF-1); i++) {printf "%s", $i; if (i < NF-1) printf " "}}') #echo "${a_record_per_zone[$domain]}" #echo "xcdnslbpoolname: $xcdnslbpoolname" # Check if xcdnslbpoolname has multiple strings if [[ $xcdnslbpoolname == *" "* ]]; then echo "Multiple strings found in xcdnslbpoolname" # Split xcdnslbpoolname into an array based on space IFS=' ' read -ra pool_names <<< "$xcdnslbpoolname" # Initialize an empty string to store the JSON strings pools_string="" # Loop through each pool name in the array for pool_name in "${pool_names[@]}"; do # Create JSON string for each member and append to the existing string pools_string+="{\"geo_location_set\":{\"tenant\":\"cag-waap2023-gwjvytud\",\"namespace\":\"system\",\"name\":\"geo-1\",\"kind\":\"geo_location_set\"},\"pool\":{\"tenant\":\"cag-waap2023-gwjvytud\",\"namespace\":\"system\",\"name\":\"$pool_name\",\"kind\":\"dns_lb_pool\"},\"score\":100}," done # Remove the trailing comma from the JSON string pools_string="${pools_string%,}" else pools_string="{\"geo_location_set\":{\"tenant\":\"cag-waap2023-gwjvytud\",\"namespace\":\"system\",\"name\":\"geo-1\",\"kind\":\"geo_location_set\"},\"pool\":{\"tenant\":\"cag-waap2023-gwjvytud\",\"namespace\":\"system\",\"name\":\"$xcdnslbpoolname\",\"kind\":\"dns_lb_pool\"},\"score\":100}" fi dnslbname=$(echo "dnslb-$record-$domain" | sed 's/\./-/g') #create_dnslb curl -X POST -H "Authorization: APIToken XXXXX" -H "Accept: application/json" -H "Access-Control-Allow-Origin: *" -H "x-volterra-apigw-tenant: cag-waap2023" -H "Content-Type: application/json" -d "{\"metadata\":{\"name\":\"$dnslbname\",\"namespace\":\"system\",\"labels\":{},\"annotations\":{},\"disable\":false},\"spec\":{\"record_type\":\"A\",\"rule_list\":{\"rules\":[$pools_string]},\"response_cache\":{\"disable\":{}}}}" https://cag-waap2023.console.ves.volterra.io/api/config/dns/namespaces/system/dns_load_balancers a_records_string+="{\"ttl\":3600,\"lb_record\": {\"name\":\"$record\",\"value\":{\"namespace\": \"system\",\"name\":\"$dnslbname\"}}}," done # Remove the trailing comma from the JSON string a_records_string="${a_records_string%,}" # Print the final JSON string echo "$a_records_string" #update zone record curl -X PUT -H "Authorization: APIToken XXXXX" -H "Accept: application/json" -H "Access-Control-Allow-Origin: *" -H "x-volterra-apigw-tenant: cag-waap2023" -H "Content-Type: application/json" -d "{\"metadata\":{\"name\":\"$domain\",\"namespace\":\"system\"},\"spec\":{\"primary\":{\"allow_http_lb_managed_records\":true,\"default_rr_set_group\":[$a_records_string],\"default_soa_parameters\":{},\"dnssec_mode\":{},\"rr_set_group\":[],\"soa_parameters\":{\"refresh\":3600,\"expire\":0,\"retry\":60,\"negative_ttl\":0,\"ttl\":0}}}}" https://cag-waap2023.console.ves.volterra.io/api/config/dns/namespaces/system/dns_zones/$domain done unset_arrays
