Advanced WAF IP Exceptions Manager
This solution introduces a dedicated Python-based GUI tool for managing IP whitelist exceptions in F5 BIG-IP Advanced WAF policies. The tool was developed to simplify a common operational workflow: searching WAF policies, adding IP or CIDR-based exceptions, checking whether an IP already exists across policies, deleting outdated exceptions, applying policy changes, and tracking actions through an activity log. The solution is especially useful in large environments where manual exception handling can create operational overhead, inconsistent policy configuration, or audit challenges. It demonstrates how focused automation can deliver practical value without requiring a large orchestration platform or full CI/CD pipeline.DGCat-Admin — F5 BIG-IP Datagroup & URL Category Manager
Available in two versions with identical functionality: Bash (dgcat-admin.sh) — For Linux, macOS, or directly on BIG-IP/Big-IQ PowerShell (dgcat-admin.ps1) — For Windows (PowerShell 5.1+) Why This Tool? SSL Orchestrator (SSLO) policies rely heavily on datagroups and URL categories for traffic classification. While you can add sites directly to SSLO policies, this approach has limitations: SSLO uses iAppLX to generate APM per-request policies under the hood Each host or site added directly becomes an expression in the APM policy Large lists could degrade policy performance and are not easily manageable The recommended approach: Use datagroups or URL categories for SSLO security policy rules. They're optimized for fast lookups, keep policies clean and are operationally easier to maintain. DGCat-Admin makes managing those lists very easy. Need to export a few massive datagroups or custom URL categories so you can precisely replicate existing SSLO business logic at another site in just minutes? Need to ingest a large number of subnets or hosts from an Excel spreadsheet into a datagroup for SSLO security policy use? Want to take a custom URL category and convert it to a datagroup? Want to take a datagroup and convert it to a custom URL category? This tool was designed specifically for those purposes. https://github.com/hauptem/F5-SSL-Orchestrator-Tools/tree/main/DGCat-AdminXC Distributed Cloud and how to keep the Source IP from changing with customer edges(CE)!
Code is community submitted, community supported, and recognized as ‘Use At Your Own Risk’. Old applications sometimes do not accept a different IP address to be used by the clients during the session/connection. How can make certain the IP stays the same for a client? The best will always be the application to stop tracking users based on something primitive as an ip address and sometimes the issue is in the Load Balancer or ADC after the XC RE and then if the persistence is based on source IP address on the ADC to be changed in case it is BIG-IP to Cookie or Universal or SSL session based if the Load Balancer is doing no decryption and it is just TCP/UDP layer . As an XC Regional Edge (RE) has many ip addresses it can connect to the origin servers adding a CE for the legacy apps is a good option to keep the source IP from changing for the same client HTTP requests during the session/transaction. Before going through this article I recommend reading the links below: F5 Distributed Cloud – CE High Availability Options: A Comparative Exploration | DevCentral F5 Distributed Cloud - Customer Edge | F5 Distributed Cloud Technical Knowledge Create Two Node HA Infrastructure for Load Balancing Using Virtual Sites with Customer Edges | F5 Distributed Cloud Technical Knowledge RE to CE cluster of 3 nodes The new SNAT prefix option under the origin pool allows no mater what CE connects to the origin pool the same IP address to be seen by the origin. Be careful as if you have more than a single IP with /32 then again the client may get each time different IP address. This may cause "inet port exhaustion " ( that is what it is called in F5BIG-IP) if there are too many connections to the origin server, so be careful as the SNAT option was added primary for that use case. There was an older option called "LB source IP persistence" but better not use it as it was not so optimized and clean as this one. RE to 2 CE nodes in a virtual site The same option with SNAT pool is not allowed for a virtual site made of 2 standalone CE. For this we can use the ring hash algorithm. Why this works? Well as Kayvan explained to me the hashing of the origin is taking into account the CE name, so the same origin under 2 different CE will get the same ring hash and the same source IP address will be send to the same CE to access the Origin Server. This will not work for a single 3-node CE cluster as it all 3 nodes have the same name. I have seen 503 errors when ring hash is enabled under the HTTP LB so enable it only under the XC route object and the attached origin pool to it! CE hosted HTTP LB with Advertise policy In XC with CE you can do do HA with 3-cluster CE that can be layer2 HA based on VRRP and arp or Layer 3 persistence based bgp that can work 3 node CE cluster or 2 CE in a virtual site and it's control options like weight, as prepend or local preference options at the router level. For the Layer 2 I will just mention that you need to allow 224.0.0.8 for the VRRP if you are migrating from BIG-IP HA and that XC selects 1 CE to hold active IP that is seen in the XC logs and at the moment the selection for some reason can't be controlled. if a CE can't reach the origin servers in the origin pool it should stop advertising the HTTP LB IP address through BGP. For those options Deploying F5 Distributed Cloud (XC) Services in Cisco ACI - Layer Three Attached Deployment is a great example as it shows ECMP BGP but with the BGP attributes you can easily select one CE to be active and processing connections, so that just one ip address is seen by the origin server. When a CE gets traffic by default it does prefer to send it to the origin as by default "Local Preferred" is enabled under the origin pool. In the clouds like AWS/Azure just a cloud native LB is added In front of the 3 CE cluster and the solution there is simple as to just modify the LB to have a persistence. Public Clouds do not support ARP, so forget about Layer 2 and play with the native LB that load balances between the CE 😉 CE on Public Cloud (AWS/Azure/GCP) When deploying on a public cloud the CE can be deployed in two ways one is through the XC GUI and adding the AWS credentials but this way you have not a big freedom to be honest as you can't deploy 2 CE and make a virtual site out of them and add cloud LB in-front of them as it always will be 3-CE cluster with preconfigured cloud LB that will use all 3 LB! Using the newer "clickops" method is much better https://docs.cloud.f5.com/docs-v2/multi-cloud-network-connect/how-to/site-management/deploy-site-aws-clickops or using terraform but with manual mode and aws as a provider (not XC/volterra as it is the same as the XC GUI deployment) https://docs.cloud.f5.com/docs-v2/multi-cloud-network-connect/how-to/site-management/deploy-aws-site-terraform This way you can make the Cloud LB to use just one CE or have some client Persistence or if traffic comes from RE to CE to implement the virtual site 2 CE node! There is no Layer 2 ARP support as I mentioned in public cloud with 3-node cluster but there is NAT policy https://docs.cloud.f5.com/docs-v2/multi-cloud-network-connect/how-tos/networking/nat-policies but I haven't tried it myself to comment on it. Hope you enjoyed this article!
Action On Log
Problem this snippet solves: Perl script that runs as a daemon to watch a log file (like tail in the background), and runs a command when a certain message is found. Smart enough to keep working when log files get rotated, etc. Very handy when you need to run a command to start gathering debugging data or recover from failure that is indicated by a log message. How to use this snippet: Note that when the script is first started it sends SIGHUP to syslogd (or syslog-ng if present) which causes the daemons to reload their configuration files and flush their log buffers. This is needed in case the message you want to find is currently being suppressed by "last message repeated XX times". The script is configured by editing the first 3 variables at the top of the script that specify the log file to watch via $LOG_FILE, log message to look for via $LOG_MESSAGE, and the action to take via $ACTION (i.e. command to run). The $LOG_MESSAGE to match is a regular expression matched against each new line in the log file like this: if (/$LOG_MESSAGE/) So if you specify a simple $LOG_MESSAGE like this: $LOG_MESSAGE = "test string"; This would match so long as any part of the log line contained "test string". So "foobar test string asdf" would still match "test string". There are 2 additional (optional) configuration parameters after the first 3 that specify how often to check if the log file has been rotated via $IDLE_TIMER, and how many lines to read when re-opening a log file that has been rotated via $MAX_ROTATED_LINES. The script can be tested like so: Determine which facility/severity messages are sent to /var/log/messages based on your syslog.conf or syslog-ng.conf. In my case, uucp.notice goes there. Start the script with default settings: perl action_on_log.pl The default action is to append a string to a file named "/var/tmp/mylog.txt", so first cat this file and make sure it's empty: cat /var/tmp/mylog.txt Now send the log messages the script is checking for: logger -p uucp.notice "foo watchdog bar" Repeat step #3 -- this time you'll see that the file exists and contains the expected message. You can stop the daemon like so: kill $(cat /var/run/action_on_log.pid) OS Support Older versions of this script have been tested against a fairly wide variety of unix-like platforms with great success, including BSDi 3.x, FreeBSD 4.x, Linux (2.4.x-era and current 2.6.x-era distros). Some have reported success running this script directly on BIG-IP 4.x and 9.x, though obviously F5 wouldn't support doing so. (Modify $LOG_FILE, $LOG_MESSAGE, and $ACTION as appropriate, and optionally $IDLE_TIMER, and $MAX_ROTATE_LINES) Code : #!/usr/bin/perl ## VERSION v0.5 use warnings; use strict; ################ # script settings ###### my $LOG_FILE = "/var/log/messages"; my $LOG_MESSAGE = "watchdog"; my $ACTION = "echo log message found >> /var/tmp/mylog.txt"; # misc my $IDLE_TIMER = 5; # if ! receiving log entries, how long before checking if log is rotated my $MAX_ROTATED_LINES = 500; # max lines to read from file we're re-reading because it's been rotated my $DEBUG = 0; # 0/1 #################################################### # END OF THINGS THAT SHOULD BE CONFIGURED #################################################### ######## # set defaults ### my($PID_FILE); $LOG_FILE ||= "/var/log/messages"; $LOG_MESSAGE ||= "FAILED"; $IDLE_TIMER ||= 5; $PID_FILE ||= "/var/run/action_on_log.pid"; $DEBUG ||= 0; ###### # misc global variable declaration ########## my($TAIL_CHILD_PID, $TAIL_OUTPUT_PIPE); my($CURRENT_SIZE, $CURRENT_INODE, $LAST_SIZE, $LAST_INODE); $CURRENT_SIZE = $CURRENT_INODE = 0; $LAST_SIZE = $LAST_INODE = -1; $|++; ########### # functions ####### # exit function that does does necessary child handling sub finish () { if (defined $TAIL_CHILD_PID && $TAIL_CHILD_PID != 0) { $DEBUG && print "INTERRUPT: sending SIGINT and SIGTERM to: $TAIL_CHILD_PID\n"; kill(2, $TAIL_CHILD_PID); sleep(1); kill(15, $TAIL_CHILD_PID); waitpid($TAIL_CHILD_PID, 0); } $DEBUG && print "INTERRUPT: unlink pidfile and exit\n"; unlink($PID_FILE); exit(0); } $SIG{INT} = sub { finish(); }; # simple sub to send SIGHUP to syslogd sub restart_syslogd () { if (-f "/var/run/syslog.pid") { open(PIDFILE, "; chomp; kill(1, ($_)); 1; } sub fork_to_background ($) { my $cmd = shift(); my $pid = fork(); if ($pid == 0) { exec($cmd) || die "exec() failed: $!\n"; } else { return($pid); } } sub popen_read ($) { my $cmd = shift(); my $child; $DEBUG && print "Background: \"$cmd\"\n"; pipe(READLOG, WRITELOG); select(READLOG); $|++; select(WRITELOG); $|++; select(STDOUT); ## dup STDOUT and STDERR open(T_STDOUT, ">&STDOUT"); open(T_STDERR, ">&STDERR"); ## redir STDOUT to pipe for child open(STDOUT, ">&WRITELOG"); open(STDERR, ">&WRITELOG"); $child = fork_to_background($cmd); ## close STDOUT and STDERR inherited by child close(STDOUT); close(STDERR); ## re-open STDOUT as normal for this process and close dup open(STDOUT, ">&T_STDOUT"); close(T_STDOUT); open(STDERR, ">&T_STDERR"); close(T_STDERR); return($child, \*READLOG); } sub open_log ($$) { my $LOG_FILE = shift(); my $lines = shift(); if (defined($TAIL_OUTPUT_PIPE) && defined(fileno($TAIL_OUTPUT_PIPE)) && $TAIL_CHILD_PID != 0) { $DEBUG && print "Killing tail child pid $TAIL_CHILD_PID before closing LOG\n"; kill(15, $TAIL_CHILD_PID); waitpid($TAIL_CHILD_PID, 0); $TAIL_CHILD_PID = 0; $DEBUG && print "Closing LOG\n"; close($TAIL_OUTPUT_PIPE); } if (! -f $LOG_FILE) { $DEBUG && print "Log file \"$LOG_FILE\" not found\n"; $TAIL_OUTPUT_PIPE = undef; return 0; } $DEBUG && print "Opening \"$LOG_FILE\"\n"; ($TAIL_CHILD_PID, $TAIL_OUTPUT_PIPE) = popen_read("tail -n $lines -f $LOG_FILE"); 1; } ## check to see if log is rotated, returns true if rotated sub is_rotated ($) { my $LOG_FILE = shift(); $DEBUG && print "enter is_rotated()\n"; ($CURRENT_INODE, $CURRENT_SIZE) = (stat($LOG_FILE))[1,7]; if (!defined $CURRENT_INODE || !defined $CURRENT_SIZE) { $CURRENT_INODE = 0; $CURRENT_SIZE = 0; } if (($LAST_SIZE > 0) && ($LAST_SIZE > $CURRENT_SIZE)) { $DEBUG && print "File is smaller. File must have been rotated\n"; ($LAST_INODE, $LAST_SIZE) = ($CURRENT_INODE, $CURRENT_SIZE); open_log($LOG_FILE, $MAX_ROTATED_LINES); return(1); } elsif (($LAST_INODE > 0) && ($LAST_INODE != $CURRENT_INODE)) { $DEBUG && print "Inode changed. File must have been rotated\n"; ($LAST_INODE, $LAST_SIZE) = ($CURRENT_INODE, $CURRENT_SIZE); open_log($LOG_FILE, $MAX_ROTATED_LINES); return(1); } elsif ($LAST_INODE == 0 && $LAST_SIZE == 0) { $DEBUG && print "Size and inode were 0 last time. File must have been removed\n"; ($LAST_INODE, $LAST_SIZE) = ($CURRENT_INODE, $CURRENT_SIZE); open_log($LOG_FILE, $MAX_ROTATED_LINES); return(1); } ($LAST_INODE, $LAST_SIZE) = ($CURRENT_INODE, $CURRENT_SIZE); 0; } ########### # MAIN ######## ####### DAEMONIZE ############# chdir("/"); exit unless (fork() == 0); # kill old self, write pid file if (-f $PID_FILE) { open(PIDFILE, "<$PID_FILE"); kill(15, ); close(PIDFILE); } open(PIDFILE, ">$PID_FILE"); syswrite(PIDFILE, $$); close(PIDFILE); ## set handler for SIGTERM (regular kill) $SIG{TERM} = sub { finish(); }; ################ # watch log-based trigger #################### print "\nWaiting for \"$LOG_MESSAGE\" in \"$LOG_FILE\". Will run:\n\"$ACTION\"\n"; # creates global $TAIL_OUTPUT_PIPE filehandle of $LOG_FILE open_log($LOG_FILE, 0); # flush syslogd's buffers (avoid never getting the message due to "last message repeated....") restart_syslogd(); LOOP: # tail -f the log and wait for message while (1) { if (!defined($TAIL_OUTPUT_PIPE) || !defined(fileno($TAIL_OUTPUT_PIPE)) ) { # this suggests the file was removed while we were using it... sleep($IDLE_TIMER); is_rotated($LOG_FILE); next; } eval { $SIG{ALRM} = sub { die("ALRM\n"); }; alarm($IDLE_TIMER); $_ = readline($TAIL_OUTPUT_PIPE); alarm(0); }; if ($@) { # this only occurs if we're idle for $IDLE_TIMER seconds because no new log entries are occuring $@ = undef; is_rotated($LOG_FILE); next; } chop(); $DEBUG && print "in LOG reading loop, current line: \"$_\"\n"; if (/$LOG_MESSAGE/) { $DEBUG && print "Current line matches: \"$LOG_MESSAGE\"\n"; last; } $DEBUG && print "no match, next\n"; } ## log message found: DO SOMETHING system($ACTION); goto LOOP; # kill all children; doesn't return finish(); 0;
Syslog NG Email Configuration
Problem this snippet solves: We describe here how to setup syslog-ng to send syslog messages via email. There is an AskF5 solution - SOL3667: Configuring SNMP trap alerts to send email notifications - which describes how to configure messages to be emailed via alertd and which is the only supported way. This procedure provides full control over the form of the email and the messages being sent, but is not supported. A supported alternative is to configure syslog-ng to forward messages to another server where you could apply any customizations you need, including what we describe here. How to use this snippet: Simply copy the configuration script source below to a text editor, edit to fit your environment, then install the configuration following the procedure detailed in this article: LTM 9.4.2+: Custom Syslog Configuration Please note the following: (Enable and disable by removing or inserting hash/comment marks at the beginning of the line.) 5.With a Postfix destination enabled, the Postfix server process will accept the mail and inject into the queue even if the Postfix system is unable to send mail. Next time Postfix is enabled, all queued mail will be delivered. If you've been testing without Postfix running, you might want to take advantage of these commands to manage the queue before bringing the service up: To inspect the Postfix queue use the command: postqueu –p To forcibly empty the queue: postsuper -dALL How it works When syslog-ng first parses the production version of this configuration, it starts the program specified in the d_email destination stanza (/usr/sbin/sendmail –bs), which is the Postfix SMTP server running in standalone daemon mode. When a message arrives that matches the filter, syslog-ng will use the template to compose the message and feed it via SMTP to Postfix. You have full access to every aspect of the email via the template. You can set the subject, the body, add any extra headers you wish, such as X-headers. You can insert any of the syslog-ng defined macros anywhere you like. For more information, please see the The syslog-ng Administrator Guide If for whatever reason the SMTP server process exits, syslog-ng will restart it, but only one copy is ever executed concurrently. Code : syslog include " template t_smtp { template_escape(no); template(\"NOOP HELO localhost MAIL From: RCPT To: DATA Date: $R_STAMP From: To: Subject: $S_DATE [$FACILITY:$PRIORITY] $MSG . \"); }; filter f_test { match(\" \") ; }; # destination #1 - to logfile for initial stanza development # Disable in production destination d_logfile { file(\"/var/log/syslog-email-config.log\" ); }; # destination #2 - to email with postfix logging for debugging mail transport issues # Disable in production #destination d_logfile_email { # program(\"/usr/sbin/sendmail -bs > /var/log/syslog-ng-sendmail.log 2>&1\" # ts_format(\"rfc3339\") # template(t_smtp) # ); #}; # destination #3 - to email for production # Enable only after validating filters and mail transport are operating as expected #destination d_email { # program(\"/usr/sbin/sendmail -bs\" ts_format(\"rfc3339\") template(t_smtp)); #}; log { source(local);source(s_bcm56xxd);source(s_tomcat4);source(s_tmm); destination(d_logfile); # destination(d_logfile_email); # destination(d_email); filter(f_test); }; "Create Pools/Vips with tmsh via MS-Excel script
Problem this snippet solves: This spreadsheet Excel is useful to create VIPs, pools, nodes, policy, etc. It is also included script for ASM. The AFM script, I ended up not using and therefore I have not tested very well. I decided to share because it might help someone to gain some time to configure the BigIP. How to use this snippet: Fill the information and then get a script. So, connect by ssh and in the tmsh, paste the script (preferably in blocks, slowly for reasons of buffer length), then, if we have success, give us some time to do other steps...
BIG-IP Backup Script In Bash
Problem this snippet solves: A script that can backup F5 configurations, runs daily, and FTP's the backup to a defined remote server. You will need to change the ftphost, user and password variables to reflect the FTP server you are connecting to as well as change the directory where the script has "cd /F5/JAX1" to reflect the clients directory structure. Cron job setup To ensure a daily backup, save the script source below in /etc/cron.daily: Code : #!/bin/bash # set the date variable today=$(date +'%Y%m%d') ftphost="ADD FTP IP HERE" user="ADD FTP USERNAME HERE" password="ADD FTP PASSWORD HERE" #run the F5 bigpipe config builder cd / bigpipe config save /config.ucs #Rename the config.ucs and append the date to the end NUM=0 until [ "$NUM" -eq 5 ] do if [ -f /config.ucs ] then mv config.ucs config-$today.ucs ; break else sleep 5 fi NUM=`expr "$NUM" + 1` done [[ ! -f /config-$today.ucs ]] && exit 1 #Open the FTP connection and move the file ftp -in <BIG-IP Multi Host CPU Monitor
Problem this snippet solves: This script polls the BIG-IP via SNMP and displays the current CPU utilization for each processor and the global utilization for all processors. How to use this snippet: Install script on remote system with appropriate permissions. Make sure Perl and the Net::SNMP perl module are installed. Code : #!/usr/bin/perl -w # This code is not supported by F5 Network and is offered under the GNU General Public License. Use at your own risk. use strict; use Net::SNMP qw(:snmp); my ($host, $snmp_comm); $host = $ARGV[0]; $snmp_comm = $ARGV[1]; chomp ($host , $snmp_comm); my $cpuIndex = ".1.3.6.1.4.1.3375.2.1.7.5.2.1.2.1.48"; my $cpuUsageRatio5s = ".1.3.6.1.4.1.3375.2.1.7.5.2.1.19.1.48."; my $cpuUsageRatio1m = ".1.3.6.1.4.1.3375.2.1.7.5.2.1.27.1.48."; my $cpuUsageRatio5m = ".1.3.6.1.4.1.3375.2.1.7.5.2.1.35.1.48."; my $gcpuUsageRatio5s = ".1.3.6.1.4.1.3375.2.1.1.2.20.21.0"; my $gcpuUsageRatio1m = ".1.3.6.1.4.1.3375.2.1.1.2.20.29.0"; my $gcpuUsageRatio5m = ".1.3.6.1.4.1.3375.2.1.1.2.20.37.0"; my ($session, $error) = Net::SNMP->session( -hostname => $host, -community => $snmp_comm, -port => 161, -version => 'snmpv2c', -nonblocking => 0 ); if (!defined $session) { print "Received no SNMP response from $host\n"; print STDERR "Error: $error\n"; exit -1; } my $allCPU = $session->get_table ( -baseoid => $cpuIndex ); my %cpu_table = %{$allCPU}; my $x = 0; print "\n\nCPU Utilization:\t5s\t1m\t5m\n\n"; foreach my $key (sort keys %cpu_table) { my @oid_index = split(/\./, $key); my $ltm_cpu5s = $cpuUsageRatio5s . $oid_index[-1]; my $ltm_cpu1m = $cpuUsageRatio1m . $oid_index[-1]; my $ltm_cpu5m = $cpuUsageRatio5m . $oid_index[-1]; my $oid_ratios = $session->get_request( -varbindlist => [$ltm_cpu5s, $ltm_cpu1m, $ltm_cpu5m] ); print "\tCPU$x\t\t$oid_ratios->{$ltm_cpu5s}\t$oid_ratios->{$ltm_cpu1m}\t$oid_ratios->{$ltm_cpu5m}\n"; $x++; } my $oid_gratios = $session->get_request( -varbindlist => [$gcpuUsageRatio5s, $gcpuUsageRatio1m, $gcpuUsageRatio5m] ); print "\tGlobal\t\t$oid_gratios->{$gcpuUsageRatio5s}\t$oid_gratios->{$gcpuUsageRatio1m}\t$oid_gratios->{$gcpuUsageRatio5m}\n\n\n";
ASM All Policy Export
Problem this snippet solves: ASM all policy export v1.4 - 15 Aug 2007 Writes out the active policy for each ASM web application to a gzip'd tar archive (see usage subrouting for details) Tested on BIG-IP ASM versions: 9.2.4, 9.4.0, 9.4.1 Aaron Hooley - LODOGA Security Limited (hooleylists at gmail dot com) Code : #!/usr/bin/perl # # ASM policy export # v1.4 - 15 Aug 2007 # # Writes out the active policy for each ASM web application to a gzip'd tar archive # (see usage subrouting for details) # # Tested on BIG-IP ASM versions: 9.2.4, 9.4.0, 9.4.1 # # Aaron Hooley - LODOGA Security Limited ([email protected]) # use strict; use warnings; use DBI; use lib '/ts/packages/'; use GenUtils qw//; use POSIX qw/strftime/; use File::Path; # pass_through: all unknown options are left in @ARGV use Getopt::Long qw(:config pass_through); use File::Basename; $SIG{PIPE} = 'IGNORE'; use constant DEBUG => 1; # 1 = log debug messages to standard out; 0 = don't use debug use constant OUTPUT_DIR => '/shared/asm_all_policy_export/output/'; # Default output directory for finalized compressed tar archive containing each output file # options below these shouldn't need to be modified use constant UCS_DIR => '/var/local/ucs/'; # If writing output file as a UCS for access to files via the GUI, use the UCS directory use constant TMP_DIR => '/shared/tmp/'; # Temporary directory to write the individual policies to use constant VERSION_FILE => '/VERSION'; # File which contains BIG-IP version info use constant LOCK_FILE => '/ts/lock/asm_all_policy_export.lock'; # Lock file used to ensure only one copy of the script is executing at a time use constant USERS_CFG_FILE => '/ts/common/users.cfg'; # System file containing user and database details use constant VERSION_FILE => '/VERSION'; # System file which contains BIG-IP version info use constant DRIVER => GenUtils::cfg_get_config_item(USERS_CFG_FILE, 'DATABASE', 'Driver'); use constant DCC => GenUtils::cfg_get_config_item(USERS_CFG_FILE, 'DATABASE', 'Name'); use constant PLC => GenUtils::cfg_get_config_item(USERS_CFG_FILE, 'POLICY_DATABASE', 'Name'); use constant DB_USER => GenUtils::cfg_get_config_item(USERS_CFG_FILE, 'DATABASE', 'User'); use constant F5_EXPORT_SCRIPT => '/ts/tools/import_export_policy.pl'; # Stop this script if ASM is not licensed check_license(); # Create a lock so only one instance of script runs at a time my ($lock_sub,$unlock_sub,$cleanup_lock_sub) = GenUtils::lock_factory(LOCK_FILE) or fatal_error("Cannot open lockfile '".LOCK_FILE."': $!"); $lock_sub->(); # Stop the script if the F5 single policy export script doesn't exist if (not (-e F5_EXPORT_SCRIPT)){ fatal_error(qq{Cannot access F5 export script: }.F5_EXPORT_SCRIPT."\n\n"); } # Get command line options if the script was called correctly # Return variables (and their default values) are: debug ('not_set'), output_path ('not_set'), output_file ('not_set'), use_ucs_dir (0) # As the handle_commandline_arguments subroutine handles the help flag itself, we don't need to consider it my %args = handle_commandline_arguments(); # Set the debug level to the higher level of the constant and command line interface (CLI) option. See usage for debug levels. my $debug = $args{debug}; if ($debug eq 'not_set') { $debug = DEBUG; } print "Verbose debug enabled\n" if $debug > 1; # Get MySQL password (and verify the BIG-IP version is supported by this script) my $mysql_pass = get_mysql_password(); # Verify required config items were read successfully if (not defined DRIVER or not defined DCC or not defined PLC or not defined DB_USER or not defined $mysql_pass){ fatal_error(qq{Problems accessing database users configuration file\n\n}); } # Get current timestamp for output filename, formatted as %Y-%m-%d_%H-%M-%S my $date = get_date(); # Set output directory according to CLI option or default from constant. # If use_ucs_dir option is enabled, ignore any other setting for the path of the output and use UCS_DIR constant value my $use_ucs_dir = 0; my $output_path; # Check if the CLI option for using the UCS dir is set if ($args{use_ucs_dir}==1){ # It is; so we're writing out the output file to the UCS directory $use_ucs_dir = 1; $output_path = UCS_DIR; } elsif ($args{output_path} ne 'not_set' and length($args{output_path}) > 1){ # UCS option wasn't set, and another output path was set via CLI option $output_path= $args{output_path}; } else { # UCS option wasn't set and no output path was specified in CLI option, so use the default path from the constant $output_path = OUTPUT_DIR; } # Add a trailing slash to the output path if it's not there already if (substr($output_path,-1,1) ne "/") { $output_path .= "/"; } # Create the output path if it doesn't exist already (skip this for the UCS path as it has to already exist) if (not $use_ucs_dir){ make_output_path($output_path); } # Set the output fully qualified filename according to CLI if set, else default to the constant value my $output_file; if ($args{output_file} eq 'not_set' or length($args{output_file})==0){ # Output file wasn't specified in CLI option, so use the default $output_file = build_output_filename_with_path($output_path, $date); } else { # An output file was specified in the CLI option so use that for the filename $output_file = $output_path.$args{output_file}; } # Append .ucs to the filename if using the UCS path for output, so the GUI will present it in the archive list. # Check that the filename doesn't already end in .ucs if ($use_ucs_dir==1 and not ($output_file =~ /.*?\.ucs$/i)){ $output_file .= ".ucs"; } # Check if mysql is up. If not, start it my $restore_mysql_to_prior_state = check_mysql(); # Run the export subroutine export(); # Stop mysql if it wasn't running before we started $restore_mysql_to_prior_state->(); # Release the lock $unlock_sub->(); $cleanup_lock_sub->(); exit; # # Main subroutine which exports the active policies and writes the output to a compressed tar archive # sub export { # Initialize an array to store the full path/filename for the individual exported policies my @exported_filenames; # SQL query to retrieve the names of each web app, the active policy name and the active policy ID my $sql_statement = qq{ SELECT a.domain_name AS 'Web App', p.name AS 'Policy', p.id AS 'POLICY.ID' FROM DCC.ACCOUNTS a INNER JOIN PLC.PL_POLICIES p ON a.active_policy_id = p.id; }; # Connect to mysql database my $dbh = connect_to_db($mysql_pass); # Prepare the SQL statement my $sth = $dbh->prepare($sql_statement) or fatal_error(qq{Problems preparing sql: |$sql_statement|: $DBI::errstr\n\n}); # Execute the SQL statement $sth->execute() or fatal_error(qq{Problems executing sql: |$sql_statement|: $DBI::errstr\n\n}); print $sth->rows()." active policies in resultset\n" if $debug > 1; # Loop through each web app and export the active policy using the F5 export policy script while (my @row = $sth->fetchrow_array()) { # Parse the three column values for this record my ($webapp, $policy_name, $policy_id) = @row; print "\$webapp = $webapp; \$policy_name = $policy_name, \$policy_id = $policy_id\n" if $debug > 1; # Build file name for a single policy my $single_policy = $webapp.'_'.$policy_name.'_'.$date.'.plc'; # Add current single policy file to the array of files push (@exported_filenames, $single_policy); # Call the F5 supplied policy export command: # /ts/tools/import_export_policy.pl -a ACTION -p POLICY_ID -f OUTPUT_FILE # example: /ts/tools/import_export_policy.pl -a export -p 1 -f /shared/tmp/policy_export/my-web-app_policy-name_2007-08-01_12-00-00.plc my @export_cmd = ( 'nice -n 19', F5_EXPORT_SCRIPT, '-a export', '-p', $policy_id, '-f', TMP_DIR.$single_policy); print "@export_cmd \n" if $debug > 1; # Run the export command for a single policy my $out = `@export_cmd`; print "policy export result: $out\n" if $debug > 1; if (not ($out =~ "successfully")){ fatal_error( qq/Failed to export policy using '@export_cmd'\n\n/ ); } } # tar up the files (uses 'nice -n 19' to lower the CPU priority for the tar'ing) my @tar_cmd = ( 'nice -n 19 tar', '-z', '-c', '-f', $output_file, '-C', TMP_DIR, @exported_filenames); print @tar_cmd if $debug > 1; system("@tar_cmd") == 0 or fatal_error( qq/Failed to create tar using '@tar_cmd'/ ); # Clean up temp files cleanup(@exported_filenames); # Report success with the output file name print "Exported ".$sth->rows()." policies to " . $output_file . "\n" if $debug > 0; # Log a successful message to syslog GenUtils::ts_syslog( 'info', 'All active ASM policies exported to %s', $output_file ); } # # Other subroutines # sub get_mysql_password { # Verify BIG-IP version is supported and get MySQL password based on version my $bigip_version = get_bigip_version(); # Initialize a variable to store the MySQL password. MySQL password is different between (9.2.x/9.3.x) and 9.4.x+ my $mysql_pass; # If version is 9.4.x use F5 function get_mysql_password if (substr($bigip_version,0,3) eq "9.4"){ $mysql_pass = GenUtils::get_mysql_password(); } elsif (substr($bigip_version,0,3) eq "9.2" or substr($bigip_version,0,3) eq "9.3"){ # For 9.2.x and 9.3.x, get it from the users.cfg file $mysql_pass = GenUtils::cfg_get_config_item(USERS_CFG_FILE, 'DATABASE', 'Password'); } else { # Parsed version was unknown, so kill the script fatal_error(qq{Version $bigip_version unknown\n\n}); } print "\$mysql_pass: $mysql_pass\n" if $debug > 1; return $mysql_pass; } sub build_output_filename_with_path { # Build the path/filename for the output file. Expect the path and date as parameters. my ($output_path, $date) = @_; # Get BIG-IP hostname my $bigip_hostname = get_bigip_hostname(); # Build name of output tar ball with fully qualified path return ($output_path . $bigip_hostname."_policy-export_$date.tgz"); } sub get_bigip_hostname { # Get the BIG-IP hostname my $hostname = `hostname`; if (not defined $hostname){ fatal_error(qq{Cannot get BIG-IP hostname using hostname command}); } else { chomp($hostname); return $hostname; } } sub get_bigip_version { # Get the BIG-IP version to determine how to get MySQL root password my $version; # Check if version file exists if (-f VERSION_FILE){ # Open the version file or stop script open (my $fh, '<', VERSION_FILE) or fatal_error(qq{get_bigip_version: cannot open version file }.VERSION_FILE); # Read in the contents of the version file while (<$fh>){ # Do case insensitive search for "version: xxx" and get the match from a backreference ($1) if ($_ =~ /^version:\s+(\S+)$/i){ $version = $1; print "get_bigip_version: \$version: $version\n" if $debug > 1; } } if (not $version){ # Kill the script as we couldn't parse the version fatal_error(qq{get_bigip_version: Could not parse version from: }.VERSION_FILE."\n"); } close $fh; } else { fatal_error(qq{get_bigip_version: version file does not exist: }.VERSION_FILE); } return ("$version"); } sub cleanup { # Clean up temp files used during policy export print "Cleaning up\n" if $debug > 1; my (@files_to_cleanup) = @_; foreach my $file (@files_to_cleanup) { if ( -e TMP_DIR.$file ) { print "Deleting temp file: " . TMP_DIR . $file . "\n" if $debug > 1; unlink TMP_DIR.$file or warn qq/Failed to remove file '/. TMP_DIR . $file . qq/': $!/; } } } sub check_mysql { # Is mysql running? my $mysql_was_up = GenUtils::check_mysql(); return sub { GenUtils::stop_mysql() unless $mysql_was_up; }; } sub connect_to_db { # Open database connection #my $mysql_pass = @_; my $database = 'DCC'; my $dbh = DBI->connect( "DBI:".DRIVER.":".$database, DB_USER, $mysql_pass, { PrintError => 0, RaiseError => 0, FetchHashKeyName => 'NAME_lc', } ) or fatal_error( "Cannot connect to $database: $DBI::errstr\n" ); return $dbh; } sub check_license { # Kill script if ASM isn't licensed if ( not GenUtils::is_ts_licensed() ) { GenUtils::ts_syslog( 'info', 'ASM support is not enabled' ); fatal_error(qq{ASM is not licensed. Aborting script.\n}); } } sub get_date { # Get date in format year-month-day_hour-minute-second return POSIX::strftime( "%Y-%m-%d_%H-%M-%S", localtime ); } sub make_output_path { # Create the output path if it doesn't exist already my ($output_path) = @_; if (not (-d $output_path)){ eval { mkpath($output_path) }; if ($@) { fatal_error(qq{Could not create path $output_path: $@\n}); } else { print "make_output_path: made path $output_path" if $debug > 1; } } } sub fatal_error { # Write error details to syslog and exit script my ($msg,%args) = @_; GenUtils::ts_syslog('err', "%s", $msg); print $msg; exit(1); } sub handle_commandline_arguments { # Set some default values for the options my %arguments = (); $arguments{debug} = 'not_set'; $arguments{output_path} = 'not_set'; $arguments{output_file} = 'not_set'; $arguments{use_ucs_dir} = 0; $arguments{help} = 0; # Parse the options from the command line my $options = GetOptions ( "debug|d:i" => \$arguments{debug}, # match for debug or d as an integer "output_path|p:s" => \$arguments{output_path}, # match on output_path or p as a string "output_filename|f:s" => \$arguments{output_file}, # match on output_file or f as a string "use_ucs_directory|u" => \$arguments{use_ucs_dir}, # match on use_ucs_directory or u, binary type "help|h" => \$arguments{help}, # match on help or h, binary type ); # Debug printing of options if ($arguments{debug} ne 'not_set' and $arguments{debug} > 1){ while (my ($key, $value) = each(%arguments)){ print $key.": ".$value."\n"; } if (@ARGV){ print "extra args found: \n"; print "@ARGV\n"; } } # print the usage info if the options couldn't be parsed, help option was enabled, unknown options were used, or the output_path doesn't start with a / if ($arguments{help}){ print "Help option set\n".usage(); } elsif (@ARGV) { print "Unparsed options set: \n"; print "@ARGV\n"; print usage(); } elsif ($arguments{output_path} ne "not_set" and substr($arguments{output_path},0,1) ne "/"){ print "output_file value must start with a forward slash\n".usage(); } else { return %arguments; } } sub usage { # stop the script if this sub is called, as we're just printing the options for calling the script # get script name my ($script_name, $gar, $bage) = File::Basename::fileparse($0); die (qq{ -------------------------------------------------------------- NAME $script_name - ASM all policy export Write out each active policy to a single compressed tar archive DESCRIPTION $script_name - Write out the policy that is marked as active for each ASM web application. Each policy is exported from the internal database and written to a combined gzip compressed tar archive. The output path and filename can be configured using command line options or in constants defined in the script. OPTIONS -d --debug 0 - print nothing to standard output 1 - print output filenames to standard output 2 - print verbose logging to standard ouput Note: values greater than 2 are considered as 2 Default: 1 -p --output_path String containing the directory to write the output file. If the path does not exist it will be created. Must start with a leading forward slash. Note: this option is ignored if the -u/--use_ucs_dir option is enabled Default: }.OUTPUT_DIR.qq{ -f --output_filename String containing the name of the output file. Note: if the -u/--use_ucs_dir option is enabled, the file extension ".ucs" will be appended to the filename if not already present. Default: BIGIP_HOSTNAME_policy-export_DATE.tgz -u --use_ucs_directory If enabled, the output file will be written to the UCS directory with a file extension of ".ucs". This allows access to the output file via the BIG-IP admin GUI under System >> Archives. Note: use of this option overrides the use of the -f/--output_dir option Default: Disabled -h --help Print this help menu EXAMPLES $script_name -d 2 -p /path/to/ -f output.tgz Use debug level 2, write output to the specified path and file $script_name --debug 2 --ucs --output_filename asm_policy_export.tgz.ucs Use debug level 2, write output archive to the UCS directory with the specified filename. -------------------------------------------------------------- }); }
