BigIP UCS Backup script; looking for some guidance on design
Greetings, I've began to work on a bash script, intended to be ran locally on each F5 appliance via a cron task. The criteria for this script has been, Saves the UCS /w encryption using {Hostname}-YYYY-MM-DD.ucs naming format. Uploads the generated UCS file to a SFTP server SFTP native commands are a MUST, SCP will not work due to it's reliance on command shell/login. Rollover after X # of saved files in order to prevent storage exhaustion on the target SFTP Server I strongly doubt any form of deduplication will work with a encrypted UCS Sends an email notification if the backup failed I've so far written a script that addresses the first 3 criteria and have been waiting for those to go through their paces in testing before adding in notification logic. The commands and logic being used have gotten more complex, the further I've gotten into the script's development. This has lead to some concerns about whether this is the best approach given the nature of the F5 BigIP systems being a vendor appliance and worry that there's a large possibility commands may stop working correctly after a major x. version update, requiring an overhaul of a fairly complex script. I'm almost wondering if setting up an AWX/Tower host in our environment and then using the f5networks Ansible Module for the majority of the heavy lifting followed by some basic logic for file rotation, would be a better long term approach. Ansible would also be a bit more flexible in that I wouldn't have to hardcore values that diverge between individual hosts into the script itself. It's however not clear if the F5networks ansible module supports SFTP as I only see SCP referenced. https://my.f5.com/manage/s/article/K35454259 Advice and insight is much appreciated! #!/bin/bash # F5 backup script based on https://my.f5.com/manage/s/article/K000138297 # User-configurable Variables UCS_DIR="/var/ucs" REMOTE_USER="svc_f5backup" REMOTE_HOST="myhost.contoso.local" REMOTE_DIR="/data/f5/dev" SSH_KEY="/shared/scripts/f5-backup/mykeys/f5user" ENCRYPTION_PASSPHRASE='' # Blank out the value to not encrypt the UCS backup. LOG_FILE="/var/log/backupscript.log" MAX_FILES=45 # Maximum number of backup files to keep # Dynamic Variables (do not edit) HOSTNAME=$(/bin/hostname) DATE=$(date +%Y-%m-%d) UCS_FILE="${UCS_DIR}/${HOSTNAME}-${DATE}.ucs" # Start logging echo "$(date +'%Y-%m-%d %H:%M:%S') - Starting backup script." >> ${LOG_FILE} # Save the UCS backup file if [ -n "${ENCRYPTION_PASSPHRASE}" ]; then echo "Running the UCS save operation (encrypted)." >> ${LOG_FILE} tmsh save /sys ucs ${UCS_FILE} passphrase "${ENCRYPTION_PASSPHRASE}" >> ${LOG_FILE} 2>&1 else echo "Running the UCS save operation (not encrypted)." >> ${LOG_FILE} tmsh save /sys ucs ${UCS_FILE} >> ${LOG_FILE} 2>&1 fi # Create a temporary batch file for SFTP commands BATCH_FILE=$(mktemp) echo "cd ${REMOTE_DIR}" > $BATCH_FILE echo "put ${UCS_FILE}" >> $BATCH_FILE echo "bye" >> $BATCH_FILE # Log that the transfer is starting echo "Starting SFTP transfer." >> ${LOG_FILE} # Execute SFTP command and capture the output transfer_command_output=$(sftp -b "$BATCH_FILE" -i "${SSH_KEY}" -oBatchMode=no "${REMOTE_USER}@${REMOTE_HOST}" 2>&1) transfer_status=$? # Extract the "Transferred:" line transfer_summary=$(echo "$transfer_command_output" | grep "^Transferred: sent") if [ $transfer_status -eq 0 ]; then if [ -n "$transfer_summary" ]; then echo "UCS file copied to the SFTP server successfully (remote:${REMOTE_HOST}:${REMOTE_DIR}/${UCS_FILE}). $transfer_summary" >> ${LOG_FILE} else echo "UCS file copied to the SFTP server successfully (remote:${REMOTE_HOST}:${REMOTE_DIR}/${UCS_FILE}). Please check the log for details." >> ${LOG_FILE} fi else echo "$transfer_command_output" >> ${LOG_FILE} echo "UCS SFTP copy operation failed. Please read the log for details." >> ${LOG_FILE} rm -f $BATCH_FILE exit 1 fi # Clean up the temporary batch file rm -f $BATCH_FILE # Rollover backup files if the number exceeds MAX_FILES echo "Checking and maintaining the maximum number of backup files." >> ${LOG_FILE} # Create a list of files to delete sftp -i "${SSH_KEY}" -oBatchMode=no "${REMOTE_USER}@${REMOTE_HOST}" <<EOF > file_list.txt cd ${REMOTE_DIR} ls -1 ${HOSTNAME}-*.ucs bye EOF # Filter out unwanted lines and sort the files alphanumerically grep -v 'sftp>' file_list.txt | grep -v '^cd ' | sort > filtered_file_list.txt # Determine files to delete files_to_delete=$(head -n -${MAX_FILES} filtered_file_list.txt) if [ -n "$files_to_delete" ]; then # Create a temporary batch file for SFTP cleanup commands CLEANUP_BATCH_FILE=$(mktemp) echo "cd ${REMOTE_DIR}" > $CLEANUP_BATCH_FILE for file in $files_to_delete; do echo "Deleting $file" >> ${LOG_FILE} echo "rm $file" >> $CLEANUP_BATCH_FILE done echo "bye" >> $CLEANUP_BATCH_FILE # Execute SFTP cleanup command and log the output cleanup_command_output=$(sftp -b "$CLEANUP_BATCH_FILE" -i "${SSH_KEY}" -oBatchMode=no "${REMOTE_USER}@${REMOTE_HOST}" 2>&1) echo "$cleanup_command_output" >> ${LOG_FILE} # Clean up the temporary batch file rm -f $CLEANUP_BATCH_FILE else echo "No files to delete. Total files within limit." >> ${LOG_FILE} fi # Clean up the file lists rm -f file_list.txt filtered_file_list.txt # Delete the local copy of the UCS archive tmsh delete /sys ucs ${UCS_FILE} >> ${LOG_FILE} 2>&1 echo "$(date +'%Y-%m-%d %H:%M:%S') - Backup script completed." >> ${LOG_FILE}List BIG-IP Next Instance Backups on Central Manager
In the Central Manager GUI, you can create/schedule BIG-IP Next Instance backups, but outside of the listing shown there, you can't download the files from that view if you want to archive them for off-box requirements. Finding them in the Central Manager command line to download them via secure copy (scp) requires some kubernetes-fu knowhow, mainly, interrogating the persistent volume claims and persistent volumes: kubectl get pvc mbiq-local-storage-pv-claim -o yaml | grep volumeName kubectl get pv <volumename result> -o yaml | grep "path: " This script takes the guesswork out of all that and let's you focus on more important things. Example output: admin@cm1:~$ ./lbu.sh Backup path: /var/lib/rancher/k3s/storage/pvc-ae75faee-101e-49eb-89f7-b66542da1281_default_mbiq-local-storage-pv-claim/backup total 3860 4 drwxrwxrwx 2 root root 4096 Mar 7 19:33 . 4 drwxrwxrwx 7 root root 4096 Feb 2 00:01 .. 1780 -rw-r--r-- 1 ubuntu lxd 1821728 Feb 28 18:40 3b9ef4d8-0f0b-453d-b350-c8720a30db16.2024-02-28.18-39-59.backup.tar.gz 288 -rw-r--r-- 1 ubuntu lxd 292464 Feb 28 18:39 7bf4e3ac-e8a2-44a3-bead-08be6c590071.2024-02-28.18-39-15.backup.tar.gz 1784 -rw-r--r-- 1 ubuntu lxd 1825088 Mar 7 19:33 7bf4e3ac-e8a2-44a3-bead-08be6c590071.2024-03-07.19-32-56.backup.tar.gz Script Source
Issue with a simple Bash Script for adding an iRule to a list of Virtual Servers.
Hello Community, I am having an issue with a bash script for an F5 BIG-IP Load Balancer, intended to read and iterate over a .txt list of Virtual Server names, look up the partition for a given VS, and add an iRule to it. When running the script I am only hitting the outermost 'else' statement for being unable to find the partition and VS name. My script logic is based on F5 Support solution K41961653: <p> #!/bin/bash # Prompt the user for the iRule name and read it into the 'new' variable echo "Please enter the iRule name:" read new noneRules='rules none' while IFS= read -r vs_name; do # Retrieve the partition and virtual server name full_vs_info=$(tmsh -c "cd /; list ltm virtual recursive" | grep "$vs_name" | grep -m1 "^ltm virtual") echo "Full VS Info Debug: $full_vs_info" # Extract the partition and virtual server name from the retrieved information if [[ $full_vs_info =~ ltm\ virtual\ (.+)/(.+) ]]; then partition="${BASH_REMATCH[1]}" vs_name="${BASH_REMATCH[2]}" # Format the tmsh command to include the partition rule=$(tmsh list ltm virtual /$partition/$vs_name rules | egrep -v "\{|\}" | xargs) if [[ "$rule" == "$noneRules" ]]; then tmsh modify ltm virtual /$partition/$vs_name rules { $new } echo "iRule $new was added to $vs_name in partition $partition" else# tmsh modify ltm virtual /$partition/$vs_name rules { $rule $new } echo "iRules $rule were conserved and added $new to $vs_name in partition $partition" fi else echo "Could not find partition and virtual server name for $vs_name" fi done < /shared/tmp/test_list.txt tmsh save sys config </p> As far as I was able to troubleshoot, the problem I am encountering appears to be with line 11 of my script where I attempt to assign the string "ltm virtual SomePartition/VS_Example.com {" to the "full_vs_info" variable using: full_vs_info=$(tmsh -c "cd /; list ltm virtual recursive" | grep "$vs_name" | grep -m1 "^ltm virtual") When I run the tmsh command [tmsh -c "cd /; list ltm virtual recursive" | grep "VS_Example.com" | grep -m1 "^ltm virtual"] on its own, from the F5's Bash shell, I am getting the output I expect: "ltm virtual SomePartition/VS_Example.com {" However, when I run the script with the debug echo , it only outputs "Full VS Info Debug:", and ends the script with "Could not find partition and virtual server name for $vs_name" and a sys config save. I am attempting to run this on a BIG-IP, version 15.1.10.2, build 0.44.2. I am quite new to both Bash scripting and F5 LBs. All feedback and criticism is highly appreciated! Thanks in advance!
F5 Automation - TCL & Bash
Problem this snippet solves: This is a really simple way to automate CLI command execution on multiple F5 devices using Bash & TCL scripting. How to use this snippet: On a linux machine that is utilized to connect to the F5 device: Create a directory mkdir F5_Check Within the "F5_Check" directory, create the following 3 files: F5_Host.txt (This file contains F5's IP address) F5_Bash_v1 (This is the bash script used to collect username/password for F5) F5_Out_v1.exp (This is the TCL script executes the relevant commands on F5) Explanation of the 3 files: File Content: F5_Out_v1.exp is provided as code share. This is the main TCL script that is utiliezd to execute CLI commands on multiple F5 devices. File Content: F5_Bash_v1 #!/bin/bash # Collect the username and password for F5 access echo -n "Enter the username " read -s -e user echo -ne '\n' echo -n "Enter the password " read -s -e password echo -ne '\n' # Feed the expect script a device list & the collected username & passwords for device in `cat ~/F5_Check/F5_Host.txt`; do ./F5_Out_v1.exp $device $password $user ; done File Contents: F5_Host.txt This contains the management IP of the F5 devices. Example: cat F5_Host.txt 10.12.12.200 10.12.12.201 10.12.12.202 10.12.12.203 Code : #!/usr/bin/expect -f # Set variables set hostname [lindex $argv 0] set password [lindex $argv 1] set username [lindex $argv 2] # Log results log_file -a ~/F5_Check/F5LOG.log # Announce which device we are working on and the time send_user "\n" send_user ">>>>> Working on $hostname @ [exec date] <<<<<\n" send_user "\n" # SSH access to device spawn ssh $username@$hostname expect { "no)? " { send "yes\n" expect "*assword: " sleep 1 send "$password\r" } "*assword: " { sleep 1 send "$password\r" } } expect "(tmos)#" send "sys\n" expect "(tmos.sys)#" send "show software\n" expect "#" send "exit\n" expect "#" send "quit\n" expect ":~\$" exit Tested this on version: 11.5
Use F5 LTM as HTTP Proxy
Problem this snippet solves: LTM product can be used as a HTTP Proxy for servers and PC. This code explains minimum requirements to configure proxy feature without SWG module (configurations from Explicit Forward Proxy documentation without documentation ) and without explicit proxy iApp. How to use this snippet: All these commands must be run in bash shell. Create HTTP PROXY VIRTUAL SERVER Configure variables used in next commands Variable HTTPBaseName is used to create : Resolver object : RESOLVER_${HTTPBaseName} HTTP profile : http_${HTTPBaseName} virtual server : VS_${HTTPBaseName} HTTPBaseName="HTTP_FORWARD_PROXY" VS_IP="192.168.2.80" VS_PORT="8080" create DNS resolver with your DNS server (1.1.1.1 is for demo using cloudflare) tmsh create net dns-resolver RESOLVER_${HTTPBaseName} { forward-zones replace-all-with { . { nameservers replace-all-with { 1.1.1.1:domain { } } } } route-domain 0 } create HTTP profile type explicit, using DNS resolver. The parameter default-connect-handling allow enables HTTPS connections without SSL inspection tmsh create ltm profile http http_${HTTPBaseName} { defaults-from http-explicit explicit-proxy { default-connect-handling allow dns-resolver RESOLVER_${HTTPBaseName} } proxy-type explicit } create HTTP proxy Virtual server tmsh create ltm virtual VS_${HTTPBaseName} { destination ${VS_IP}:${VS_PORT} ip-protocol tcp mask 255.255.255.255 profiles replace-all-with { http_${HTTPBaseName} { } tcp } source 0.0.0.0/0 source-address-translation { type automap } translate-address enabled translate-port enabled} ENABLE SSL FORWARD PROXY This section is not required to forward HTTPS requests but only to enable SSL inspection on HTTPS requests. Note : Following configuration requires SSL, Forward Proxy License. Configure variables used in next commands Variable SSLBaseName is used to create : certificate / key pair : ${SSLBaseName} Client SSL profile : clientssl_${SSLBaseName} Server SSL profile : serverssl_${SSLBaseName} virtual server : VS_${SSLBaseName} SSLBaseName="SSL_FORWARD_PROXY" dirname="/var/tmp" CASubject="/C=FR/O=DEMO\ COMPANY/CN=SSL\ FORWARD\ PROXY\ CA" Create self-signed certificate for CA purpose (not available in WebUI) Self-signed certificates created in WebUI doesn't have CA capability required for SSL FORWARD PROXY. openssl genrsa -out ${dirname}/${SSLBaseName}.key 4094 openssl req -sha512 -new -x509 -days 3650 -key ${dirname}/${SSLBaseName}.key -out ${dirname}/${SSLBaseName}.crt -subj "${CASubject}" Import certificates in TMOS tmsh install sys crypto key ${SSLBaseName}.key from-local-file ${dirname}/${SSLBaseName}.key; tmsh install sys crypto cert ${SSLBaseName}.crt from-local-file ${dirname}/${SSLBaseName}.crt; After CA Certificate is imported, browse in WebUI, retrieve it and import it in client browsers trusted CA Create SSL profiles for SSL FORWARD PROXY tmsh create ltm profile client-ssl clientssl_${SSLBaseName} { cert-lookup-by-ipaddr-port disabled defaults-from clientssl mode enabled proxy-ca-cert ${SSLBaseName}.crt proxy-ca-key ${SSLBaseName}.key ssl-forward-proxy enabled } tmsh create ltm profile server-ssl serverssl_${SSLBaseName} { defaults-from serverssl ssl-forward-proxy enabled } create SSL FORWARD PROXY Virtual server tmsh create ltm virtual VS_${SSLBaseName} { destination 0.0.0.0:https ip-protocol tcp profiles replace-all-with { clientssl_${SSLBaseName} { context clientside } serverssl_${SSLBaseName} { context serverside } http { } tcp { } } source 0.0.0.0/0 translate-address disabled translate-port disabled vlans replace-all-with { http-tunnel } vlans-enabled } Change HTTP EXPLICIT PROXY Default Connect Handling to Deny tmsh modify ltm profile http http_${HTTPBaseName} explicit-proxy { default-connect-handling deny } Note : These commands were tested in both 12.1 and 13.1 versions. Code : No Code
/var/ directory running out of space
I'm getting a broadcast message on the bash CLI that warns me about the /var/ directory running out of room. 011d0004:3: Disk partition /var has only 14% free There are four Tomcat files of identical size taking up almost a GB of space, but I don't know what purpose they serve. -rw-r--r--. 1 tomcat tomcat 240192271 2017-11-05 19:51 1509911444508.upload -rw-r--r--. 1 tomcat tomcat 240192271 2017-11-05 10:46 1509878765278.upload -rw-r--r--. 1 tomcat tomcat 240192271 2017-11-05 10:39 1509878372606.upload -rw-r--r--. 1 tomcat tomcat 240192271 2017-11-05 10:38 1509878267154.upload Does anyone know what the purpose of these files are, and whether or not it's safe to remove them?External monitor Realsec Cryptosec HSM
Hi all, I am trying to monitor a HSM appliance using the external monitor template provided on this link: link text I modified this part in the template so it should send CCCCNC and the response 00000000, I really am not sure if this is the correct line. I uploaded the script and attached it to the pool and it is available and actively sending monitor request towards the pool members, however looking at the payload in wireshark there is no data being send. I verify it in wireshark (Follow TCP stream). Send the request request and check the response echo -n 'CCCCNC' | nc $IP $PORT | grep "00000000" 2>&1 > /dev/null Someone over here with some bash scripting experience? Thanks in advance.Combine and Modify Grep Output
Hi, I'm trying to write a script to output virtual server stats. Can anyone help me modify the output? Right now the GREP output looks like this: Ltm::Virtual Server: www.website.com.443 Bits In 2.1G 0 - Bits Out 2.3G 0 - Ltm::Virtual Server: www.website.com.80 Bits In 1.4G 0 - Bits Out 740.8M 0 - I need help making the output like this: www.website.com.443 Bits In 2.1G Bits Out 2.3G www.website.com.80 Bits In 1.4G Bits Out 740.8M Afterwards, I can diff files from different dates, and figure out which VIPS are not being used.
Windows-File-Share-Monitor-SMB-CIFS
Hi, I am trying to use the: https://devcentral.f5.com/wiki/AdvDesignConfig.Windows-File-Share-Monitor-SMB-CIFS.ashx?lc=1 In the article the monitor for gtm is detailed as: monitor "smb_external_monitor" { defaults from "external" interval 10 timeout 40 probe_interval 1 probe_timeout 5 probe_num_probes 1 probe_num_successes 1 dest *:* "SEARCH_STRING" "got it" "DEBUG" "1" run "smb_monitor.bash" "USERNAME" "aaron" "FILE" "/share/test.txt" args "" "PASSWORD" "Test123!" partition "Common" } My monitor is 11.5.1 so the tmsh syntax is a little different: gtm monitor external /Common/smb_external_monitor { defaults-from /Common/external destination *:* interval 30 probe-timeout 5 run /Common/smb_monitor.bash timeout 120 user-defined DEBUG 1 user-defined FILE /F5GTM/F5GTMTST.txt user-defined PASSWORD ****** user-defined SEARCH_STRING up user-defined USERNAME f5gtm } I have also tried manually setting the debug to 1 in the script as suggested. I get nothing in /var/log/ltm and the monitor is failing. Any ideas? Thanks, BenLet's Encrypt with Cloudflare DNS and F5 REST API
Hi all This is a followup on the now very old Let's Encrypt on a Big-IP article. It has served me, and others, well but is kind of locked to a specific environment and doesn't scale well. I have been going around it for some time but couldn't find the courage (aka time) to get started. However, due to some changes to my DNS provider (they were aquired and shut down) I finally took the plunges and moved my domains to a provider with an API and that gave me the opportunity to make a more nimble solution. To make things simple I chose Cloudflare as the community proliferation is enormous and it is easy to find examples and tools. I though think that choosing another provide with an open API isn't such a big deal. After playing around with different tools I realized that I didn't need them as it ended up being much easier to just use curl. So, if the other providers have just a somewhat close resemblance it shouldn't be such a big task converting the scripts to fit. There might be finer and more advanced solutions out there, but my goal was that I needed a solution that had as few dependencies as possible and if I could make that only Bash and Curl it would be perfect. And that is what I ended up with 😎 Just put 5 files in the same directory, adjust the config to your environment, and BAM you're good to go!!😻 And if you need to run it somewhere else just copy the directory over and continue like nothing was changed. That is what I call portability 😁 Find all the details here: Let's Encrypt with Cloudflare DNS and F5 REST API Please just drop me a line if you have any questions or feedback or find any bugs.
