BigIP UCS Backup script; looking for some guidance on design
Greetings, I've began to work on a bash script, intended to be ran locally on each F5 appliance via a cron task. The criteria for this script has been, Saves the UCS /w encryption using {Hostname}-YYYY-MM-DD.ucs naming format. Uploads the generated UCS file to a SFTP server SFTP native commands are a MUST, SCP will not work due to it's reliance on command shell/login. Rollover after X # of saved files in order to prevent storage exhaustion on the target SFTP Server I strongly doubt any form of deduplication will work with a encrypted UCS Sends an email notification if the backup failed I've so far written a script that addresses the first 3 criteria and have been waiting for those to go through their paces in testing before adding in notification logic. The commands and logic being used have gotten more complex, the further I've gotten into the script's development. This has lead to some concerns about whether this is the best approach given the nature of the F5 BigIP systems being a vendor appliance and worry that there's a large possibility commands may stop working correctly after a major x. version update, requiring an overhaul of a fairly complex script. I'm almost wondering if setting up an AWX/Tower host in our environment and then using the f5networks Ansible Module for the majority of the heavy lifting followed by some basic logic for file rotation, would be a better long term approach. Ansible would also be a bit more flexible in that I wouldn't have to hardcore values that diverge between individual hosts into the script itself. It's however not clear if the F5networks ansible module supports SFTP as I only see SCP referenced. https://my.f5.com/manage/s/article/K35454259 Advice and insight is much appreciated! #!/bin/bash # F5 backup script based on https://my.f5.com/manage/s/article/K000138297 # User-configurable Variables UCS_DIR="/var/ucs" REMOTE_USER="svc_f5backup" REMOTE_HOST="myhost.contoso.local" REMOTE_DIR="/data/f5/dev" SSH_KEY="/shared/scripts/f5-backup/mykeys/f5user" ENCRYPTION_PASSPHRASE='' # Blank out the value to not encrypt the UCS backup. LOG_FILE="/var/log/backupscript.log" MAX_FILES=45 # Maximum number of backup files to keep # Dynamic Variables (do not edit) HOSTNAME=$(/bin/hostname) DATE=$(date +%Y-%m-%d) UCS_FILE="${UCS_DIR}/${HOSTNAME}-${DATE}.ucs" # Start logging echo "$(date +'%Y-%m-%d %H:%M:%S') - Starting backup script." >> ${LOG_FILE} # Save the UCS backup file if [ -n "${ENCRYPTION_PASSPHRASE}" ]; then echo "Running the UCS save operation (encrypted)." >> ${LOG_FILE} tmsh save /sys ucs ${UCS_FILE} passphrase "${ENCRYPTION_PASSPHRASE}" >> ${LOG_FILE} 2>&1 else echo "Running the UCS save operation (not encrypted)." >> ${LOG_FILE} tmsh save /sys ucs ${UCS_FILE} >> ${LOG_FILE} 2>&1 fi # Create a temporary batch file for SFTP commands BATCH_FILE=$(mktemp) echo "cd ${REMOTE_DIR}" > $BATCH_FILE echo "put ${UCS_FILE}" >> $BATCH_FILE echo "bye" >> $BATCH_FILE # Log that the transfer is starting echo "Starting SFTP transfer." >> ${LOG_FILE} # Execute SFTP command and capture the output transfer_command_output=$(sftp -b "$BATCH_FILE" -i "${SSH_KEY}" -oBatchMode=no "${REMOTE_USER}@${REMOTE_HOST}" 2>&1) transfer_status=$? # Extract the "Transferred:" line transfer_summary=$(echo "$transfer_command_output" | grep "^Transferred: sent") if [ $transfer_status -eq 0 ]; then if [ -n "$transfer_summary" ]; then echo "UCS file copied to the SFTP server successfully (remote:${REMOTE_HOST}:${REMOTE_DIR}/${UCS_FILE}). $transfer_summary" >> ${LOG_FILE} else echo "UCS file copied to the SFTP server successfully (remote:${REMOTE_HOST}:${REMOTE_DIR}/${UCS_FILE}). Please check the log for details." >> ${LOG_FILE} fi else echo "$transfer_command_output" >> ${LOG_FILE} echo "UCS SFTP copy operation failed. Please read the log for details." >> ${LOG_FILE} rm -f $BATCH_FILE exit 1 fi # Clean up the temporary batch file rm -f $BATCH_FILE # Rollover backup files if the number exceeds MAX_FILES echo "Checking and maintaining the maximum number of backup files." >> ${LOG_FILE} # Create a list of files to delete sftp -i "${SSH_KEY}" -oBatchMode=no "${REMOTE_USER}@${REMOTE_HOST}" <<EOF > file_list.txt cd ${REMOTE_DIR} ls -1 ${HOSTNAME}-*.ucs bye EOF # Filter out unwanted lines and sort the files alphanumerically grep -v 'sftp>' file_list.txt | grep -v '^cd ' | sort > filtered_file_list.txt # Determine files to delete files_to_delete=$(head -n -${MAX_FILES} filtered_file_list.txt) if [ -n "$files_to_delete" ]; then # Create a temporary batch file for SFTP cleanup commands CLEANUP_BATCH_FILE=$(mktemp) echo "cd ${REMOTE_DIR}" > $CLEANUP_BATCH_FILE for file in $files_to_delete; do echo "Deleting $file" >> ${LOG_FILE} echo "rm $file" >> $CLEANUP_BATCH_FILE done echo "bye" >> $CLEANUP_BATCH_FILE # Execute SFTP cleanup command and log the output cleanup_command_output=$(sftp -b "$CLEANUP_BATCH_FILE" -i "${SSH_KEY}" -oBatchMode=no "${REMOTE_USER}@${REMOTE_HOST}" 2>&1) echo "$cleanup_command_output" >> ${LOG_FILE} # Clean up the temporary batch file rm -f $CLEANUP_BATCH_FILE else echo "No files to delete. Total files within limit." >> ${LOG_FILE} fi # Clean up the file lists rm -f file_list.txt filtered_file_list.txt # Delete the local copy of the UCS archive tmsh delete /sys ucs ${UCS_FILE} >> ${LOG_FILE} 2>&1 echo "$(date +'%Y-%m-%d %H:%M:%S') - Backup script completed." >> ${LOG_FILE}List BIG-IP Next Instance Backups on Central Manager
In the Central Manager GUI, you can create/schedule BIG-IP Next Instance backups, but outside of the listing shown there, you can't download the files from that view if you want to archive them for off-box requirements. Finding them in the Central Manager command line to download them via secure copy (scp) requires some kubernetes-fu knowhow, mainly, interrogating the persistent volume claims and persistent volumes: kubectl get pvc mbiq-local-storage-pv-claim -o yaml | grep volumeName kubectl get pv <volumename result> -o yaml | grep "path: " This script takes the guesswork out of all that and let's you focus on more important things. Example output: admin@cm1:~$ ./lbu.sh Backup path: /var/lib/rancher/k3s/storage/pvc-ae75faee-101e-49eb-89f7-b66542da1281_default_mbiq-local-storage-pv-claim/backup total 3860 4 drwxrwxrwx 2 root root 4096 Mar 7 19:33 . 4 drwxrwxrwx 7 root root 4096 Feb 2 00:01 .. 1780 -rw-r--r-- 1 ubuntu lxd 1821728 Feb 28 18:40 3b9ef4d8-0f0b-453d-b350-c8720a30db16.2024-02-28.18-39-59.backup.tar.gz 288 -rw-r--r-- 1 ubuntu lxd 292464 Feb 28 18:39 7bf4e3ac-e8a2-44a3-bead-08be6c590071.2024-02-28.18-39-15.backup.tar.gz 1784 -rw-r--r-- 1 ubuntu lxd 1825088 Mar 7 19:33 7bf4e3ac-e8a2-44a3-bead-08be6c590071.2024-03-07.19-32-56.backup.tar.gz Script Source
Complete F5 Automated Backup Solution
Problem this snippet solves: Hi all, Often I've been scouring the devcentral fora and codeshares to find that one piece of handywork that will drastically simplify my automated backup needs on F5 devices. Based on the works of Jason Rahm in his post "Third Time's the Charm: BIG-IP Backups Simplified with iCall" on the 26th of June 2013, I went ahead and created my own iApp that pretty much provides the answers for all my backup-needs. Here's a feature list of this iApp: It allows you to choose between both UCS or SCF as backup-types. (whilst providing ample warnings about SCF not being a very good restore-option due to the incompleteness in some cases) It allows you to provide a passphrase for the UCS archives (the standard GUI also does this, so the iApp should too) It allows you to not include the private keys (same thing: standard GUI does it, so the iApp does it too) It allows you to set a Backup Schedule for every X minutes/hours/days/weeks/months or a custom selection of days in the week It allows you to set the exact time, minute of the hour, day of the week or day of the month when the backup should be performed (depending on the usefulness with regards to the schedule type) It allows you to transfer the backup files to external devices using 4 different protocols, next to providing local storage on the device itself * SCP (username/private key without password) SFTP (username/private key without password) FTP (username/password) SMB (using smbclient, with username/password) Local Storage (/var/local/ucs or /var/local/scf) It stores all passwords and private keys in a secure fashion: encrypted by the master key of the unit (f5mku), rendering it safe to store the backups, including the credentials off-box It has a configurable automatic pruning function for the Local Storage option, so the disk doesn't fill up (i.e. keep last X backup files) It allows you to configure the filename using the date/time wildcards from the tcl clock command, as well as providing a variable to include the hostname It requires only the WebGUI to establish the configuration you desire It allows you to disable the processes for automated backup, without you having to remove the Application Service or losing any previously entered settings For the external shellscripts it automatically generates, the credentials are stored in encrypted form (using the master key) It allows you to no longer be required to make modifications on the linux command line to get your automated backups running after an RMA or restore operation It cleans up after itself, which means there are no extraneous shellscripts or status files lingering around after the scripts execute Enjoy! Thomas Schockaert Contributed by: Thomas Schockaert How to use this snippet: minimum version 11.4 Code : 67735 Tested this on version: 11.4
F5 iApp Automated Backup
Problem this snippet solves: This is now available on GitHub! Please look on GitHub for the latest version, and submit any bugs or questions as an "Issue" on GitHub: (Note: DevCentral admin update - Daniel's project appears abandoned so it's been forked and updated to the link below. @damnski on github added some SFTP code that has been merged in as well.) https://github.com/f5devcentral/f5-automated-backup-iapp Intro Building on the significant work of Thomas Schockaert (and several other DevCentralites) I enhanced many aspects I needed for my own purposes, updated many things I noticed requested on the forums, and added additional documentation and clarification. As you may see in several of my comments on the original posts, I iterated through several 2.2.x versions and am now releasing v3.0.0. Below is the breakdown! Also, I have done quite a bit of testing (mostly on v13.1.0.1 lately) and I doubt I've caught everything, especially with all of the changes. Please post any questions or issues in the comments. Cheers! Daniel Tavernier (tabernarious) Related posts: Git Repository for f5-automated-backup-iapp (https://github.com/tabernarious/f5-automated-backup-iapp) https://community.f5.com/t5/technical-articles/f5-automated-backups-the-right-way/ta-p/288454 https://community.f5.com/t5/crowdsrc/complete-f5-automated-backup-solution/ta-p/288701 https://community.f5.com/t5/crowdsrc/complete-f5-automated-backup-solution-2/ta-p/274252 https://community.f5.com/t5/technical-forum/automated-backup-solution/m-p/24551 https://community.f5.com/t5/crowdsrc/tkb-p/CrowdSRC v3.2.1 (20201210) Merged v3.1.11 and v3.2.0 for explicit SFTP support (separate from SCP). Tweaked the SCP and SFTP upload directory handling; detailed instructions are in the iApp. Tested on 13.1.3.4 and 14.1.3 v3.1.11 (20201210) Better handling of UCS passphrases, and notes about characters to avoid. I successfully tested this exact passphrase in the 13.1.3.4 CLI (surrounded with single quote) and GUI (as-is): `~!@#$%^*()aB1-_=+[{]}:./? I successfully tested this exact passphrase in 14.1.3 (square-braces and curly-braces would not work): `~!@#$%^*()aB1-_=+:./? Though there may be situations these could work, avoid these characters (separated by spaces): " ' & | ; < > \ [ ] { } , Moved changelog and notes from the template to CHANGELOG.md and README.md. Replaced all tabs (\t) with four spaces. v3.1.10 (20201209) Added SMB Version and SMB Security options to support v14+ and newer versions of Microsoft Windows and Windows Server. Tested SMB/CIFS on 13.1.3.4 and 14.1.3 against Windows Server 2019 using "2.0" and "ntlmsspi" v3.1.0: Removed "app-service none" from iCall objects. The iCall objects are now created as part of the Application Service (iApp) and are properly cleaned up if the iApp is redeployed or deleted. Reasonably tested on 11.5.4 HF2 (SMB worked fine using "mount -t cifs") and altered requires-bigip-version-min to match. Fixing error regarding "script did not successfully complete: (can't read "::destination_parameters__protocol_enable": no such variable" by encompassing most of the "implementation" in a block that first checks $::backup_schedule__frequency_select for "Disable". Added default value to "filename format". Changed UCS default value for $backup_file_name_extension to ".ucs" and added $fname_noext. Removed old SFTP sections and references (now handled through SCP/SFTP). Adjusted logging: added "sleep 1" to ensure proper logging; added $backup_directory to log message. Adjusted some help messages. New v3.0.0 features: Supports multiple instances! (Deploy multiple copies of the iApp to save backups to different places or perhaps to keep daily backups locally and send weekly backups to a network drive.) Fully ConfigSync compatible! (Encrypted values now in $script instead of local file.) Long passwords supported! (Using "-A" with openssl which reads/writes base64 encoded strings as a single line.) Added $script error checking for all remote backup types! (Using 'catch' to prevent tcl errors when $script aborts.) Backup files are cleaned up after any $script errors due to new error checking. Added logging! (Run logs sent to '/var/log/ltm' via logger command which is compatible with BIG-IP Remote Logging configuration (syslog). Run logs AND errors sent to '/var/tmp/scriptd.out'. Errors may include plain-text passwords which should not be in /var/log/ltm or syslog.) Added custom cipher option for SCP! (In case BIG-IP and the destination server are not cipher-compatible out of the box.) Added StrictHostKeyChecking=no option. (This is insecure and should only be used for testing--lots of warnings.) Combined SCP and SFTP because they are both using SCP to perform the remote copy. (Easier to maintain!) Original v1.x.x and v2.x.x features kept (copied from an original post): It allows you to choose between both UCS or SCF as backup-types. (whilst providing ample warnings about SCF not being a very good restore-option due to the incompleteness in some cases) It allows you to provide a passphrase for the UCS archives (the standard GUI also does this, so the iApp should too) It allows you to not include the private keys (same thing: standard GUI does it, so the iApp does it too) It allows you to set a Backup Schedule for every X minutes/hours/days/weeks/months or a custom selection of days in the week It allows you to set the exact time, minute of the hour, day of the week or day of the month when the backup should be performed (depending on the usefulness with regards to the schedule type) It allows you to transfer the backup files to external devices using 4 different protocols, next to providing local storage on the device itself SCP (username/private key without password) SFTP (username/private key without password) FTP (username/password) SMB (now using TMOS v12.x.x compatible 'mount -t cifs', with username/password) Local Storage (/var/local/ucs or /var/local/scf) It stores all passwords and private keys in a secure fashion: encrypted by the master key of the unit (f5mku), rendering it safe to store the backups, including the credentials off-box It has a configurable automatic pruning function for the Local Storage option, so the disk doesn't fill up (i.e. keep last X backup files) It allows you to configure the filename using the date/time wildcards from the tcl [clock] command, as well as providing a variable to include the hostname It requires only the WebGUI to establish the configuration you desire It allows you to disable the processes for automated backup, without you having to remove the Application Service or losing any previously entered settings For the external shellscripts it automatically generates, the credentials are stored in encrypted form (using the master key) It allows you to no longer be required to make modifications on the linux command line to get your automated backups running after an RMA or restore operation It cleans up after itself, which means there are no extraneous shellscripts or status files lingering around after the scripts execute How to use this snippet: Find and download the latest iApp template on GitHub (e.g "f5.automated_backup.v3.2.1.tmpl.tcl"). Import the text file as an iApp Template in the BIG-IP GUI. Create an Application Service using the imported Template. Answer the questions (paying close attention to the help sections). Check /var/tmp/scriptd.out for general logs and errors. Tested this on version: 16.0Third Time's the Charm: BIG-IP Backups Simplified with iCall
Backing up the BIG-IP Configuration is something I've written about a couple times (here and here) previously. Well, third time's the charm, thanks to the new iCall feature in the 11.4 release. This time, I've even wrapped in scp support to send the backup to a remote server! The great thing about this solution is the only thing required outside of tmsh is setting up the ssh keys. SSH Key Configuration 1. On Big_IP, create your keys [root@ltm1:Active:Standalone] config # ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: fd:d0:07:64:1d:f6:21:86:49:47:85:77:74:15:2c:36 root@ltm1.dc.local 2. copy the public key to your archive server [root@ltm1:Active:Standalone] config # scp /root/.ssh/id_rsa.pub jrahm@192.168.6.10:/var/tmp/id_rsa.pub-ltm1 jrahm@192.168.6.10's password: id_rsa.pub 3. Login to your server and append the public key to authorized keys, recommending not a root account! For my ubuntu installation, I have an encrypted home directory, so there are a couple extra steps to apply the authorized keys: 3a. Create a user-specific directory in /etc/ssh and change permissions/ownership sudo mkdir /etc/ssh/jrahm sudo chmod 755 /etc/ssh/jrahm sudo chown jrahm:jrahm /etc/ssh/jrahm 3b. Edit /etc/ssh/sshd_config to update authorized keys file: sudo vi /etc/ssh/sshd_config #update these two lines: AuthorizedKeysFile /etc/ssh/%u/authorized_keys PubkeysAuthentication yes 3c. Add your key to the user: sudo cat /var/tmp/id_rsa.pub-ltm1 >> /etc/ssh/jrahm/authorized_keys 3d. restart sshd on archive server sudo service ssh restart 4. Test login from BIG-IP to server (no password prompt, this is good!) [root@ltm1:Active:Standalone] config # ssh jrahm@192.168.6.10 Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-23-generic x86_64) Last login: Mon Apr 1 15:15:22 2013 from 192.168.6.5 5. Test scp functionality: [root@ltm1:Active:Standalone] tmp # scp f5backup-ltm1.dc.local-20130326160303.tar.gz jrahm@192.168.6.10:/var/tmp/ f5backup-ltm1.dc.local-20130326160303.tar.gz Create the iCall Script iCall scripts are created in the vim editor much like tmsh scripts by (in the tmsh shell) calling create sys icall script <script name>. The skeleton looks like this: create script testme { app-service none definition { } description none events none } For this script, we only need to focus on the definition. There is no rocket science in this script at all, just setting date and file information, saving the archive, creating the tarball, zipping it up, and sending it off. sys icall script f5.config_backup.v1.0.0 { app-service none definition { #Set Current Date/Time for Filename set cdate [clock format [clock seconds] -format "%Y%m%d%H%M%S"] #Pull hostname from config for Filename set host [tmsh::get_field_value [lindex [tmsh::get_config sys global-settings] 0] hostname] #Create Temp Directory set tmpdir [exec mktemp -d /var/tmp/f5backup.XXXXXXXXXX] #Set Filename Root set fname "f5backup-$host-$cdate" #Export UCS tmsh::save /sys ucs $tmpdir/$fname #Create Backup exec tar cvzf /var/tmp/$fname.tar.gz -C $tmpdir . 2> /dev/null #Remove Temp Directory exec rm -rf $tmpdir #SSH settings exec scp /var/tmp/$fname.tar.gz jrahm@192.168.6.10:/var/tmp/ } description none events none } As you can probably surmise, an iCall script is pretty much a tmsh script, same Tcl / tmsh, just stored differently to be utilized by iCall handlers. Create the iCall Handler Since backups are typically run once a day, the handler we'll need is a periodic handler. There are several arguments you can set on a periodic handler: root@(ltm2)(cfg-sync Standalone)(Active)(/Common)(tmos)# create sys icall handler periodic testme ? Identifier: [object identifier] Specify a name for the handler item Properties: "{" Optional delimiter app-service arguments Specifies a set of name/value pairs to be passed in as data to the script for every execution description User defined explanation of the item first-occurrence Specifies the date and time of the first occurrence this handler should execute interval Specifies the number of seconds between each occurrence of this handler's automatic execution last-occurrence Specifies the date and time after which no more occurrences will execute script Specifies the handler's script to execute upon invocation status Manage the perpetual process by specifying active or inactive In my case, I only need to set the first-occurrence, the interval, and the script to call: sys icall handler periodic f5.config_backup.v1.0.0 { first-occurrence 2013-06-26:08:18:00 interval 360 script f5.config_backup.v1.0.0 } A normal interval would be once per day (86400), but since this is a test scenario, I set the interval low so I could see it happen at least twice. On the BIG-IP, notice that the temp directories are gone (but the archives remain, you can add a line to the script to clean up if you like) [root@ltm2:Active:Standalone] tmp # ls -las total 13100 8 drwxrwxrwt 6 root root 4096 Jun 26 08:23 . 8 drwxr-xr-x 21 root root 4096 Jun 20 09:41 .. 8 -rw-r--r-- 1 root root 718 Jun 24 09:59 audit.out 8 -rw-r--r-- 1 root root 1013 Jun 24 10:00 csyncd.out 4 -rw-r--r-- 1 root root 0 Jun 24 10:00 devmgmtd++.out 4 -rw-r--r-- 1 root root 0 Jun 24 09:59 evrouted.out 476 -rw-r--r-- 1 root root 478717 Jun 26 08:18 f5backup-ltm2.dc.local-20130626081800.tar.gz 476 -rw-r--r-- 1 root root 478556 Jun 26 08:23 f5backup-ltm2.dc.local-20130626082336.tar.gz 8 drwxr-xr-x 4 root root 4096 Jun 20 09:34 install And finally, the same files in place on my remote server: jrahm@u1204lts:/var/tmp$ ls -las total 2832 4 drwxrwxrwt 2 root root 4096 Jun 26 10:24 . 4 drwxr-xr-x 13 root root 4096 Mar 27 13:35 .. 468 -rw-r--r-- 1 jrahm jrahm 478717 Jun 26 10:18 f5backup-ltm2.dc.local-20130626081800.tar.gz 468 -rw-r--r-- 1 jrahm jrahm 478556 Jun 26 10:24 f5backup-ltm2.dc.local-20130626082336.tar.gz Going Further By this point in the article, you might been thinking..."Wait, the previous articles wrapped all that goodness in an iApp. What gives?" Well, I am not leaving you hanging--its' already in the iCall codeshare waiting for you! Stay tuned for future iCall articles, where I'll dive into some perpetual and triggered handler use cases.
