F5 iApp Automated Backup

Problem this snippet solves:

This is now available on GitHub!

Please look on GitHub for the latest version, and submit any bugs or questions as an "Issue" on GitHub:

(Note: DevCentral admin update - Daniel's project appears abandoned so it's been forked and updated to the link below. @damnski on github added some SFTP code that has been merged in as well.)

https://github.com/f5devcentral/f5-automated-backup-iapp

Intro

Building on the significant work of Thomas Schockaert (and several other DevCentralites) I enhanced many aspects I needed for my own purposes, updated many things I noticed requested on the forums, and added additional documentation and clarification. As you may see in several of my comments on the original posts, I iterated through several 2.2.x versions and am now releasing v3.0.0. Below is the breakdown!

Also, I have done quite a bit of testing (mostly on v13.1.0.1 lately) and I doubt I've caught everything, especially with all of the changes. Please post any questions or issues in the comments.

Cheers!

Daniel Tavernier (tabernarious)

v3.2.1 (20201210)

Merged v3.1.11 and v3.2.0 for explicit SFTP support (separate from SCP).
Tweaked the SCP and SFTP upload directory handling; detailed instructions are in the iApp.
Tested on 13.1.3.4 and 14.1.3

v3.1.11 (20201210)

Better handling of UCS passphrases, and notes about characters to avoid.
I successfully tested this exact passphrase in the 13.1.3.4 CLI (surrounded with single quote) and GUI (as-is): `~!@#$%^*()aB1-_=+[{]}:./?
I successfully tested this exact passphrase in 14.1.3 (square-braces and curly-braces would not work): `~!@#$%^*()aB1-_=+:./?
Though there may be situations these could work, avoid these characters (separated by spaces): " ' & | ; < > \ [ ] { } ,
Moved changelog and notes from the template to CHANGELOG.md and README.md.
Replaced all tabs (\t) with four spaces.

v3.1.10 (20201209)

Added SMB Version and SMB Security options to support v14+ and newer versions of Microsoft Windows and Windows Server.
Tested SMB/CIFS on 13.1.3.4 and 14.1.3 against Windows Server 2019 using "2.0" and "ntlmsspi"

v3.1.0:

Removed "app-service none" from iCall objects. The iCall objects are now created as part of the Application Service (iApp) and are properly cleaned up if the iApp is redeployed or deleted.
Reasonably tested on 11.5.4 HF2 (SMB worked fine using "mount -t cifs") and altered requires-bigip-version-min to match.
Fixing error regarding "script did not successfully complete: (can't read "::destination_parameters__protocol_enable": no such variable" by encompassing most of the "implementation" in a block that first checks $::backup_schedule__frequency_select for "Disable".
Added default value to "filename format".
Changed UCS default value for $backup_file_name_extension to ".ucs" and added $fname_noext.
Removed old SFTP sections and references (now handled through SCP/SFTP).
Adjusted logging: added "sleep 1" to ensure proper logging; added $backup_directory to log message.
Adjusted some help messages.

New v3.0.0 features:

Supports multiple instances! (Deploy multiple copies of the iApp to save backups to different places or perhaps to keep daily backups locally and send weekly backups to a network drive.)
Fully ConfigSync compatible! (Encrypted values now in $script instead of local file.)
Long passwords supported! (Using "-A" with openssl which reads/writes base64 encoded strings as a single line.)
Added $script error checking for all remote backup types! (Using 'catch' to prevent tcl errors when $script aborts.)
Backup files are cleaned up after any $script errors due to new error checking.
Added logging! (Run logs sent to '/var/log/ltm' via logger command which is compatible with BIG-IP Remote Logging configuration (syslog). Run logs AND errors sent to '/var/tmp/scriptd.out'. Errors may include plain-text passwords which should not be in /var/log/ltm or syslog.)
Added custom cipher option for SCP! (In case BIG-IP and the destination server are not cipher-compatible out of the box.)
Added StrictHostKeyChecking=no option. (This is insecure and should only be used for testing--lots of warnings.)
Combined SCP and SFTP because they are both using SCP to perform the remote copy. (Easier to maintain!)

Original v1.x.x and v2.x.x features kept (copied from an original post):

It allows you to choose between both UCS or SCF as backup-types. (whilst providing ample warnings about SCF not being a very good restore-option due to the incompleteness in some cases)
It allows you to provide a passphrase for the UCS archives (the standard GUI also does this, so the iApp should too)
It allows you to not include the private keys (same thing: standard GUI does it, so the iApp does it too)
It allows you to set a Backup Schedule for every X minutes/hours/days/weeks/months or a custom selection of days in the week
It allows you to set the exact time, minute of the hour, day of the week or day of the month when the backup should be performed (depending on the usefulness with regards to the schedule type)
It allows you to transfer the backup files to external devices using 4 different protocols, next to providing local storage on the device itself
SCP (username/private key without password)
SFTP (username/private key without password)
FTP (username/password)
SMB (now using TMOS v12.x.x compatible 'mount -t cifs', with username/password)
Local Storage (/var/local/ucs or /var/local/scf)
It stores all passwords and private keys in a secure fashion: encrypted by the master key of the unit (f5mku), rendering it safe to store the backups, including the credentials off-box
It has a configurable automatic pruning function for the Local Storage option, so the disk doesn't fill up (i.e. keep last X backup files)
It allows you to configure the filename using the date/time wildcards from the tcl [clock] command, as well as providing a variable to include the hostname
It requires only the WebGUI to establish the configuration you desire
It allows you to disable the processes for automated backup, without you having to remove the Application Service or losing any previously entered settings
For the external shellscripts it automatically generates, the credentials are stored in encrypted form (using the master key)
It allows you to no longer be required to make modifications on the linux command line to get your automated backups running after an RMA or restore operation
It cleans up after itself, which means there are no extraneous shellscripts or status files lingering around after the scripts execute

How to use this snippet:

Find and download the latest iApp template on GitHub (e.g "f5.automated_backup.v3.2.1.tmpl.tcl").
Import the text file as an iApp Template in the BIG-IP GUI.
Create an Application Service using the imported Template.
Answer the questions (paying close attention to the help sections).
Check /var/tmp/scriptd.out for general logs and errors.

Tested this on version:

16.0

Updated Feb 15, 2022

Version 2.0

Cirrostratus

Joined December 19, 2012

View Profile

102 Comments

erdem_kara_pure7
Nimbostratus
Feb 21, 2025
Hi All,
I edited tcl file for version 3.2.0 like below and its solved on my deployment. i changed style of sending user pass credentials and add smb version string.
ex: mount -t cifs //\${server}/\${msshare}\${mssubdir} \${mountp} -o user=\${username},password=\${password},domain=\${msdomain},vers=3.0
on file :
Line:87
sys application template /Common/f5.automated_backup.v3.2.0.edited {
Line:414

exec echo -e "\#\!/bin/sh\nf5masterkey=\$(f5mku -K)\nusername=\$(echo \"ENCRYPTEDUSERNAME\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\npassword=\$(echo \"ENCRYPTEDPASSWORD\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\nmsdomain=\$(echo \"ENCRYPTEDMSDOMAIN\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\nserver=\$(echo \"ENCRYPTEDSERVER\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\nmsshare=\$(echo \"ENCRYPTEDMSSHARE\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\nmssubdir=\$(echo \"ENCRYPTEDMSSUBDIR\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\nmountp=\$(echo \"ENCRYPTEDMOUNTP\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\ncd BACKUPDIRECTORY\nif \[ \! -d \${mountp} \]\nthen\n\tmkdir -p \${mountp}\n\tif \[ \$? -ne 0 \]\n\tthen\n\t\trm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT*\n\t\texit 1\n\tfi\nfi\n\# The password must be surrounded by two single-quotes to successfully handle special characters. Still does not support comma, single-quote, and double-quote.\nmount -t cifs //\${server}/\${msshare}\${mssubdir} \${mountp} -o user=\${username},password=\${password},domain=\${msdomain},vers=3.0 2>> /var/tmp/scriptd.out\nif \[ \$? -ne 0 \]\n\tthen\n\techo \"DEBUG: Failed to mount //\${server}/\${msshare}\${mssubdir}\ to \${mountp}\" >> /var/tmp/scriptd.out\n\trm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT*\n\texit 1\nelse\n\techo \"DEBUG: Successfully mounted //\${server}/\${msshare}\${mssubdir}\ to \${mountp}\" >> /var/tmp/scriptd.out\nfi\n\nlatestFileOnSMB=\$(ls -t \${mountp}/\*.BACKUPFILENAMEEXTENSION_NODOT 2>/dev/null| head -n 1 2>/dev/null)\necho \"DEBUG: Latest BACKUPFILENAMEEXTENSION_NODOT file found on SMB mount: \$latestFileOnSMB\" >> /var/tmp/scriptd.out\n\nif \[ \"X\"\${latestFileOnSMB} \!= \"X\" \]\n\tthen\n\tsum1=\$(md5sum ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT | awk '{print \$1}')\n\tsum2=\$(md5sum \${latestFileOnSMB} | awk \'{print \$1}\')\n\tif \[ \${sum1} == \${sum2} \]\n\tthen\n\t\techo \"ERROR: File ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT already exists in //\${server}/\${msshare}\${mssubdir}\" >> /var/tmp/scriptd.out\n\t\tumount \${mountp}\n\t\trm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT*\n\t\texit 1\n\telse\n\t\techo \"DEBUG: File ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT does not already exist in //\${server}/\${msshare}\${mssubdir} (continuing...)\" >> /var/tmp/scriptd.out\n\tfi\nelse\n\techo \"DEBUG: Destination SMB mount contains no BACKUPFILENAMEEXTENSION_NODOT files (continuing...)\" >> /var/tmp/scriptd.out\nfi\ncp ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT* \${mountp}\nrm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT*\n\nif \[ \"PRUNINGMODE\" \!= \"Disabled\" \]; then\n\n\tfiles_tokeep=\$(ls -t \${mountp}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT 2>/dev/null | head -n CONSERVE\)\n\tfor current_archive_file in `ls \${mountp}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT 2>/dev/null` ; do\n\t\tcurrent_archive_file_basename=`basename \$current_archive_file`\n\t\tcheck_file=\$(echo \$files_tokeep | grep -w \$current_archive_file_basename)\n\t\tif \[ \"\$check_file\" == \"\" \] ; then\n\t\t\trm -f \$current_archive_file\n\t\tfi\n\tdone\n\tif \[ \"BACKUPFILENAMEEXTENSION_NODOT\" == \"scf\" \] ; then\n\t\ttar_files_tokeep=\$(ls -t \${mountp}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT.tar 2>/dev/null | head -n CONSERVE\)\n\t\tfor current_archive_tar_file in `ls \${mountp}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT.tar 2>/dev/null` ; do\n\t\t\tcurrent_archive_tar_file_basename=`basename \$current_archive_tar_file`\n\t\t\tcheck_file=\$(echo \$tar_files_tokeep | grep -w \$current_archive_tar_file_basename)\n\t\t\tif \[ \"\$check_file\" == \"\" \] ; then\n\t\t\t\trm -f \$current_archive_tar_file\n\t\t\tfi\n\t\tdone\n\tfi\nfi\n\numount \${mountp}\n\nexit 0\n" > $scriptfile

BR,

Erdem
THE_BLUE
Cirrostratus
Feb 15, 2023
great iApp, but is it possible to download the file localy and then delete it from GUI by itself ?
THE_BLUE
Cirrostratus
Nov 22, 2022
I'm using local backup (on this F5) and i need to modify the script to allow downloading and then delete the file. i mean, we need to to auto generate the backup then auto download it and save it localy ( will spesify the path) and then delete the file from F5. is it possible?
legan
Nimbostratus
Oct 14, 2020
My /var/local/ucs/ directory keeps flooding with UCS files that are not cleanup. Ater investigation, it appears the scriptd.log file only says:
f5.automated_backup iApp one_time_backup: STARTED
f5.automated_backup iApp one_time_backup: myunit_20201014213500.ucs GENERATING
f5.automated_backup iApp one_time_backup: myunit_20201014213500.ucs SAVED LOCALLY (/var/local/ucs)
f5.automated_backup iApp one_time_backup: myunit_20201014213500.ucs REMOTE COPY (FTP) STARTING

And the file is copied successfully, but the logging never shows that FTP was finished and the files are not cleaned. I don't see anyhting in the log file that might incidate the issue.

Any help would be appreciated.
Omar2
Cirrus
Oct 06, 2020
hello Dears ,
we were using iAPP 3.1.9 and it was working normally but after upgrading our F5 to 13.1.3.4.0.65.0 , it is not working now and we get this error :
Script (/Common/f5.automated_backup__MNS-Automatic_backup) generated this Tcl error:
script did not successfully complete: ("ucs" unexpected argument
while executing
"tmsh::save /sys ucs $fname passphrase Voda@123 " line:19)

how can i solve this issue ? is it solved in 3.2 ?
Shazee
Altocumulus
Sep 23, 2020
This issue is fixed in 3.2 available in Git.
changed to new template and the SFTP copy worked.
Shazee
Altocumulus
Sep 22, 2020
how can i make sure that this uses SFTP and not SCP. this is creating a local backup but failing to push with SFTP.

these are the steps i have performed

Generated key pair with public key assigned to the user in S3 SFTP server
ssh-keygen -t rsa -b 4096 -C f5_backups

from command line i am able to access the SFTP server and list and put/mput the files

ucs # sftp -i id_rsa f5-backup-test@10.164.2.31
Connected to 10.164.2.31.
sftp> ls
test

from the iApp script in th e/var/tmp/scriptd.out gives this logs

Bckup iApp admin: ip-10-164-2-49..ucs SAVED LOCALLY (/var/local/ucs)
f5.automated_backup iApp admin: ip-10-164-2-49..ucs REMOTE COPY (SCP) STARTING

which uses SCP..

is there a way to specifically make it SFTP transfer,

or any idea why this is failing,

i have copied the private and public files from location /root/.ssh to /var/ssh/root/ locatins as well (both the locations have the private and public key pair.)

/var/ssh/root/authorized_keys file is empty

and authorised

This is using SCP rather than SFTP.
Shazee
Altocumulus
Sep 04, 2020
Thank you for the information. Really useful.
Appreciate mate 👌
paulfish
Nimbostratus
Sep 03, 2020
Hi Shazee, I can confirm SFTP on 14.1.2.6 to REHL6. I can also confirm I'm upgrading from 14.1.2.6 to 14.1.2.7 for a memory leak issue. One pair is using ASM and support are tracking the issue with me. Second pair is LTM only and the UI is slowly becoming more unresponsive. I have not looked deeper or opened a ticket on it.
Shazee
Altocumulus
Sep 03, 2020
Hi guys,
Has anyone managed to confirm 14.1.2.6 version to get connected using SFTP protocol to the AWS Transfer for SFTP service plase