F5 iApp Automated Backup

Problem this snippet solves:

This is now available on GitHub!

Please look on GitHub for the latest version, and submit any bugs or questions as an "Issue" on GitHub:

(Note: DevCentral admin update - Daniel's project appears abandoned so it's been forked and updated to the link below. @damnski on github added some SFTP code that has been merged in as well.)

https://github.com/f5devcentral/f5-automated-backup-iapp

Intro

Building on the significant work of Thomas Schockaert (and several other DevCentralites) I enhanced many aspects I needed for my own purposes, updated many things I noticed requested on the forums, and added additional documentation and clarification. As you may see in several of my comments on the original posts, I iterated through several 2.2.x versions and am now releasing v3.0.0. Below is the breakdown!

Also, I have done quite a bit of testing (mostly on v13.1.0.1 lately) and I doubt I've caught everything, especially with all of the changes. Please post any questions or issues in the comments.

Cheers!

Daniel Tavernier (tabernarious)

v3.2.1 (20201210)

Merged v3.1.11 and v3.2.0 for explicit SFTP support (separate from SCP).
Tweaked the SCP and SFTP upload directory handling; detailed instructions are in the iApp.
Tested on 13.1.3.4 and 14.1.3

v3.1.11 (20201210)

Better handling of UCS passphrases, and notes about characters to avoid.
I successfully tested this exact passphrase in the 13.1.3.4 CLI (surrounded with single quote) and GUI (as-is): `~!@#$%^*()aB1-_=+[{]}:./?
I successfully tested this exact passphrase in 14.1.3 (square-braces and curly-braces would not work): `~!@#$%^*()aB1-_=+:./?
Though there may be situations these could work, avoid these characters (separated by spaces): " ' & | ; < > \ [ ] { } ,
Moved changelog and notes from the template to CHANGELOG.md and README.md.
Replaced all tabs (\t) with four spaces.

v3.1.10 (20201209)

Added SMB Version and SMB Security options to support v14+ and newer versions of Microsoft Windows and Windows Server.
Tested SMB/CIFS on 13.1.3.4 and 14.1.3 against Windows Server 2019 using "2.0" and "ntlmsspi"

v3.1.0:

Removed "app-service none" from iCall objects. The iCall objects are now created as part of the Application Service (iApp) and are properly cleaned up if the iApp is redeployed or deleted.
Reasonably tested on 11.5.4 HF2 (SMB worked fine using "mount -t cifs") and altered requires-bigip-version-min to match.
Fixing error regarding "script did not successfully complete: (can't read "::destination_parameters__protocol_enable": no such variable" by encompassing most of the "implementation" in a block that first checks $::backup_schedule__frequency_select for "Disable".
Added default value to "filename format".
Changed UCS default value for $backup_file_name_extension to ".ucs" and added $fname_noext.
Removed old SFTP sections and references (now handled through SCP/SFTP).
Adjusted logging: added "sleep 1" to ensure proper logging; added $backup_directory to log message.
Adjusted some help messages.

New v3.0.0 features:

Supports multiple instances! (Deploy multiple copies of the iApp to save backups to different places or perhaps to keep daily backups locally and send weekly backups to a network drive.)
Fully ConfigSync compatible! (Encrypted values now in $script instead of local file.)
Long passwords supported! (Using "-A" with openssl which reads/writes base64 encoded strings as a single line.)
Added $script error checking for all remote backup types! (Using 'catch' to prevent tcl errors when $script aborts.)
Backup files are cleaned up after any $script errors due to new error checking.
Added logging! (Run logs sent to '/var/log/ltm' via logger command which is compatible with BIG-IP Remote Logging configuration (syslog). Run logs AND errors sent to '/var/tmp/scriptd.out'. Errors may include plain-text passwords which should not be in /var/log/ltm or syslog.)
Added custom cipher option for SCP! (In case BIG-IP and the destination server are not cipher-compatible out of the box.)
Added StrictHostKeyChecking=no option. (This is insecure and should only be used for testing--lots of warnings.)
Combined SCP and SFTP because they are both using SCP to perform the remote copy. (Easier to maintain!)

Original v1.x.x and v2.x.x features kept (copied from an original post):

It allows you to choose between both UCS or SCF as backup-types. (whilst providing ample warnings about SCF not being a very good restore-option due to the incompleteness in some cases)
It allows you to provide a passphrase for the UCS archives (the standard GUI also does this, so the iApp should too)
It allows you to not include the private keys (same thing: standard GUI does it, so the iApp does it too)
It allows you to set a Backup Schedule for every X minutes/hours/days/weeks/months or a custom selection of days in the week
It allows you to set the exact time, minute of the hour, day of the week or day of the month when the backup should be performed (depending on the usefulness with regards to the schedule type)
It allows you to transfer the backup files to external devices using 4 different protocols, next to providing local storage on the device itself
SCP (username/private key without password)
SFTP (username/private key without password)
FTP (username/password)
SMB (now using TMOS v12.x.x compatible 'mount -t cifs', with username/password)
Local Storage (/var/local/ucs or /var/local/scf)
It stores all passwords and private keys in a secure fashion: encrypted by the master key of the unit (f5mku), rendering it safe to store the backups, including the credentials off-box
It has a configurable automatic pruning function for the Local Storage option, so the disk doesn't fill up (i.e. keep last X backup files)
It allows you to configure the filename using the date/time wildcards from the tcl [clock] command, as well as providing a variable to include the hostname
It requires only the WebGUI to establish the configuration you desire
It allows you to disable the processes for automated backup, without you having to remove the Application Service or losing any previously entered settings
For the external shellscripts it automatically generates, the credentials are stored in encrypted form (using the master key)
It allows you to no longer be required to make modifications on the linux command line to get your automated backups running after an RMA or restore operation
It cleans up after itself, which means there are no extraneous shellscripts or status files lingering around after the scripts execute

How to use this snippet:

Find and download the latest iApp template on GitHub (e.g "f5.automated_backup.v3.2.1.tmpl.tcl").
Import the text file as an iApp Template in the BIG-IP GUI.
Create an Application Service using the imported Template.
Answer the questions (paying close attention to the help sections).
Check /var/tmp/scriptd.out for general logs and errors.

Tested this on version:

16.0

Updated Feb 15, 2022

Version 2.0

Cirrostratus

Joined December 19, 2012

View Profile

Daniel_Tavernie
Cirrostratus
May 29, 2018
@PhillyPDXmike, As long as you use the ${host} variable in the file name iApp setting (this is included in the default file name setting) it will generate archives with unique names per device (regardless of clustering). Are there other settings you are hoping to set independently?
PhillyPDXmike_1
Nimbostratus
May 29, 2018
@tabernarious, Thanks for the quick response!! I am using version 3.1.2 of the iApp and can confirm that the backup worked on both the active and standby units after I copied the public/private key from the active to the standby, edited the hostname in the public key (/root/.ssh/id_rsa.pub) on the standby, then added that public key to my sftp server's authorized_keys file.

My initial question was prompted from a cosmetic situation where the sample output for the "Select the filename format > Destination Parameters" setting carried over from the active appliance (dev02a) to the standby appliance (dev02b).

Your question: "Are there other settings you are hoping to set independently?" Since the UCS backup is specific to each appliance, it'd be nice to somehow have device independence within the iApp which is probably more of an F5 iApp software architecture thing than anything to do with this specific iApp. What are your thoughts?
Daniel_Tavernie
Cirrostratus
May 29, 2018
I’m glad it’s working, though I don’t think you should have had to mess with the keys (copying between boxes)—I’ll test that too. I see what you mean about the cosmetic file name example issue. I’ll have to think about if that can be fixed.

As for your question about the overall architecture, I definitely agree that there are situations when it would be nice to keep apps independent, but I think the vast majority of cases support the existing keep-everything-in-sync model.
aj11
Nimbostratus
May 30, 2018
Will this work in version 13.1.x also?
Daniel_Tavernie
Cirrostratus
May 30, 2018
@aj1, This should definitely work on 13.1.x. Most of my recent testing has been on 13.1.0.x.

keithhubb

Employee

May 31, 2018

Hi,

I'm using v3.1.3, and getting this message:

script did not successfully complete: (can't read "::destination_parameters__pruning_mode": no such variable
while executing
"if { $::destination_parameters__pruning_mode eq "Only Prune iApp-Generated Archives" } {
set pruning_suffix $::destination_par..."
invoked from within
"if { $freq != "Disable" } {
Ensure a default $filename_format is set
if { $::destination_parameters__filenam..." line:44)

I Fixed it by changing line 54 from this:

if { $::destination_parameters__pruning_mode eq "Only Prune iApp-Generated Archives" } {

To this:

if { [info exists ::destination_parameters__pruning_mode] && ($::destination_parameters__pruning_mode eq "Only Prune iApp-Generated Archives") } {

Basically validating the variable exists as a condition to the 'if' statement.

aj11
Nimbostratus
Jun 01, 2018
I'm having some issues with authentication: In the logs, it shows the warning banner on the host I'm trying to send the backup to, but then shows this:

Permission denied, please try again. Permission denied, please try again. Received disconnect from xx.xx.xx.xx: 2: Too many authentication failures for backuphost lost connection f5.automated_backup iApp automated_backups_iapp:loadbalancer.com_20180601_085600.ucs REMOTE COPY (SCP) FAILED (check for errors above) f5.automated_backup iApp automated_backups_iapp: FINISHED

We tried to enter the username/password, and I also copied the private key from the backuphost but still the same. Also, I checked /var/local/ucs for the local copy it would (should?) be copying over but there is only a "config.ucs" file (~70M) and the file's last dated a few weeks ago when we first installed it, I believe. It's been running every few minutes but failing and I thought it was just the copy part that fails. Any guidance would help. Thanks.
PhillyPDXmike_1
Nimbostratus
Jun 01, 2018
In my testing, using the backup files via SCP/SFTP option doesn't save a copy locally.

For the "Copy/Paste the SSH private key to be used for passwordless authentication:" option, I copied/pasted the entire RSA private key including -----BEGIN RSA PRIVATE KEY-----, -----END RSA PRIVATE KEY-----, and everything in between. Also, I found these links F5 K13454 and tecmint article very helpful in getting the ssh passwordless authentication working properly, although I didn't have to change any settings on my linux server.

On F5 BIG-IP appliance, create 2048 bit RSA key using the following command at a bash prompt: ssh-keygen -t rsa -b 2048
Accept the default location to save the key (/root/.ssh/id_rsa) by hitting enter
Leave the passphrase blank by hitting enter then hit enter again.
The private key will be save as /root/.ssh/id_rsa and the public key as /root/.ssh/id_rsa.pub
If necessary, create the .ssh directory on the SSH server using the command: ssh user@X.X.X.X mkdir -p .ssh
Copy the public key from the F5 to your SSH server using the following using the command: cat /root/.ssh/id_rsa.pub | ssh user@X.X.X.X 'cat >> .ssh/authorized_keys'
If necessary, change the permission on the .ssh directory and authorized_keys file using the command: ssh username@ip.address "chmod 700 .ssh; chmod 640 .ssh/authorized_keys"
Daniel_Tavernie
Cirrostratus
Jun 01, 2018
@aj1, Regarding your search for the local copies of the UCS Archives, the iApp automatically cleans up the local copy whether the remote copy succeeds or fails. If you want local copies I would recommend deploying a second copy of the iApp that just backs up locally and auto-prunes to prevent filling up the /var partition.
cbrandys
Nimbostratus
Jun 04, 2018
Hello, I a using v3.1.3. When I attempt to implement the script using FTP for remote copy, I receive the following error.

script did not successfully complete: (can't read "::destination_parameters__pruning_mode": no such variable while executing "if { $::destination_parameters__pruning_mode eq "Only Prune iApp-Generated Archives" } { set pruning_suffix $::destination_par..." invoked from within "if { $freq != "Disable" } { Ensure a default $filename_format is set if { $::destination_parameters__filenam..." line:44)

I am trying to understand the reason for this error. When I choose the "Select the Filename Format" drop-down, I am selecting an option that does include ${host}. Am using code 11.6.1 HF2 on my box. I saw keithhubb post and resolution but would like to understand the reason for the
text
error before I implement on prod boxes. Thank you.

F5 iApp Automated Backup

This is now available on GitHub!

Intro

Related posts:

v3.2.1 (20201210)

v3.1.11 (20201210)

v3.1.10 (20201209)

v3.1.0:

New v3.0.0 features:

Original v1.x.x and v2.x.x features kept (copied from an original post):

How to use this snippet:

Tested this on version:

ABOUT DEVCENTRAL

RESOURCES

SUPPORT

PARTNERS