F5 Automated Backups - The Right Way
Hi all,
Often I've been scouring the devcentral fora and codeshares to find that one piece of handywork that will drastically simplify my automated backup needs on F5 devices. Based on the works of Jason Rahm in his post "Third Time's the Charm: BIG-IP Backups Simplified with iCall" on the 26th of June 2013, I went ahead and created my own iApp that pretty much provides the answers for all my backup-needs.
Here's a feature list of this iApp:
- It allows you to choose between both UCS or SCF as backup-types. (whilst providing ample warnings about SCF not being a very good restore-option due to the incompleteness in some cases)
- It allows you to provide a passphrase for the UCS archives (the standard GUI also does this, so the iApp should too)
- It allows you to not include the private keys (same thing: standard GUI does it, so the iApp does it too)
- It allows you to set a Backup Schedule for every X minutes/hours/days/weeks/months or a custom selection of days in the week
- It allows you to set the exact time, minute of the hour, day of the week or day of the month when the backup should be performed (depending on the usefulness with regards to the schedule type)
- It allows you to transfer the backup files to external devices using 4 different protocols, next to providing local storage on the device itself
- SCP (username/private key without password)
- SFTP (username/private key without password)
- FTP (username/password)
- SMB (using smbclient, with username/password)
- Local Storage (/var/local/ucs or /var/local/scf)
- It stores all passwords and private keys in a secure fashion: encrypted by the master key of the unit (f5mku), rendering it safe to store the backups, including the credentials off-box
- It has a configurable automatic pruning function for the Local Storage option, so the disk doesn't fill up (i.e. keep last X backup files)
- It allows you to configure the filename using the date/time wildcards from the tcl [clock] command, as well as providing a variable to include the hostname
- It requires only the WebGUI to establish the configuration you desire
- It allows you to disable the processes for automated backup, without you having to remove the Application Service or losing any previously entered settings
- For the external shellscripts it automatically generates, the credentials are stored in encrypted form (using the master key)
- It allows you to no longer be required to make modifications on the linux command line to get your automated backups running after an RMA or restore operation
- It cleans up after itself, which means there are no extraneous shellscripts or status files lingering around after the scripts execute
I wasn't able to upload the iApp template to this article, so I threw it on pastebin: http://pastebin.com/YbDj3eMN
Enjoy!
Thomas Schockaert
Published Mar 13, 2014
Version 1.0
Thomas_Schocka1
Altocumulus
AltocumulusThomas_Schocka1Altocumulus
Altocumulus
Joe_5599_134300Nimbostratus
How can I change the default time script keeps trying to prune old backup ucs files? I see in the Audit log every minute seems to run.- Joe_5599_134300I have change this in the iApp from default 60 to 3600 set script [string map [list CONSERVE $prune_conserve] $script] iapp::conf create sys icall script f5.automated_backup_pruning definition \{ $script \} app-service none set cdate [clock format [clock seconds] -format "%Y-%m-%d:%H:%M"] iapp::conf create sys icall handler periodic f5.automated_backup_pruning-handler \{ \ interval 3600 \
Roy_van_Dongen_Hi! I have updated the FTP statement with the Binary toggle. There is only one bug left as far as i'm concerned. On an HA pair, the secondary device does not run this iApp succesfully resulting in /var running low on diskspace. This is because the f5.automated_backup_iapp file is not synchronised. I have no idea how to solve this. ( FTP Binary patch ) : http://pastebin.com/DYRszZE4
datago_205305The pastebin link no longer works. Please repost.- MichealRP_61305I realize that this may be old school and not as nifty, but what I do with ours is to set up a remote nfs mount point in the /etc/fstab as you would with most any *nix OS's, set it to noauto so if the system can't reach it on a reboot it won't hang, have a scrpt that is run via crontab that mounts the partition, the scf / ucs, the umount the partition. I mean they are Linux after all and have almost all of the functions of a regular *nix host.
Mark_WolzakThis is excellent work! So far this iApp runs fine on our HA pair. @Roy van Dongen, on a HA pair the catch is to reconfigure and re-apply the iApp/Application Service on the second node after configuring it on the first node. Hope this helps.
CirrusCirrus
Is there any update for Firmware 12.1.0? Because i just installed the new software on my F5-Lab but as i reconfigured the iApp i saw that the file /config/f5.automated_backup_smb.conf is empty Do you have any idea why this happend?
Sylvain_QAfter the SFTP scheduled backup runs, it doesn't seems to delete the file in /var/local/ucs folder. Is there a way to be sure that the deletion is in place? I have this message in the log file Script (/Common/f5.automated_backup) generated this Tcl error: (script did not successfully complete: (+-----------------------------------------------------------------------+ | WARNING | | ------- | | | | The programs and data stored on this system are licensed to or are | | private property of this company and are lawfully available only to | | authorized users for approved purposes. Unauthorized access to any | | program or data on this system is not permitted, and any unauthorized | | access beyond this point may lead to prosecution. This system may be | | monitored at any time for operational reasons, therefore, if you are | | not an authorized user, | | DO NOT ATTEMPT TO LOG IN. | |
Mark_BurrowsHow can this be used on a HA pair using the scp and ssh keys method? If you create it on node1 and sync to node2, then update the key on node2, node1 will blow it away the next time you sync.
tatmotivCirrostratus
I'm running into the same problem, syncing between two HA members renders this iApp inoperational on the standby node. Even when using the same key for both machines, I always end up with only one ucs on the remote (SCP) server. reconfigure and re-apply the iApp/Application Service on the second node after configuring it on the first node as suggested above also won't help. Has anybody solved that issue? I'm running 11.6.0 HF6.
EDIT: After doing some analysis it appears that the standby machine (.210) fails to establish the SSH connection, whereas the active machine (.110) successfully can set it up (and transfer files over it):
Sep 23 12:58:05 my_upload_target sshd[58009]: Accepted publickey for lbupload from 10.x.x.110 port 40067 ssh2: RSA xxxxx [MD5] Sep 23 12:58:05 my_upload_target sshd[58040]: Connection closed by 10.x.x.210 [preauth]Obviously, the key synchronisation between both machines seems not to work...
EDIT2: In order to affirm that assumption, I replaced the private key on the active machine with rubbish to see if that provokes the same logging entry. The device-group is configured with config auto-sync, so this change will also immediately get synced to the standby device. Now, when the iCall script is running, these messages are logged by the upload server:
Sep 23 13:30:08 my_upload_target sshd[54899]: Failed password for my_upload_user from 10.x.x.110 port 42862 ssh2 Sep 23 13:30:08 my_upload_target sshd[54899]: Connection closed by 10.x.x.110 [preauth] Sep 23 13:30:11 my_upload_target sshd[54904]: Connection closed by 10.x.x.210 [preauth]As expected, the active machine is now rejected due to bad authentication (failed password), but the other one is not - it's just closing the connection during preauth and it seems that it does not even start any key exchange...
EDIT3: OK, it's actually that easy... The entry in /root/.ssh/known_hosts was missing on the failover unit, but was present in the active one. After manually establishing an ssh connection to the destination host, thus adding its public key to known_hosts, the automated backup now works for both.
EDIT4 (and that will probably be the last one - I hope this help anybody experiencing the same problems...): I added
to the scp command in the iApp definition in order avoid future trouble (e.g. after hotfix installations etc). Now it works like a charm.-o StrictHostKeyChecking=no