cancel
Showing results for 
Search instead for 
Did you mean: 
Thomas_Schocka1
Altocumulus
Altocumulus

Hi all,

 

Often I've been scouring the devcentral fora and codeshares to find that one piece of handywork that will drastically simplify my automated backup needs on F5 devices. Based on the works of Jason Rahm in his post "Third Time's the Charm: BIG-IP Backups Simplified with iCall" on the 26th of June 2013, I went ahead and created my own iApp that pretty much provides the answers for all my backup-needs.

Here's a feature list of this iApp:

  • It allows you to choose between both UCS or SCF as backup-types. (whilst providing ample warnings about SCF not being a very good restore-option due to the incompleteness in some cases)
  • It allows you to provide a passphrase for the UCS archives (the standard GUI also does this, so the iApp should too)
  • It allows you to not include the private keys (same thing: standard GUI does it, so the iApp does it too)
  • It allows you to set a Backup Schedule for every X minutes/hours/days/weeks/months or a custom selection of days in the week
  • It allows you to set the exact time, minute of the hour, day of the week or day of the month when the backup should be performed (depending on the usefulness with regards to the schedule type)
  • It allows you to transfer the backup files to external devices using 4 different protocols, next to providing local storage on the device itself
    • SCP (username/private key without password)
    • SFTP (username/private key without password)
    • FTP (username/password)
    • SMB (using smbclient, with username/password)
    • Local Storage (/var/local/ucs or /var/local/scf)
  • It stores all passwords and private keys in a secure fashion: encrypted by the master key of the unit (f5mku), rendering it safe to store the backups, including the credentials off-box
  • It has a configurable automatic pruning function for the Local Storage option, so the disk doesn't fill up (i.e. keep last X backup files)
  • It allows you to configure the filename using the date/time wildcards from the tcl [clock] command, as well as providing a variable to include the hostname
  • It requires only the WebGUI to establish the configuration you desire
  • It allows you to disable the processes for automated backup, without you having to remove the Application Service or losing any previously entered settings
  • For the external shellscripts it automatically generates, the credentials are stored in encrypted form (using the master key)
  • It allows you to no longer be required to make modifications on the linux command line to get your automated backups running after an RMA or restore operation
  • It cleans up after itself, which means there are no extraneous shellscripts or status files lingering around after the scripts execute

I wasn't able to upload the iApp template to this article, so I threw it on pastebin: http://pastebin.com/YbDj3eMN

 

Enjoy!

Thomas Schockaert

Comments
Mohamed_Lrhazi
Altocumulus
Altocumulus
Thank you so much! Super cool.
Hi!

 

 

Could someone tell me how to import this in my F5's

 

 

Gavin_Connell-O
Nimbostratus
Nimbostratus
Is this supposed to be saved as a .tmpl file and then imported? Created manually by creating a new iapp template? Tried both and not got anywhere. Bit of a pointer please? 🙂
Gavin_Connell-O
Nimbostratus
Nimbostratus
Is this only for Big-IP 11.4 and above?
Mohamed_Lrhazi
Altocumulus
Altocumulus
Yes, it seems to require 11.4 and above. Save the source to a file, import it as a template, then create a new iApp based off of the template.
Gavin_Connell-O
Nimbostratus
Nimbostratus
Thanks Mohamed, I'll have to upgrade to 11.4 first 🙂 Only just finished settling down on 11.3! Keeps me in work I suppose!
Thomas_Schocka1
Altocumulus
Altocumulus
It requires v11.4 because of the iCall feature to do the scheduling. It should be possible to create something that is capable of writing the scheduling information to a crontab entry instead of doing it via iCall too.
gvo_142555
Nimbostratus
Nimbostratus
I get the following errog:

 

Ending iApp /Common/f5.automated_backup.v2.0 03/20/2014 17:37:36

 

Run time 240 msec

 

 

Saving active configuration...

 

/var/local/ucs/bigipA.local_20140320_173737.ucs is saved.

 

Script (/Common/f5.automated_backup) generated this Tcl error:

 

script did not successfully complete: (Permission denied, please try again.

 

Permission denied, please try again.

 

Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

 

lost connection

 

while executing

 

"exec $scriptfile" line:18)

 

 

 

Where can I find the SSH Private Key to add in the iApp?

 

Thanks and kind regards
Thomas_Schocka1
Altocumulus
Altocumulus
Hi,

 

 

You can find the private key by first generating a public/private keypair for SSH by using the command 'sshkeygen -t rsa'.

 

 

This will save id_rsa and id_rsa.pub in ~/.ssh.

 

You will then need to copy the contents of id_rsa.pub to the authorized_keys file on the target server and copy the contents of id_rsa into the iApp as the private key.

 

 

I hope this helps,

 

 

Kind regards,

 

 

Thomas
gvo_142555
Nimbostratus
Nimbostratus
Hi Thomas,

 

 

Thanks for the comment. The public key is stored on the target server and I added the content of the private key in the iApp.

 

 

When performing a scp from the LTM towards the target server using SSH this seems to work.

 

 

Have I some other way to troubleshoot this issue?

 

 

Thanks and kind regards

 

 

 

Gert
Thomas_Schocka1
Altocumulus
Altocumulus
Hi Gert,

 

 

Does it also work when you specify the private key file (which you upload manually, in unencrypted format, to the F5 first, for testing purposes only) on the commandline while doing an ssh command?

 

 

Steps:

 

1) copy contents of private key file to a file on the F5

 

2) run the command "ssh -i /path/to/private/key/file myusername@10.1.2.3"

 

 

If the SSH-server on 10.1.2.3 is configured to look for an authorized_keys file under the user's home directory, the file /home/myusername/.ssh/authorized_keys should contains, on a single line, the contents of the public key part (id_rsa.pub).

 

If you uploaded the corresponding private key part, and you specified on the command line where it it found, it should normally log you in without asking for a password. This tells you that your privatekey+publickey ssh authentication is working fine server-wise and for this keypair. This also tells you that the problem lies with the way the iApp tries to do it.

 

 

The iApp actually reads the field for the private key, which does not contain any newlines due to the field being a textbox and the private key being copied into it from your paste buffer. The iApp then puts newlines where needed, so the scp command has access to a "decent" private key file. It stores this under /config/, but it does so in encrypted format, using the F5 Unit Master Key as salt. You can decrypt this file as such, which should yield no error and give you a decent private key file:

 

 

sed -n '4,$p' /config/f5.automated_backup_scp.conf | openssl aes-256-ecb -salt -a -d -k $(f5mku -K)

 

 

Kind regards,

 

 

Thomas
daemon8814_1288
Nimbostratus
Nimbostratus
How do you configure the SMB piece? What format does an AD username have to be in? What format does the file path have to be in? Thanks.
daemon8814_1288
Nimbostratus
Nimbostratus
I am getting the following error on the during creation of the iApp:

 

 

01070712:3: Can't load node: 139 type: 2 - ltm/validation/MonitorRule.cpp, line 2114
Dan_L1
Nimbostratus
Nimbostratus
I've followed all the comments here trying to get this working; with running the sed -n command you put I get the output:

 

 

bad magic number

 

 

I can successfully ssh from the f5 > destination defining the id_rsa without a problem. When it tries to run it fails "permission denied"

 

 

Script (/Common/f5.automated_backup) generated this Tcl error:

 

script did not successfully complete: (bad decrypt

 

10933:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:330:

 

bad magic number

 

 

Any ideas? Thanks!
PatrickG
Nimbostratus
Nimbostratus
The iApp is very nice, but how can i use it in an redundant environment? The iApp configuration is synchronized between two nodes, means only one node performs an Backup
Woody_
Nimbostratus
Nimbostratus
Thanks for this code. It worked in LTM training today (11.4) but errored out when I attempted to import it into our LTM (11.2.1). I'll be using this when I upgrade to 11.5.
mr_evil_116524
Nimbostratus
Nimbostratus
Hi Thomas,

 

 

Can you pleas let me know how do I copy the content of the private key? Do I cat /root/.ssh/id_rsa from F5 CLI and paste that in iAPP GUI txtbox? or do I need to hardcore that somewhere else?

 

 

Please help, Thanks
lnxgeek
Cirrostratus
Cirrostratus
Hi Thomas

 

Great work with the iApp!

 

 

I think I found one "weakness" in the way you encryption (using openssl with the -a flag). If you have more than 30 characters in either username, path or server, it will break over two lines due to the base64 conversion. This will make the "sed" logic fail as it assumes the top three objects to be one-liners.

 

 

Maybe you could put in some marker in between them?

 

 

Regards

 

Thomas
lnxgeek
Cirrostratus
Cirrostratus
@mr.evil - you're right, you just copy the id_rsa into the "private key" text box. The script logic takes care of the rest.

 

 

Regards

 

Thomas
lnxgeek
Cirrostratus
Cirrostratus
I created an updated version of the iApp which handles longer than 30 characters strings.

 

For the time being it is only "SCP" that I modified.

 

Get it here:

 

http://download.domingo.dk/data/public/bc4c17.php
annielee_13548
Nimbostratus
Nimbostratus
Can I have 2 backup using this iApp ? one for locally and one for remotely ? I tried but got an error.. so can i confirm that cant be done ?
annielee_13548
Nimbostratus
Nimbostratus
Hi Thomas, I've used your iApp on our F5 and it worked successfully. And if possible can you help me with modifying the script as i need the UCS file to be compressed to smaller size (as the smallest one we have is about 25MB) Thanks..
Greg_Crosby_319
Historic F5 Account
Nice script Thomas! I had troubles when using ftp because my transfer was using ascii mode. Adding \nbinary to force a binary transfer into the iCall script procedure which creates the temporary ftp.sh script file resolved my problem. Thought it would be worth mentioning.
whswhswhs124_98
Nimbostratus
Nimbostratus
'
Evan_25555
Historic F5 Account
Has anyone gotten the SMB piece to work? I'm getting errors similar to what Dan L reported some months ago. (bad decrypt 27666:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:330: while executing "exec $scriptfile" line:17))
lnxgeek
Cirrostratus
Cirrostratus
I've made an other version of the iApp which also fixes length problems when using SMB. Find it here: http://download.domingo.dk/data/public/c04a2a.php
Jeff_121982
Nimbostratus
Nimbostratus
Has anyone integrated alerting into this script? It would be nice to be able to receive some type of notification that the backups occurred as scheduled and were copied to the remote repository.
Peter_J_146842
Nimbostratus
Nimbostratus
Has anyone got this to work 100% properly? I am not able to restore from FTP. I have no problem with making a job FTP the UCS file (without passphrase) to a FTP server. But when I download it back to the F5, and try to restore it from the file, it fails with: Saving active configuration... Current configuration backed up to /var/local/ucs/cs_backup.ucs. tar: Skipping to next header tar: Archive contains obsolescent base-64 headers gzip: stdin: invalid compressed data--format violated tar: Child returned status 1 tar: SPEC-Files: Not found in archive tar: Error exit delayed from previous errors Fatal: executing: tar -zxf /var/local/ucs/20140919_backup_test.com.ucs SPEC-Manifest SPEC-Files Operation aborted. /var/tmp/configsync.spec: Error installing package Config install aborted. Unexpected Error: UCS loading process failed. I am not sure what this mean, but I have notices that the version is “unknown” when opening the backup file on the F5. Furthermore, if I set the job to save on F5 instead of FTP, I can restore it without problem, and the version is now shown right version. Please advice. P.S. This is testet in both version 2.0, 2.0.1-tdd and 2.0.2-tdd
Nfordhk_66801
Nimbostratus
Nimbostratus
Does anyone have instructions how to set this up?
lkchen
Nimbostratus
Nimbostratus
These are new 7200v's, not yet in production. They had initially come as v11.4.1, we had contracted a PS to get 3 vCMPs configured along with some initial configuration of LTM/GTM/AFM. We had discussions with them about doing the system with v11.5.1 (since there was interest in doing forward proxying on the LTM rather than load balancing to forward proxy servers behind it.) We were strong recommended to not do it, as they considered 11.5.1 too buggy. But, we pushed for it anyways. Though seems there have been other useful things from 11.5.1, like being able to turn appliance mode on for the vCMPs (and being able to turn it off should we get stuck looking on finding a tmsh way to accomplish the task...) Though not soon enough before a different consultant had edited bigip_base.conf to fix a typo that was made during the initial setup using configuration utility. Or finding the tmsh way (which I had done a couple weeks earlier to adjust one of the ip ranges...but not the one that had typo, had entered a range in two places as x.x.x.6/29 when it should've been x.x.x.16/29.) Don't know if the 11.5.1 was done as an upgrade from 11.4.1 or as a new install, I was in the hospital that day, and they were in hurry to get PS working on it, though networking wasn't done until the end of the second day....for only one of the 3 vCMP pairs, and the failover link still hasn't been done.) The only big upgrade I've even done was from 9.3.1 to 10.2.3 (that was a scary 1am upgrade....done a little over 2 months 9.3.1 was to EOL.)
lnxgeek
Cirrostratus
Cirrostratus
I've made a third edition. http://download.domingo.dk/data/public/backup-iapp.php
THi
Altocumulus
Altocumulus
Hi tried to load this in to 11.6.0 (both HF1 & HF3). When importing the template, it loads, but LTM log gets the following warning: Jan 29 12:20:54 beta warning mcpd[6837]: 01071859:4: Warning generated : /Common/f5.automated_backup.v2.0.3-tdd definition:234: warning: [use curly braces to avoid double substitution][$everyxminutes*60] /Common/f5.automated_backup.v2.0.3-tdd definition:243: warning: [use curly braces to avoid double substitution][$everyxhours*3600] /Common/f5.automated_backup.v2.0.3-tdd definition:253: warning: [use curly braces to avoid double substitution][$everyxdays*86400] /Common/f ...... So is this intentional or is there something wrong in the syntax?
Abdessamad1
Cirrostratus
Cirrostratus
Thanks a lot Thomas for sharing this iApp. THi , the error in v11.6.0 can be resolved by adding braces to each "expr" command ( for exp: expr {$everyxminutes*60} ).
Max_Q_factor_12
Cirrocumulus
Cirrocumulus
Thomas, this is a great iapp, is there anyway you can add some object variables so we can have multiple iApp instances on a box? say one for local backups and one for remote backups?
jdcarp_179627
Nimbostratus
Nimbostratus
I had issues targeting backups to an SMB file server in a Active Directory domain. So I modified this template to request allow users to specify the workgroup -- https://dl.dropboxusercontent.com/u/43115505/f5-iapps/f5.automated_backup/f5.automated_backup.v2.0.4-jc.zip. ** Warning: This does NOT work with Windows 2012 R2 file servers. The template will write a 0 KB .ucs files. My guess is this is due to the old version of smbclient 3.0.33 that is included in the BIG-IP software releases. It is up to F5 to update smbclient.
B1r0
Altostratus
Altostratus
Very good thanks... saved me a lot of troubles. Is there a way to backup only if there has been an actual change in the configuration?
B1r0
Altostratus
Altostratus
Sorry guys. I was configuring a second UCS backup schedule to be run weekly instead of the daily SCF one, but I couldn't save it as it was returning this message: 010715bc:3: The application service (/Common/SConf_Backup_to_server.app/SConf_Backup_to_server) has strict updates enabled, the object (icall periodic handler /Common/f5.automated_backup-handler) must be updated using an application management interface. How can I change the template so it creates a new handler for each new iApp? Also adding a check that monitors if there were any actual changes. I did in the past my own script run by a cron event, but it got removed when I upgraded from 11.4.x to 11.6.x Thanks, Roberto
TacticalDragon_
Nimbostratus
Nimbostratus
Hello Thomas, I just found your script, it's pretty awesome from what I see here. I am having trouble setting up sftp from my F5 to a cifs share. Currently my backup files live under ./var/local/ucs and my cifs share is under \\servername\sharename$ I have a private public key already created.
sachin_80710
Nimbostratus
Nimbostratus
Hi, somebody please me in how to start using this code. There no steps on to use it. Copy complete code then ? Thanks, Sachin
coriolis_75734
Nimbostratus
Nimbostratus
Anyone come across this error? (script did not successfully complete: (Host key verification failed. lost connection while executing "exec $scriptfile" line:18)) I've copied the private key into the iApp and placed the public key in the right place on the FTP server? I've also successfully connected using sftp and specifying the file manually.
Joe_5599_134300
Nimbostratus
Nimbostratus
How can I change the default time script keeps trying to prune old backup ucs files? I see in the Audit log every minute seems to run.
Joe_5599_134300
Nimbostratus
Nimbostratus
I have change this in the iApp from default 60 to 3600 set script [string map [list CONSERVE $prune_conserve] $script] iapp::conf create sys icall script f5.automated_backup_pruning definition \{ $script \} app-service none set cdate [clock format [clock seconds] -format "%Y-%m-%d:%H:%M"] iapp::conf create sys icall handler periodic f5.automated_backup_pruning-handler \{ \ interval 3600 \
Roy_van_Dongen_
Nimbostratus
Nimbostratus
Hi! I have updated the FTP statement with the Binary toggle. There is only one bug left as far as i'm concerned. On an HA pair, the secondary device does not run this iApp succesfully resulting in /var running low on diskspace. This is because the f5.automated_backup_iapp file is not synchronised. I have no idea how to solve this. ( FTP Binary patch ) : http://pastebin.com/DYRszZE4
datago_205305
Nimbostratus
Nimbostratus
The pastebin link no longer works. Please repost.
MichealRP_61305
Nimbostratus
Nimbostratus
I realize that this may be old school and not as nifty, but what I do with ours is to set up a remote nfs mount point in the /etc/fstab as you would with most any *nix OS's, set it to noauto so if the system can't reach it on a reboot it won't hang, have a scrpt that is run via crontab that mounts the partition, the scf / ucs, the umount the partition. I mean they are Linux after all and have almost all of the functions of a regular *nix host.
Mark_Wolzak
Nimbostratus
Nimbostratus
This is excellent work! So far this iApp runs fine on our HA pair. @Roy van Dongen, on a HA pair the catch is to reconfigure and re-apply the iApp/Application Service on the second node after configuring it on the first node. Hope this helps.
Is there any update for Firmware 12.1.0? Because i just installed the new software on my F5-Lab but as i reconfigured the iApp i saw that the file /config/f5.automated_backup_smb.conf is empty Do you have any idea why this happend?
Sylvain_Q
Nimbostratus
Nimbostratus
After the SFTP scheduled backup runs, it doesn't seems to delete the file in /var/local/ucs folder. Is there a way to be sure that the deletion is in place? I have this message in the log file Script (/Common/f5.automated_backup) generated this Tcl error: (script did not successfully complete: (+-----------------------------------------------------------------------+ | WARNING | | ------- | | | | The programs and data stored on this system are licensed to or are | | private property of this company and are lawfully available only to | | authorized users for approved purposes. Unauthorized access to any | | program or data on this system is not permitted, and any unauthorized | | access beyond this point may lead to prosecution. This system may be | | monitored at any time for operational reasons, therefore, if you are | | not an authorized user, | | DO NOT ATTEMPT TO LOG IN. | |
Mark_Burrows
Nimbostratus
Nimbostratus

How can this be used on a HA pair using the scp and ssh keys method? If you create it on node1 and sync to node2, then update the key on node2, node1 will blow it away the next time you sync.

 

tatmotiv
Cirrus
Cirrus

I'm running into the same problem, syncing between two HA members renders this iApp inoperational on the standby node. Even when using the same key for both machines, I always end up with only one ucs on the remote (SCP) server. reconfigure and re-apply the iApp/Application Service on the second node after configuring it on the first node as suggested above also won't help. Has anybody solved that issue? I'm running 11.6.0 HF6.

 

EDIT: After doing some analysis it appears that the standby machine (.210) fails to establish the SSH connection, whereas the active machine (.110) successfully can set it up (and transfer files over it):

 

Sep 23 12:58:05 my_upload_target sshd[58009]: Accepted publickey for lbupload from 10.x.x.110 port 40067 ssh2: RSA xxxxx [MD5] Sep 23 12:58:05 my_upload_target sshd[58040]: Connection closed by 10.x.x.210 [preauth]

Obviously, the key synchronisation between both machines seems not to work...

 

EDIT2: In order to affirm that assumption, I replaced the private key on the active machine with rubbish to see if that provokes the same logging entry. The device-group is configured with config auto-sync, so this change will also immediately get synced to the standby device. Now, when the iCall script is running, these messages are logged by the upload server:

 

Sep 23 13:30:08 my_upload_target sshd[54899]: Failed password for my_upload_user from 10.x.x.110 port 42862 ssh2 Sep 23 13:30:08 my_upload_target sshd[54899]: Connection closed by 10.x.x.110 [preauth] Sep 23 13:30:11 my_upload_target sshd[54904]: Connection closed by 10.x.x.210 [preauth]

As expected, the active machine is now rejected due to bad authentication (failed password), but the other one is not - it's just closing the connection during preauth and it seems that it does not even start any key exchange...

 

EDIT3: OK, it's actually that easy... The entry in /root/.ssh/known_hosts was missing on the failover unit, but was present in the active one. After manually establishing an ssh connection to the destination host, thus adding its public key to known_hosts, the automated backup now works for both.

 

EDIT4 (and that will probably be the last one - I hope this help anybody experiencing the same problems...): I added -o StrictHostKeyChecking=no to the scp command in the iApp definition in order avoid future trouble (e.g. after hotfix installations etc). Now it works like a charm.

 

Version history
Last update:
‎12-Mar-2014 20:34
Updated by:
Contributors