F5 Automated Backups - The Right Way
Hi all, Often I've been scouring the devcentral fora and codeshares to find that one piece of handywork that will drastically simplify my automated backup needs on F5 devices. Based on the works of Jason Rahm in his post "Third Time's the Charm: BIG-IP Backups Simplified with iCall" on the 26th of June 2013, I went ahead and created my own iApp that pretty much provides the answers for all my backup-needs. Here's a feature list of this iApp: It allows you to choose between both UCS or SCF as backup-types. (whilst providing ample warnings about SCF not being a very good restore-option due to the incompleteness in some cases) It allows you to provide a passphrase for the UCS archives (the standard GUI also does this, so the iApp should too) It allows you to not include the private keys (same thing: standard GUI does it, so the iApp does it too) It allows you to set a Backup Schedule for every X minutes/hours/days/weeks/months or a custom selection of days in the week It allows you to set the exact time, minute of the hour, day of the week or day of the month when the backup should be performed (depending on the usefulness with regards to the schedule type) It allows you to transfer the backup files to external devices using 4 different protocols, next to providing local storage on the device itself SCP (username/private key without password) SFTP (username/private key without password) FTP (username/password) SMB (using smbclient, with username/password) Local Storage (/var/local/ucs or /var/local/scf) It stores all passwords and private keys in a secure fashion: encrypted by the master key of the unit (f5mku), rendering it safe to store the backups, including the credentials off-box It has a configurable automatic pruning function for the Local Storage option, so the disk doesn't fill up (i.e. keep last X backup files) It allows you to configure the filename using the date/time wildcards from the tcl [clock] command, as well as providing a variable to include the hostname It requires only the WebGUI to establish the configuration you desire It allows you to disable the processes for automated backup, without you having to remove the Application Service or losing any previously entered settings For the external shellscripts it automatically generates, the credentials are stored in encrypted form (using the master key) It allows you to no longer be required to make modifications on the linux command line to get your automated backups running after an RMA or restore operation It cleans up after itself, which means there are no extraneous shellscripts or status files lingering around after the scripts execute I wasn't able to upload the iApp template to this article, so I threw it on pastebin: http://pastebin.com/YbDj3eMN Enjoy! Thomas Schockaert
F5 Archiver Ansible Playbook
Problem this snippet solves: Centralized scheduled archiving (backups) on F5 BIG-IP devices are a pain however, in the new world of Infrastructure as Code (IaC) and Super-NetOps tools like Ansible can provide the answer. I have a playbook I have been working on to allow me to backup off box quickly, UCS files are saves to a folder names tmp under the local project folder, this can be changed by editing the following line in the f5Archiver.yml file: dest: "tmp/{{ inventory_hostname }}-{{ date['stdout'] }}.ucs" The playbook can be run from a laptop on demand or via some scheduler (like cron ) or as part of a CI/CD pipelines. How to use this snippet: F5 Archiver Ansible Playbook Gitlab: StrataLabs: AnsibleF5Archiver Overview This Ansible playbook takes a list of F5 devices from a hosts file located within the inventory directory, creates a UCS archive and copies locally into the 'tmp' direcotry. Requirements This Ansible playbook requires the following: * ansible >= 2.5 * python module f5-sdk * F5 BIG-IP running TMOS >= 12 Usage Run using the ansible-playbook command using the inventory -i option to use the invertory directory instead of the default inventory host file. NOTE: F5 username and password are not set in the playbook and so need to be passed into the playbook as extra variables using the --extra-vars option, the variables are f5User for the username and f5Pwd for the password. The below examples use the default admin:admin . To check the playbook before using run the following commands ansible-playbook -i inventory --extra-vars "f5User=admin f5Pwd=admin" f5Archiver.yml --syntax-check ansible-playbook -i inventory --extra-vars "f5User=admin f5Pwd=admin" f5Archiver.yml --check Once happy run the following to execute the playbook ansible-playbook -i inventory --extra-vars "f5User=admin f5Pwd=admin" f5Archiver.yml Tested this on version: 12.1Download a BIG-IP UCS archive with "curl".
Problem this snippet solves: Download a BIG-IP UCS archive using the program "curl" and verifies the output file's signature. Tested on 13.1.1. How to use this snippet: Edit the code to input the hostname of your F5 UI, admin credentials, source UCS file name (defaults to config.ucs), and the output file name. Code : #!/bin/bash # # Download a UCS archive (across a stable network) with curl. # #------------------------------------------------------------------------- F5_HOST='myhost.example.com' CREDENTIALS='admin:admin' FINAL_FILE='/tmp/config.ucs' ARCHIVE_NAME_ON_SERVER='config.ucs' DEBUG='' #------------------------------------------------------------------------- # # Get the md5 checksum for the archive. # #------------------------------------------------------------------------- ARCHIVE_CHECKSUM=$(curl -sku $CREDENTIALS -X POST -H "Content-type: application/json" \ -d "{\"command\":\"run\", \"utilCmdArgs\": \"-c '/usr/bin/md5sum /var/local/ucs/$ARCHIVE_NAME_ON_SERVER'\"}" \ https://$F5_HOST/mgmt/tm/util/bash | awk -F':' '{print $NF}' | awk -F'"' '{ print $2 }' | awk '{print $1}') [ -z "$ARCHIVE_CHECKSUM" ] && echo "Failed to get archive signature. Aborting." && exit 1 [ ! -z "$DEBUG" ] && echo "Archive checksum: $ARCHIVE_CHECKSUM" #------------------------------------------------------------------------- # # Find out the size of the archive and the size of the data packet. # #------------------------------------------------------------------------- Content_Range=$(curl -I -kv -u $CREDENTIALS -H 'Content-Type: application/json' -X GET "https://$F5_HOST/mgmt/shared/file-transfer/ucs-downloads/$ARCHIVE_NAME_ON_SERVER" 2>/dev/null | grep "Content-Range: " | cut -d ' ' -f 2) FIRST_CONTENT_RANGE=$(echo -n $Content_Range | cut -d '/' -f 1 | tr -d '\r') [ ! -z "$DEBUG" ] && echo -n "FIRST_CONTENT_RANGE: " [ ! -z "$DEBUG" ] && echo $FIRST_CONTENT_RANGE NUMBER_OF_LAST_BYTE=$(echo -n $FIRST_CONTENT_RANGE | cut -d '-' -f 2) [ ! -z "$DEBUG" ] && echo -n "NUMBER_OF_LAST_BYTE: " [ ! -z "$DEBUG" ] && echo $NUMBER_OF_LAST_BYTE INITIAL_CONTENT_LENGTH=$NUMBER_OF_LAST_BYTE CONTENT_LENGTH=$(($NUMBER_OF_LAST_BYTE+1)) [ ! -z "$DEBUG" ] && echo -n "CONTENT_LENGTH: " [ ! -z "$DEBUG" ] && echo $CONTENT_LENGTH DFILE_SIZE=$(echo -n $Content_Range | cut -d '/' -f 2 | tr -d '\r' ) [ ! -z "$DEBUG" ] && echo -n "DFILE_SIZE: " [ ! -z "$DEBUG" ] && echo $DFILE_SIZE LAST_END_BYTE=$((DFILE_SIZE-1)) CUMULATIVE_NO=0 [ ! -z "$DEBUG" ] && echo "CUMULATIVE_NO: $CUMULATIVE_NO" SEQ=0 LAST=0 #------------------------------------------------------------------------- # # Clean up: Remove the previous output file. # #------------------------------------------------------------------------- /bin/rm $FINAL_FILE 2>/dev/null #------------------------------------------------------------------------- # # Get the archive file. # #------------------------------------------------------------------------- while true do if [ $LAST -gt 0 ]; then [ ! -z "$DEBUG" ] && echo 'End of run reached.' break fi if [ $SEQ -eq 0 ]; then NEXT_RANGE=$FIRST_CONTENT_RANGE CUMULATIVE_NO=$NUMBER_OF_LAST_BYTE CONTENT_LENGTH=$INITIAL_CONTENT_LENGTH else START_BYTE=$(($CUMULATIVE_NO+1)) END_BYTE=$(($START_BYTE + $CONTENT_LENGTH)) if [ $END_BYTE -gt $LAST_END_BYTE ]; then [ ! -z "$DEBUG" ] && echo "END_BYTE greater than LAST_END_BYTE: $END_BYTE:$LAST_END_BYTE" LAST=1 let END_BYTE=$LAST_END_BYTE [ ! -z "$DEBUG" ] && echo "Getting the last data packet." fi NEXT_RANGE="${START_BYTE}-${END_BYTE}" CUMULATIVE_NO=$END_BYTE fi [ ! -z "$DEBUG" ] && echo "NEXT_RANGE: $NEXT_RANGE" let SEQ+=1 [ ! -z "$DEBUG" ] && echo "SEQ: $SEQ" OUTPUT_FILE_NAME="/tmp/$$_downloaded_ucs_archive_file_part_$SEQ"; curl -H "Content-Range: ${NEXT_RANGE}/${DFILE_SIZE}" -s -k -u $CREDENTIALS -H 'Content-Type: application/json' -X GET "https://$F5_HOST/mgmt/shared/file-transfer/ucs-downloads/$ARCHIVE_NAME_ON_SERVER" -o $OUTPUT_FILE_NAME cat $OUTPUT_FILE_NAME >> $FINAL_FILE /bin/rm $OUTPUT_FILE_NAME [ ! -z "$DEBUG" ] && echo "End of loop $SEQ" done #------------------------------------------------------------------------- # # Verify downloaded file. # #------------------------------------------------------------------------- FINAL_FILE_CHECKSUM=$(/usr/bin/md5sum $FINAL_FILE | awk '{print $1}') if [ "$FINAL_FILE_CHECKSUM" == "$ARCHIVE_CHECKSUM" ]; then echo "Download completed and verified." else echo "Downloaded file has incorrect checksum." exit 1 fi # END -------------------------------------------------------------------- Tested this on version: 13.0
Python BigREST ucs save and download
Problem this snippet solves: This uses BIGREST SDK API to save/download the ucs file. Adding it here since I couldn't find a similar one, How to use this snippet: Based on Python3 and BigREST (https://bigrest.readthedocs.io) Code : #Import needed libraries from bigrest.bigip import BIGIP import getpass #Replace the host name as needed host="xx.xx.xx.xx" user=input('Username') pw= getpass.getpass(prompt='Password:') #Declare the ucs filename, if needed ucsfile="test.ucs" #Connect to device device = BIGIP(host, user, pw) data = {} data["command"] = "save" data["name"] = ucsfile #You may get timeout exception below even if the file has been created task = device.task_start("/mgmt/tm/sys/ucs", data) device.task_wait(task) if device.task_completed(task): device.task_result(task) print("Backup has been completed.") else: raise Exception() #The below will download the file to the same folder as the script device.download("/mgmt/shared/file-transfer/ucs-downloads/", ucsfile) Tested this on version: 13.1
Restoring a VE with a UCS
Hello All, I am fairly new (1 year or so) working with the Virtual edition and am now writing a disaster recovery document for our deployment. I am hoping someone out there has done it before I attempt it. I took the troubleshooting class so I know that if the vm crashes and you have to import a new ovf that when you recover from the UCS you have to issue a no license switch. What I would like to know before that would be how much of the networking do you need to rebuild before you use the UCS. We do not have a management network so I have to access the VEs from a self-ip that allows HTTPS and SSH. Could I create the external vlan, the self-ip that allows me to scp the ucs to the new re-licensed (I would have already released the license and using the offline license tool relicensed it) and restore from tmsh the rest of the configuration, or do I need to create all of the vlans, and the external, internal and HA sef-ips as well? Any input would be much appreciated!
Comparing two UCS backup files
Is there anyway to compare two ucs files? A customer did an upgrade without follow thesoftware compatibility matrix, apparently everything worked ok untiluntil I detected a high cpu consume. F5 support team recommended us to do a rollback and then upgrade again followingthesoftware compatibility matrix rebuilding the configuration changes by hand. The problem is my customer doesn´t know how many changes have been made since the bad upgrade so I want to compare the current UCS and the UCS that I will get after doing the correct update process to find the configuration changesSpace in /var on physical device
I have been trying to run a BIGIP upgrade and keep getting stuck at the final part of the install where it checks for space in /var. This seems to be when the config has been put into a ucs and it wants to extract it. We have symlinked /var/local/ucs to a different location but it still fails as it seems to check just /var. We have removed everything possible but cannot get it to a point where there is enough space to work. What solutions is there for this problem on a physical device? As I understand it you can only resize /var on a VE. Thanks in advance
