F5 Automated Backups - The Right Way
Hi all, Often I've been scouring the devcentral fora and codeshares to find that one piece of handywork that will drastically simplify my automated backup needs on F5 devices. Based on the works of Jason Rahm in his post "Third Time's the Charm: BIG-IP Backups Simplified with iCall" on the 26th of June 2013, I went ahead and created my own iApp that pretty much provides the answers for all my backup-needs. Here's a feature list of this iApp: It allows you to choose between both UCS or SCF as backup-types. (whilst providing ample warnings about SCF not being a very good restore-option due to the incompleteness in some cases) It allows you to provide a passphrase for the UCS archives (the standard GUI also does this, so the iApp should too) It allows you to not include the private keys (same thing: standard GUI does it, so the iApp does it too) It allows you to set a Backup Schedule for every X minutes/hours/days/weeks/months or a custom selection of days in the week It allows you to set the exact time, minute of the hour, day of the week or day of the month when the backup should be performed (depending on the usefulness with regards to the schedule type) It allows you to transfer the backup files to external devices using 4 different protocols, next to providing local storage on the device itself SCP (username/private key without password) SFTP (username/private key without password) FTP (username/password) SMB (using smbclient, with username/password) Local Storage (/var/local/ucs or /var/local/scf) It stores all passwords and private keys in a secure fashion: encrypted by the master key of the unit (f5mku), rendering it safe to store the backups, including the credentials off-box It has a configurable automatic pruning function for the Local Storage option, so the disk doesn't fill up (i.e. keep last X backup files) It allows you to configure the filename using the date/time wildcards from the tcl [clock] command, as well as providing a variable to include the hostname It requires only the WebGUI to establish the configuration you desire It allows you to disable the processes for automated backup, without you having to remove the Application Service or losing any previously entered settings For the external shellscripts it automatically generates, the credentials are stored in encrypted form (using the master key) It allows you to no longer be required to make modifications on the linux command line to get your automated backups running after an RMA or restore operation It cleans up after itself, which means there are no extraneous shellscripts or status files lingering around after the scripts execute I wasn't able to upload the iApp template to this article, so I threw it on pastebin: http://pastebin.com/YbDj3eMN Enjoy! Thomas Schockaert8.6KViews0likes79CommentsF5 Backup procedure over SCP using iCall
Problem this snippet solves: Purpose: You should consider using this procedure under the following condition. * You want to transfer BIG-IP Backup to the remote SCP Server on a specific frequency, without entering the password each time to transfer the file. Prerequisites: You must meet the following prerequisite to use this procedure. * You have administrator access to the BIG-IP Configuration utility and Command Line Access. * You have a user account on SCP Server with file transfer privileges. Description BIG-IP offers feature to transfer files over a remote SCP Server. Secure Copy (SCP) is the preferred means of transferring files to or from an F5 device. SCP securely transfers files between hosts using the Secure Shell (SSH) protocol for authentication and encryption. Unlike FTP, SCP provides an option to preserve the original date stamp on the file during file transfers. You can use SCP to transfer files between an F5 device and a remote host using either command line SCP or Windows-based SCP. We can automate the authentication process by export the public key from BIG-IP to SCP Server. By doing this, SCP Server will establish trust relationship with BIG-IP, and will not prompt for the password every-time we transfer files from BIG-IP to SCP Server. To achieve this, we need to have a user account on SCP Server, which has file transfer privileges. In this document, we will be using Linux based SCP Server. Once the transfer of files through the SCP works successfully. We can prepare the script done in iCall (as written following) and put it to run in the required interval. Login to SCP Server 1.Create a user account with permission to accept files from remote location, we will be using f5_user as user account, or you can use root user account (default user account on every Linux system). 2.It is good to have organized directory structure to receive F5 Backup. We will be creating 2 directories on our SCP Server as following. /F5Backup is the directory to receive F5 backup on a configured frequency (weekly, monthly, yearly etc). /authorized_key is temporary directory where will be sending the public key from F5 to SCP. 3.To create a directory in any Linux machine, you can use following command, mkdir /root/…path Eg. mkdir /home/f5_user/tmp/F5Backup/ 4.Create another directory to copy F5's public RSA key. mkdir /home/f5_user/tmp/authorized_keys Login to BIG-IP CLI Login to F5 CLI Generate RSA key by executing following. ssh-keygen -t rsa The RSA key will authenticate BIG-IP when it communicates with SCP Server. It would ask you to enter name and password, ignore the prompt by pressing ENTER. It will generate the Public / Private key under the directory /root/.ssh/id_rsa Verify the generated key by executing the following command. cat /root/.ssh/id_rsa.pub This should show you the public key, you have generated using above command. Send this public key to your Linux SCP Server. (We will be using SCP command to transfer the file from F5 to Linux Server). scp id_rsa.pub f5_user@10.1.20.222:/tmp/authorized_keys It will prompt for the password, enter the password for the f5_user user. In case if you are using other user account than f5_user, which has file transfer permission granted, you can replace f5_user with that user account, and keep the rest of the command as it is. Back to SCP Server Check if the Public key sent from F5 is received successfully under /tmp/authorized_keys or not. cat /f5_user/tmp/authorized_keys Note, in case if you have used other user account than "f5_user", replace the "f5_user" with the username you are using. Copy the key to right location, in order to authorize SCP connection from F5. cat /f5_user/tmp/mykey >> /f5_user/.ssh/authorized_keys If "authorized_keys" directory isn't exist, create a one by using mkdir command as mentioned earlier in the document. Verify the key is successfully placed under /f5_user/.ssh/authorized_keys or not. cat /f5_user/.ssh/authorized_keys Once the key successfully placed to the right location, it is time to test the connectivity from F5 to SCP Server. Switch back to F5 F5's CLI, scp filetest f5_user@10.1.60.240:/home/f5_user/tmp/ filetest 100% 5 0.0KB/s 00:00 If you have notices, this time it won’t ask for the password. In case if it still prompts, means you haven’t place the RSA key to the right place. The objective to place the RSA key under the user directory/.ssh/authorized_key. Once the connectivity is tested successfully, we can try to send F5’s UCS file over SCP to the remote server by the same method. Here, we can use the following script to generate the BIG-IP Backup and send it to the remote server. F5 CLI, type the following to type the script as following. Create a script with the command "tmsh create sys icall script <script name>" and then edit with vi and insert the content below.</p> </script> How to use this snippet: sys icall script auto_backup { app-service none definition { #Delete backup files exec rm -f /shared/tmp/*.ucs #Set Current Date/Time for Filename set cdate [clock format [clock seconds] -format "%Y%m%d"] #Set source repository set localpath "/var/local/ucs/" #Set destination repository set destinationpath "/home/teste/f5_backups" #Set remote host set host "10.1.20.222" #set remote user set user "f5_user" #Set device hostname set hostname [exec uname -n | cut -d "." -f1] #Set source repository cd $localpath #Delete files created more than 45 Days. catch { exec find "/var/local/ucs/" -type f -mtime +45 | grep -v .conf | xargs rm -f {} ; } #Delete UCS file if it exists catch { tmsh::delete sys ucs $hostname } #Export UCS tmsh::save sys ucs $hostname #Set temporary path set tmpdir "/shared/tmp/" append filename $hostname "_" $cdate #Copy UCS to temporary path exec cp $localpath$hostname.ucs $tmpdir$filename.ucs #Set Remote path append destination $user "@" $host ":" $destinationpath #Set source path append source $tmpdir $hostname "_" $cdate ".ucs" #Send the files via SCP. Prerequisite: The public key of BIG-IP must be registered in the file "authorized_keys" of the remote server if { [catch { exec scp -c aes128-ctr $source $destination > /dev/null 2> aux }] } { exec logger -p local0.info "Backup upload failed." } else { exec logger -p local0.info "The backup has been successfully sent to $destination." } exec rm -f aux } description none events none `</pre> } **Create the iCall Handler** I run the backup once a day; however, the periodicity can be adjusted according to the need of each one. In this example, I set the first-occurrence, the interval (once a day), and the script to call: <pre>`sys icall handler periodic auto_backup { first-occurrence 2019-03-27:05:01:00 interval 86400 script auto_backup } I know there are other scripts available in the community (much more sophisticated), the idea is just to share a simple and functional model. Remember that each one must adapt in the best way to meet your need. Code : 92596 Tested this on version: 12.14KViews3likes3CommentsF5 Archiver Ansible Playbook
Problem this snippet solves: Centralized scheduled archiving (backups) on F5 BIG-IP devices are a pain however, in the new world of Infrastructure as Code (IaC) and Super-NetOps tools like Ansible can provide the answer. I have a playbook I have been working on to allow me to backup off box quickly, UCS files are saves to a folder names tmp under the local project folder, this can be changed by editing the following line in the f5Archiver.yml file: dest: "tmp/{{ inventory_hostname }}-{{ date['stdout'] }}.ucs" The playbook can be run from a laptop on demand or via some scheduler (like cron ) or as part of a CI/CD pipelines. How to use this snippet: F5 Archiver Ansible Playbook Gitlab: StrataLabs: AnsibleF5Archiver Overview This Ansible playbook takes a list of F5 devices from a hosts file located within the inventory directory, creates a UCS archive and copies locally into the 'tmp' direcotry. Requirements This Ansible playbook requires the following: * ansible >= 2.5 * python module f5-sdk * F5 BIG-IP running TMOS >= 12 Usage Run using the ansible-playbook command using the inventory -i option to use the invertory directory instead of the default inventory host file. NOTE: F5 username and password are not set in the playbook and so need to be passed into the playbook as extra variables using the --extra-vars option, the variables are f5User for the username and f5Pwd for the password. The below examples use the default admin:admin . To check the playbook before using run the following commands ansible-playbook -i inventory --extra-vars "f5User=admin f5Pwd=admin" f5Archiver.yml --syntax-check ansible-playbook -i inventory --extra-vars "f5User=admin f5Pwd=admin" f5Archiver.yml --check Once happy run the following to execute the playbook ansible-playbook -i inventory --extra-vars "f5User=admin f5Pwd=admin" f5Archiver.yml Tested this on version: 12.11.7KViews2likes1CommentAPI Calls to F5 limited to 1024 KB download
Hi, I am interacting with the F5 API in order to download ASM policies for the purpose of automating the backups. The process works fine however policies larger than 1024 KB are cut off at this size of 1024 KB. Initially I suspected that there was a default limit on the curl request however I have not been able to find information on how to increase this with the curl request. Is this a limitation on the F5 API or the Curl request? wget is not an option as this is not natively supported on the F5 virtual appliance. My script lives on the appliance, downloads the relevant policies and then pushes them to a SMB share. The only issue is that the ASM policies that are larger than 1024 KB are being cut off at 1024 KB. The API calls are as per the documentation here: http://cdn.f5.com/websites/devcentral.f5.com/downloads/icontrol-rest-api-user-guide-13-0-0.pdf specifically: GET https://x.x.x.x/mgmt/tm/asm/policies POST https://x.x.x.x/mgmt/tm/asm/tasks/export-policy GET https://x.x.x.x/mgmt/tm/asm/file-transfer/downloads/$asmPolicy Excluding the processing in my script the API calls I make are shown below: I expect the issue resides in the download api call. Is there a switch I can add to increase this limit? curl -ku 'username:password' -X GET https://x.x.x.x/mgmt/tm/asm/policies | jq '.items[] | "pol_name:" + .name + ";api_id:" + .id' >> $wdir/asmDetails.txt curl -ku 'username:password' -X POST https://x.x.x.x/mgmt/tm/asm/tasks/export-policy -H 'Content-Type: application/json' -d '{"filename":"'$asmPolicy'","policyReference":{"link":"https://localhost/mgmt/tm/asm/policies/'$asmIDs'"}}' curl -ku 'username:password' -X GET https://x.x.x.x/mgmt/tm/asm/file-transfer/downloads/$asmPolicy > $wdir/asmBackup/"$folderName"/$number-$asmPolicy-$hostname-"$dateStamp".xml Thanks1.4KViews0likes16CommentsAutomated ASM Backup - working bash script now to automate or convert to iCall/tcl
Hi All, I have put together a BASH script that when run performs a backup of the ASM policies and copies them to a remote location. The script runs great and I have had it set as a Cron job in my lab setup to automate the backups. Unfortunately, the business does not want a script running as a Cron job on the F5. I have had it suggested to me to use iCall. I have seen only limited information regarding iCall that was written in a way that someone that has never seen iCall could understand. This got me far enough to understand that iCall runs tcl scripts, not bash scripts! The result being if I was to use iCall I would need to re-write the script completely. I am looking for 2 options here: A means to automate running a bash script on the F5. OR detailed information or getting started with iCall - Better yet, converting bash to tcl. To illustrate my issue, my bash script lives on the F5 and does the following: reads a counter value from a file curl command to the management interface and copies a list of ASM policy details to a txt file. greps the policy names from the original txt file to a new txt file. greps the policy IDs from the original txt file to a new txt file. sets a parameter with the current data and time as the value makes a localDirectory using the data and time parameter as the folder name (this ensures a known date of the backup - also ensures you can re-run and get a new folder on the same day if required) uses curl post and get commands to get the policies from the F5. curl upload-file command to copy files to remote smb location adjust the counter performs a cleanup of any files that were created locally. If I switch over to using iCall the above all needs to be done with tcl - I am not sure how much of that is supported. I have found that "echo" is replaced with "puts", is there a "curl", "cat", etc equivalent? Thanks in advanceSolved1.2KViews0likes6CommentsAdding Cron Jobs to the F5 - Is it OK? or should it be avoided?
Hi All, I have created a backup script that would reside on the F5 device, copy all ASM policies to XML and then push them to a remote fileshare. I have planned to have this script run via a cron job on the F5 once per month. When attempting to get approval from the business to implement this on the production devices, concern was raised around setting a cron job on the F5s. I personally did not feel that this would be an issue. Can anyone shed some light on this issue? Are others setting Cron jobs on the F5 or avoiding doing so for any reason in particular. If I want to schedule a script to run every month, is there a better alternative that I could use on the F5? Thank you.730Views0likes2CommentsF5 Auto backup script for SCP for V10 and 11.
Hello Guys, I have tried looking for the subjected topic within DevCentral, but unfortunately that didn't help. (I must be doing something wrong for sure). By the way I couldn't find any script for SCP which can be use for version 10. My requirement is to craft a script for SCP server which executes on daily basis. The customer is having 2 separate clusters, one is running on v10, another on 11. Is there any script which I can use for my requirement? Thank you,699Views0likes23Commentsplatform migration carry over Geolocation data file and ASM signature data file
I am working on platform migration from i5600 to i7600 by backing up UCS file and restoring it back to the i7600. I am wondering if geolocation data file, ASM signature, and bot signature will be updated as well. Recently I restore UCS file but see the Geolocation data file is 2020 which is last year and causing customer complain. When I did the geoip_lookup, it points to /usr/share/GeoIP/v2/F5GeoIP.dat which mean that there is no Geolocation data file under /shared/GeoIP/v2/F5GeoIP.dat and use the default location. What is the best way for me to compare the settings and configuration before and after the platform migration. I thought that UCS backup and restore should cover all the settings but I still missing Geolocation data file.671Views0likes0Commentsiapps f5.automated_backup problem
Hello, I'm using the iapps f5.automated_backup version 2.0.3 (I know it is an old one), the apps is working great, I use it on a test and a prod cluster. But since few days the app on the prod cluster stopped working, it doesn't run the backup on the configured schedule. When I try to check the configuration going on the iapps and clicking "reconfigure" the config never show up, I have the message "Loading... Receiving configuration data from your device." indefinitely. Anybody know what I can try to solve the problem ? Is there a service I can restart or something like that ? I suppose that a a full reboot could solve the problem but as it is a production device I'm trying to find another way to solve the problem. thanks for your help ! LucasSolved608Views0likes3CommentsAnsible F5 Backup
I've trawled the internet and Dev/Central to find a suitable Ansible playbook to do the following. Backup and F5 with the same filename so that I can push to our Gitlab for version control. The Ansible modules seem to either generate a random filename which isn't reusable in a playbook, if I specify source then the current UCS does not get overwritten, if I copy to the local filesystem with the same target name the module appends date and time to the file which will not give any consistency to GItlab. This is so far what I have come up with, the code is in its most basic form for testing only. - name: Clean the local backup directory path: "{{ item }}" state: absent with_fileglob: - "/ansible//dailybackups/*" connection: local - name: Clean the previous UCS file from F5 bigip_ucs: state: absent ucs: "{{ inventory_hostname }}.ucs" provider: server: 1.1.1.1 user: admin password: admin validate_certs: no delegate_to: localhost - name: Save the running configuration of the BIG-IP bigip_ucs_fetch: backup: yes src: "{{ inventory_hostname }}.ucs" dest: /ansible/dailybackups/{{ inventory_hostname }}.ucs provider: server: 1.1.1.1 user: admin password: admin validate_certs: no delegate_to: localhost So to perform a repeatable function I am forced to delete the file from the local file system to be copied to, erase the current UCS file on the F5 which is used as the backup, and then backup the F5 and pull the file to the local file system. Surely there is a slicker way of doing what can be done on a Cisco device in 4 lines. (NB) I have excluded the Git function, these 3 plays are merely to pull a consistent named UCS file and store to the local filesystem.599Views0likes0Comments