BIG-IP Upgrades Part 1 - Preparation
Note: Most information contained in this article series has also been published on AskF5.com : K84554955: Overview of BIG-IP system software upgrades - https://support.f5.com/csp/article/K84554955 The information is this series is meant to help a Big-IP administrator complete a software upgrade. I have assisted with hundreds of Big-IP upgrades and hope to help you complete the task as smoothly as possible. For more information related to a specific Big-IP release, there are additional details in the release notes for each version. For more information, refer to the Release Notes page. The information in this series applies to: F5 platforms which support Big-IP version 10.x - 13.x Big-IP Virtual Edition vCMP Big-IP Host platforms vCMP Big-IP Guest instances For information regarding upgrading from Big-IP 9.x to 10.x, refer to the following guide: Manual Chapter: Upgrading from Versions 93x and 94x. Procedures Note: If you are familiar with Big-IP upgrade behavior and only need to review the actual steps, use the Performing the Software Installation section. To review Big-IP upgrade behavior in more detail, please review the other Parts. BIG-IP Upgrades Part 1 - Preparation BIG-IP Upgrades Part 2 - Upgrade Behavior BIG-IP Upgrades Part 3 - Versions, Misconceptions and a Back-Out Plan BIG-IP Upgrades Part 4 - Performing the Software Installation Prerequisites License reactivation is required prior to installing a newer Big-IP version. For more information, refer to K7727: License activation may be required prior to a software upgrade for the BIG-IP or Enterprise Manager system. Generate a qkview diagnostics file and review iHealth for any triggered upgrade-related heuristics in the Diagnostics and Upgrade Advisor tabs. Each Big-IP product has its own set of requirements pertaining to software upgrades and iHealth is the best utility to find up to date heuristics governing successful Big-IP product migrations. For more information, refer to: K12878: Generating BIG-IP diagnostic data using the qkview utility BIG-IP iHealth User Guide iHealth Website Create a UCS archive of the Big-IP's configuration and save it remotely in case it is needed for recovery purposes. For more information, refer to K4423: Overview of UCS archives. Verify your Big-IP system is running version 10.x, 11.x, 12.x or 13.x and is using the volumes formatting scheme (the command lvscan should not be blank). The command output should appear similar to the following example (the sizes may differ): ACTIVE '/dev/vg-db-sda/dat.share.1' [30.00 GB] normal ACTIVE '/dev/vg-db-sda/dat.log.1' [7.00 GB] normal ACTIVE '/dev/vg-db-sda/set.1.root' [256.00 MB] normal ACTIVE '/dev/vg-db-sda/set.1._usr' [1.34 GB] normal Check running configuration integrity by running one of the following commands. No errors should be returned (warnings may not hinder a software upgrade but should be corrected if possible). In Big-IP version 10.x bigpipe verify load In Big-IP version 11.x and 12.x tmsh load sys config verify Navigate to https://downloads.f5.com and download the desired software: Big-IP Base image ISO file (I.E. BIGIP-11.5.4.0.0.256.iso) Latest Hotfix ISO file (if available) (I.E. BIGIP-11.5.4.4.0.313-HF4.iso) For more information, refer to K167: Downloading software and firmware from F5. Verify base and hotfix software image file integrity. For more information, refer to K8337: Verifying the MD5 checksum for the downloaded F5 software file. Verify the device certificates of your Big-IP is not expired. For more information, refer to K7754: Renewing self-signed device certificates. Verify all Big-IP systems in the device group are in sync. For more information, refer to K13920: Performing a ConfigSync using the Configuration utility. Recommendations Prior to Installation From the F5 BIG-IP TMOS: Operations Guide; before you upgrade the BIG-IP software, review the release notes on AskF5 (support.f5.com) in the Documentation section for your product and version. Pay close attention to the following items: Review the Known issues list. Review the Behavior change section(s). Review the Upgrading from earlier versions section. Review the Upgrading from earlier configurations section. Review the installation checklist. Verify a connection to the serial console port is working. This not required but will allow monitoring of installation progress and provides additional recovery options. For more information, refer to K7683: Connecting a serial terminal to a BIG-IP system. When you have a date for the upgrade, we recommend starting a proactive Service Request with your plan in case there are complications during your change window. For more information, refer to K16022: Opening a proactive service request with F5 Technical Support. Testing the upgrade procedure on a lab environment using Big-IP Virtual Edition can be helpful to find potential issues before they are encountered in a production environment. BIG-IP 10Mbps Virtual Edition Lab License (11.4.1 - 13.X) Features: - Inexpensive compared to higher throughput Big-IP VE licenses. - LTM, GTM, DNS, AFM, ASM, AAM, CGN, APM Lite (10 users). - In Big-IP v11.4.1 - 11.6.x, you can import Big-IP 10.x - 11.x UCS archives. In Big-IP v12.x, you can import Big-IP 11.x - 12.x UCS archives. Copy the UCS file to /var/tmp and use the following command: tmsh load sys ucs /var/tmp/ .ucs no-license no-platform-check Limitations: - May not be available in all countries - Total system throughput limited to 10Mbps K15643: BIG-IP VE license offerings F5 Product Trials Retain a UCS archive from every Big-IP in your network on a remote filestore to aid disaster recovery. Even if the archive has aged and does not contain all configuration objects, it will provide faster recovery time than completely reconfiguring the Big-IP. For more information, refer to: K2880: Configuring a replacement BIG-IP device after a Return Materials Authorization K13551: Configuring a replacement BIG-IP device after an RMA when no UCS archive is available For full planning and assistance during an upgrade, F5 Professional Services is a good resource. Support will answer specific questions regarding the upgrade but can't be used start to finish to perform upgrades. For more information, refer to: Professional Services Scope of SupportBIG-IP Upgrades Part 2 - Upgrade Behavior
BIG-IP Upgrades Part 1 - Preparation BIG-IP Upgrades Part 2 - Upgrade Behavior BIG-IP Upgrades Part 3 - Versions, Misconceptions and a Back-Out Plan BIG-IP Upgrades Part 4 - Performing the Software Installation In this section, I'll describe some of the basics behind Big-IP software upgrade operations. Software images can be installed to any software volume except the running volume. This behavior ensures the running software volume is available should a need arise to revert to the prior Big-IP version and configuration. It is recommended to install the Big-IP update to an empty software volume to avoid potential confusion should the configuration fail to push to the new software volume. By default, the current running configuration is pushed to the new software volume automatically. The 'Install Configuration' option in the Configuration Utility can be used to update the target software volume prior to booting/activating the volume if time has elapsed since the software was installed and some of the Big-IP configuration has changed in the interim. This is generally unnecessary if you install the software and immediately boot into it. For more information, refer to K14704: Installing a configuration when activating a boot location. For a Big-IP instance, multiple Big-IP software installations can exist on disk. Use the tmsh show sys software command to view all software volumes. You can install directly over an existing software volume and the target volume will be overwritten. In order to delete a software volume, you can use the Configuration Utility or tmsh. Software installation can be performed via the Configuration Utility or tmsh. For more information, refer to: K13123: Managing BIG-IP product hotfixes (11.x - 12.x) K34745165: Managing software images on the BIG-IP system using the tmsh utility When a Hotfix ISO file is installed, two installations will happen in the background; the first will install the Final (larger) ISO and the second will install the Hotfix ISO. In the GUI, you will see two progress bars progress from 0-100%. The first boot of the new Big-IP software volume will take extra time, compared to a reboot, in order to decompress packages and to import the running configuration for the first time. Installation progress can be monitored via the serial console port or via the vconsole command in the case of vCMP Guest upgrades. For more information, refer to K15372: Overview of the vconsole utility. For Big-IP 10.x to 11.x upgrades, the previous configuration will first appear in the /config/bigpipe/ directory in the new software volume. When the new software volume is first booted, the configuration in /config/bigpipe/ will be converted and copied to the /config/ directory. So, if there are no virtual servers in /config/bigip.conf after upgrading from v10.x, the conversion process from /config/bigpipe/ likely failed for some reason. High Availability (HA) communication via network failover will function between major software branches but is only supported for the duration of the upgrade process. For example, a pair of Big-IPs running 11.5.3 and 12.1.1 can negotiate Active/Standby status via network failover. For more information, refer to K8665: BIG-IP redundant configuration hardware and software parity requirements. Configsync will not operate between different major software branches. For example, you cannot synchronize configurations from a unit running 11.5.3 unit to a unit running 11.5.4; you must wait for both units to be upgraded until configsync will operate. For more information, refer to K13946: Troubleshooting ConfigSync and device service clustering issues (11.x - 12.x). If a blank blank configuration is desired after the upgrade is complete, refer to K13438: Controlling configuration import when performing software installations (11.x - 12.x). As an alternative method to install the desired software version, prepared USB media can be used to reinitialize the disk and Big-IP software to factory defaults. Once complete, a previously saved UCS archive can be loaded to restore configuration. This method will wipe all data on the system. For more information, refer to: K13132: Backing up and restoring BIG-IP configuration files (11.x - 12.x) K13117: Performing a clean installation of BIG-IP 11.x - 12.x or Enterprise Manager 3.x If the software installation completes and you are able to boot the new volume but the Big-IP configuration fails to be migrated, you may see an error in the Configuration Utility. The configuration has not yet loaded. If this message persists, it may indicate a configuration problem. To determine what may be causing the error, run the tmsh load sys config verify command. vCMP considerations While vCMP guests are their own operating system, they inherit their license from the vCMP host or hypervisor. Before any vCMP guest can be upgraded, the vCMP host license must be reactivated. When a vCMP Host is upgraded, the Guests will go offline. A Host upgrade will not upgrade individual Guests. Only upgrade one vCMP Guest at a time to avoid excessive disk and CPU usage. For a vCMP Host, it is recommended but not required to run the same Big-IP version as the highest version Guest being hosted. For F5 platforms with multiple blades, software image ISO files are shared between blades. In 11.6.0+, ISO files are also shared from Hosts to Guests. Hosts and Guests utilize unique UCS configuration files. For instance, a Host UCS file does not contain Big-IP objects (for example, self IP addresses or virtual servers) for the Guests which are installed on the Host. For more information, refer to K15930: Overview of vCMP configuration considerations. Virtual Edition Considerations Just like Big-IP installed on a hardware platform, ISO files are used to upgrade. OVA files are only used to instantiate a Big-IP Virtual Edition.BIG-IP Upgrades Part 3 - Versions, Misconceptions and a Back-Out Plan
BIG-IP Upgrades Part 1 - Preparation BIG-IP Upgrades Part 2 - Upgrade Behavior BIG-IP Upgrades Part 3 - Versions, Misconceptions and a Back-Out Plan BIG-IP Upgrades Part 4 - Performing the Software Installation To What Big-IP Versions Can I Upgrade? Big-IP 10.x can be upgraded to any version of 11.x given your hardware supports the new version. Big-IP 11.x can be upgraded to any version of 12.x given your hardware supports the new version. You cannot upgrade directly from 10.x to 12.x. For more information, refer to: K13845: Overview of supported BIG-IP upgrade paths and an upgrade planning reference K9476: The F5 hardware/software compatibility matrix To What Big-IP Version Should I Upgrade? The Latest Maintenance Release of each Long Term Stability Release are the best choices for security and sustainability. For more information, refer to K5903: BIG-IP software support policy. The lowest-numbered version in the Latest Maintenance Release column is generally considered the most stable while the highest number contains the newest features and security fixes. Misconceptions and Perils "It's a firmware upgrade" Big-IP runs on either a physical or virtual disk with CentOS Linux installed as the base operating system. While there are also various internal firmware upgrades which take place between software versions, a Big-IP upgrade is at heart an operating system and tmos (Traffic Management Operating System) upgrade. "It will take two minutes" The fastest Big-IP upgrades will be around 15 minutes and up to an hour or more depending on the speed of the disk, CPU and size of the Big-IP configuration being upgraded. Be patient. Forcing a reboot during the installation process may lengthen recovery time and take more effort. "I can install to the current, running software volume" Engineers don’t often say this out loud but I have inferred some thinking this. Big-IP forces you to install to a software volume which is not currently in-use so that you are able to boot the old volume should there be a problem with the new one. "No need to reactivate the license" This is the #1 mistake made in regards to Big-IP upgrades. Assume a license reactivation IS necessary until you can explain exactly why it isn’t. "Didn't run tmsh load sys config verify before the upgrade" This is less common an issue but can be nasty if not caught early. Essentially, if the command does not succeed, not only should you expect upgrades problems but also config sync problems and possibly issues even reverting to the old software volume. Make sure this command completes without error before upgrading. "No need for UCS backups" What if you reboot your hardware and it never comes back up? Having a UCS archive will saves a lot of time should something go wrong and you need to restore a Big-IP quickly. "Let's upgrade and reboot both Big-IPs at the same time to save time" Again, you need to be sure one Big-IP is operational before risking losing both units in a high availability configuration. Just because a piece of electronics is running right now doesn’t mean it necessarily turn back on. "I have serial console port access but no need to check that's it's working prior to the upgrade" If you have easy access to a serial port connection to your Big-IPs, use it. A big advantage here is that you can still watch installation progress while the network ports are down. Back out Procedure If a Big-IP fails to upgrade and further troubleshooting cannot be performed due to time constraints, follow the below steps before reverting to the previous Big-IP version. Run the tmsh load /sys config command to determine what may be causing a configuration load error. Gather a qkview. Once the above data is gathered, you can boot the previous Big-IP version by using the Configuration Utility (System/Software Management/Boot Locations/ and click the Activate button which will reboot the system. You can also use K5658: Overview of the switchboot utility to change to another software volume via the command line. Note: The above back out procedure does not apply if you have reset the Big-IP to default settings. Alternatively, you can load your previously saved UCS archive. For more information, refer to K4423: Overview of UCS archives.BIG-IP Upgrades Part 4 - Performing the Software Installation
BIG-IP Upgrades Part 1 - Preparation BIG-IP Upgrades Part 2 - Upgrade Behavior BIG-IP Upgrades Part 3 - Versions, Misconceptions and a Back-Out Plan BIG-IP Upgrades Part 4 - Performing the Software Installation On a Big-IP in the Standby state (if there are more than one Big-IP in the device group), perform the following: Log in to the Configuration Utility as a user with administrator privileges. Upload the necessary ISO files (two images if a Hotfix is being applied): Via the Configuration Utility, navigate to System/Software Management and click the Import button. Alternatively. ISO files can be uploaded directly via Secure Copy (SCP) transfer directly to /shared/images/ and will automatically appear in the Configuration Utility when complete. For more information, refer to K175: Transferring files to or from an F5 system. If you are not already running the configuration to be upgraded, boot into a software volume containing the configuration to be copied to another volume. In the Configuration utility, click System. Click Software Management. Click the Hotfix List tab. If no Hotfix is being applied, click the Images tab. (Note that even if a Hotfix ISO file is installed, two installation will happen in the background; the first will install the Final [larger] ISO and the second will install the Hotfix ISO. In the GUI, you will see two progress bars move from 0-100%). Check the image name you want to install. Click the Install button. The Install window opens. Select an available disk from the Select Disk drop-down menu. From the Volume Set Name menu, select an empty volume set from the Volume Set Name drop-down menu or type a new volume set name. Note: You can use any combination of lowercase alphanumeric characters (a-z, 0-9) and the hyphen (-) character for the volume set name. The volume set name can be 1 to 32 characters in length, but cannot be only one 0 (zero) character (for example HD1.0 or MD1.0). For example, if the HD1 disk is active and you type siterequest into the volume set name, the system will create a volume set named HD1.siterequest and install the specified software to the new volume set. HD1.3 is typical (which conforms to the old v9 limitation). Click Install. If the string you type does not match an existing volume set, the system creates the volume set and installs the software. In the Installed Images section, the Install Status column first shows the base image installation progress and then the Hotfix installation progress. Once the installation process is complete: Navigate to System > Software Management > Boot Locations and click the boot location containing the desired software version. Click the Activate button to automatically reboot the system to the specified boot location. The upgrade can take up to 30 or more minutes depending on the size of the configuration. If the installation fails, running the command tmsh load sys config may show an error which will aid troubleshooting. Once the standby unit is upgraded: Verify expected objects appear in the shared and non-shared portions of the configuration. For example, check that Local Traffic > Pools and Network > Vlans has the expected objects present. Verify pool health status matches an Active unit's pool status. Check the most recent logs (/var/log/ltm for example) for obvious signs of issues like repeating messages. Comparing logs to the active unit or to the logs prior to the upgrade can be helpful. In addition or alternatively, you can generate a qkview and review it in the iHealth Diagnostics section for currently known issues. Force the Active unit to Standby (the upgraded system should take the Active role). Test client traffic to the upgraded system's traffic objects (for example, virtual servers) to confirm the upgraded unit is processing traffic correctly. Once the health of the upgraded unit is confirmed, repeat all the above upgrade steps on the peer Big-IP which is now in the Standby state and running the lower Big-IP version. Once all upgrades are complete, create a new set of UCS archives to retain backups of the new Big-IP version's configuration.
iHealth Upgrade Advisor: Making upgrades a little easier
Whether it is upgrading the firmware on a switch, the OS on a server, an important business application or the software on a BIG-IP, performing upgrades is something that makes almost all IT Admins and Network Engineers nervous. We’ve learned from (sometimes painful) experience that things don’t always go as planned. Good preparation greatly increases the likelihood that an upgrade will be successful, which is why F5 has created the iHealth Upgrade Advisor. Its goal is to provide an additional service from F5 that will complement your existing upgrade preparations, increasing the predictability of the upgrade while reducing your upgrade time. The iHealth Upgrade Advisor service provides a way for users to gain insight into potential issues with a BIG-IP upgrade before they attempt the upgrade. It provides guidance that is specific to a BIG-IP based on its configuration, the version of software it is currently running and the version you are planning to upgrade to. When an issue can be avoided by making a configuration change prior to upgrading, the Upgrade Advisor will tell you exactly what to change. For some issues, it will list the corrective actions to take after the upgrade. Demo Video This short video demonstrating the Upgrade Advisor shows you how to use it and some examples of the guidance it provides. Accessing the Upgrade Advisor The next time you are preparing to upgrade a BIG-IP, login to ihealth.f5.com, upload a .qkview file from that BIG-IP and then view the qkview after iHealth has analyzed it. The Upgrade Advisor can be accessed by clicking on its tab in the left-hand menu. Simply select the version of BIG-IP you are planning to upgrade to in the advisor and review the results. Here is a screenshot of the Upgrade Advisor: Give it a Try Try out the F5 upgrade Advisor today and let us know what you think using the feedback option (circled in red on the right side of the screenshot above).BIGdiff - A Little Help For Software Upgrades
Published on behalf of DevCentral MVP Leonardo Souza If you have been to F5 Agility in Boston and went to my presentation, you should have already an idea of what I will talk about in this article, but you will learn more things, so continue reading. If you haven’t heard of BIGdiff yet, have you been living in Mars? Don’t worry I will explain what that is and how it can help you with software upgrades, and whatever you find useful. It is not an AI that will do the upgrade for you but will help you with the upgrade. Challenges These are the challenges BIGdiff addresses: You are upgrading a F5 device with 1,000 virtual servers and 1,000 wide IPs. How do you know if you have the same number of virtual servers and wide IPs after the upgrade? How do you know if you have the same number of available virtual servers and wide IPs after the upgrade? If the number of available virtual servers or wide IPs changed after upgrade, how can you find what changed? Existing Solutions First Challenge: There are multiple solutions already for this challenge. Both for LTM and GTM, you can take a print screen of the statistics before the upgrade and compare after the upgrade. For LTM, Statistics > Module Statistics > Local Traffic For GTM, Statistics > Module Statistics > DNS > GSLB This in 13.1.0.1, but I think this exist since v9, and will be in similar place in all versions. Qkview and iHealth combination. iHealth will show you configuration totals but is mainly LTM and does not show you GTM objects. Network Map is another option. However, network map is only for LTM. Also, that is a map that start from a virtual server, so if you have a pool that is not linked to a virtual server that will not count in the totals. Second Challenge: The statistics also tell you the status of the objects, so that solution works for both challenges. Third Challenge: There is no automated way to get this. You could run multiple tmsh commands to get the status before the upgrade, or just generate a qkview that will run those commands for you. However, you will still need to compare the objects one by one. If the only slot you got for the software upgrade was 3am in a Sunday, I am sure you will miss some objects or fall asleep. Solution I hope you are thinking about the same, computers don’t need to sleep, and they are better/faster than humans to compare 2 strings or numbers (that is basically 0 or 1, so they are not that smart). So, the conclusion is simple, let the computer do the work comparing objects while you drink another coffee to keep you awake to complete the software upgrade. The idea is simple, get the list of objects, and their respective status, before and after the software upgrade, then compare them and report the result. In this context, object is any entity that has a status in a BIG-IP device that may be affected by the software upgrade. Looking BIG-IP modules, that translate to LTM and GTM objects, for example, virtual servers and wide IPs. That is where BIGdiff script comes to help you and automate that process. You run BIGdiff before the upgrade, upgrade the device, and run again after the upgrade. The script will then generate a HTML file with the results. Technical Bits BIGdiff is a bash script and uses dialog program to generate the graphical menus. Dialog is a common program for CLI menus and is what F5 uses for the config command for example. The script uses snmpwalk to query locally the device for the object status, because so far has been the faster option I tested. That basically generates the same text file before and after the upgrade. Those text files that are used after to compare the objects. The script will generate the results in a HTML format, with tables. If something already exists and do the job well, there is no reason why not to use, so the script uses the TableFilter JavaScript library, that provide filter functionalities for HTML tables. You just need to have the JavaScript library file in the same folder that you have the HTML file, and the magic will happen. If you don’t need the filter functionality, you don’t need to have the JavaScript library, and static tables will be presented. The script is optimized to use mainly bash functionality, to be as faster as possible. I tested the script to compare 13 thousand objects, and it complete the task in a couple minutes. 13K objects is a really big configuration, so even if the device you plan to run the script has a large configuration, that will be just a couple minutes in your change window to run the script. Support The script only supports BIG-IP software, no support for EM or BIG-IQ. The reason is simple, there is no use case for those software. Versions 11.x.x/12.x.x/13.x.x/14.x.x were tested and are supported. As new versions are released, I will be testing to see if any change is needed to support that version. LTM objects are supported and will be listed even if LTM is not provisioned, as majority of the other modules do use LTM internally. GTM objects and partitions are also supported. Using BIGdiff Go to the code share link: BIGdiff Download the tablefilter.js file, if you want to use the table filter functionality as described above. Download the bigdiff.sh that is the script file. In the F5 device, create a folder in /shared/tmp, as /shared is shared between all volumes. Upload the file bigdiff.sh to the F5 device. Change the file permission to run: chmod +x bigdiff.sh Run the script: ./bigdiff.sh Run the script before the upgrade. Upgrade the F5 device. Run the script after the upgrade. Download the file ending in .html from the F5 device. Open the HTML file with your favourite browser. Make sure you have the tablefilter.js in the same folder as the HTML file, if you want the filter functionalities. Other Use Cases The reason I wrote the script was to help with the software upgrades, but you are not limited to software upgrades. You can use the script to compare the objects after you have done something, that can be an upgrade or something else. You can use the software for consolidations, for example 2 devices that will be replaced by a single device. You run the script in the old devices, merge the txt files that are created with the list of objects. Import the configuration in the new device, upload the script and merged txt file you created. Run the script in the new device, and the script will report to you if the objects have the same status as in the old devices. Another use case is for major changes. You can run the script, do the changes, and run the script again. The script will then tell you if you broke something. Silent Mode Silent mode is mainly to be used to integrate with other tools. The image above explains how to use. Conclusion Read the information in the code share page about know issues. I hope you find the script useful.
