BIG-IP Upgrade Procedure Using CLI (vCMP Guest & Host)
Problem this snippet solves: Next article describes an upgrade procedure to perform only using CLI commands. The idea is not to replace an official procedure, but to give a different approach for those guys who love using CLI and they want to execute an upgrade only using commands (without GUI access). The procedure is separated in 4 sections: Data Collection & Planning - for executing some days before the upgrade. Pre-Upgrade Tasks - for executing just before the upgrade (applies to all devices in the cluster). Upgrade Tasks - Only applies for one device in the cluster for each time (normally standby device). Post-Upgrade Tasks - for executing just after the upgrade (applies to all devices in the cluster). This procedure is valid for most of the BIP-IP set-ups: Standalone & clusters vCMP Host & vCMP Guests GTM/DNS Synchronization Groups Everything that helps to fix mistakes is great, so your comments are welcome. OFFICIAL REFERENCES: Release Notes - https://support.f5.com/csp/knowledge-center/software/BIG-IP General Upgrade Procedure - https://support.f5.com/csp/article/K84554955 GTM/DNS Upgrades - https://support.f5.com/csp/article/K11661449 VCMP Host Upgrades - https://support.f5.com/csp/article/K15930#p17 HW Life-Cycle - https://support.f5.com/csp/article/K4309 SW Life-Cycle - https://support.f5.com/csp/article/K5903 HW-SW Compatibility - https://support.f5.com/csp/article/K9476 Upgrade Path - https://support.f5.com/csp/article/K13845 How to use this snippet: >> DATA COLLECTION & PLANNING (ALL CLUSTER DEVICES) >> PRE-UPGRADE TASKS (ALL CLUSTER DEVICES) >> UPGRADE TASKS (ONE DEVICE AT TIME) >> POST-UPGRADE TASKS (ALL CLUSTER DEVICES) Code : ###################################################### ## DATA COLLECTION & PLANNING (ALL CLUSTER DEVICES) ## ###################################################### ## Capture Product Code & Serial Number tmsh show sys hardware ## Capture Management IP & Blade State tmsh show sys cluster ## Capture Provision State tmsh list sys provision ##Capture Release and Volume Info tmsh show sys software ## Capture Master-key tmsh show sys crypto ## Check Relicensing Needed tmsh show sys license | grep -i 'service check date' REF - https://support.f5.com/csp/article/K7727 ##Check Certificate Expiration openssl x509 -noout -text -in /config/httpd/conf/ssl.crt/server.crt | grep Validity -A2 REF - https://support.f5.com/csp/article/K6353 ##Check RAID Integrity tmsh show sys raid tmsh run util platform_check cat /var/log/user.log cat /var/log/kern.log ##Check Mirroring Enabled tmsh show sys connection type mirror tmsh show sys ha-mirror ## Check Upgrade Disk Space (At least 20Gb) vgs ## Check ZebOS Module Running vtysh zebos/rdX/ZebOS.conf >> 'X' REPRESENTS ROUTE DOMAIN ID ## ONLY GTM/DNS - Check Devices Managed by GTM tmsh show gtm iquery all ## ONLY GTM/DNS - Check if DNSSEC keys in FIPS are Synchronized tmsh show sys crypto fips ## Capture QKView (Upload to iHealth) qkview REF - https://ihealth.f5.com/qkview-analyzer/ ##Check Release Notes For Specific Details REF - https://support.f5.com/csp/knowledge-center/software/BIG-IP ## Upload Release Image scp -p / @ :/shared/images/ ## Upload MD5 Hash Image scp -p / @ :/shared/images/ ## Upload Script to Check Pool Status scp -p /Check_Pool_Status.sh @ :/shared/tmp/ REF - https://github.com/DariuSGB/F5_Bash/blob/master/Check_Pool_Status.sh ############################################# ## PRE-UPGRADE TASKS (ALL CLUSTER DEVICES) ## ############################################# ##Disable Virtual Server Mirroring REF - https://support.f5.com/csp/article/K13478 ## Disable Config Auto-Sync (if enabled) tmsh modify cm device-group auto-sync disabled ## ONLY GTM/DNS - Disable GSLB/ZoneRunner Synchronization tmsh modify gtm global-settings general { synchronization no synchronize-zone-files no auto-discovery no } ## Save Running Config tmsh save sys config ##Check HA Cluster Synchronization tmsh show cm sync-status tmsh run cm config-sync to-group ## Check Release Image Integrity cd /shared/images/ md5sum -c ##Create Initial UCS (Backup) tmsh save sys ucs /shared/tmp/$(date '+%Y%m%d')_initial.ucs ## Capture Initial Config tmsh save sys config file /shared/tmp/$(date '+%Y%m%d')_initial.scf no-passphrase ## Capture Initial Pool Status /shared/tmp/Check_Pool_Status.sh > /shared/tmp/$(date '+%Y%m%d')_initial_pools_output.txt ## Check No Upgrade Process Running tmsh show sys software status ## OPTIONAL - Get More Free Disk Space (At least 20Gb) tmsh delete sys software volume vgs ######################################## ## UPGRADE TASKS (ONE DEVICE AT TIME) ## ######################################## ## Restart AOM to Prevent Licensing Problems (iSeries) ipmiutil reset -k REF - https://support.f5.com/csp/article/K00415052 ## ONLY VCMP HOST - Check That All Guests Are In Standby tmsh show vcmp guest >> ACCESS INDIVIDUALLY TO EACH GUEST tmsh show cm sync-status ## ONLY VCMP HOST - Deprovision All Guests (Configured) tmsh show vcmp guest >> EXECUTE FOR EACH GUEST tmsh modify vcmp guest state configured tmsh save sys config ## Re-licensing Device >> BIG-IP WITH INTERNET ACCESS tmsh install sys license registration-key add-on-keys { } REF - https://support.f5.com/csp/article/K15055 >> BIG-IP WITHOUT INTERNET ACCESS cp /config/bigip.license /config/bigip.license.backup get_dossier -b -a ** ACCESS LICENSE ACTIVATION https://activate.f5.com/license/dossier.jsp ** PASTE LICENSE FILE (ENTER 'CTRL+D' AFTER PASTING) cat > /config/bigip.license reloadlic REF - https://support.f5.com/csp/article/K2595 ## Force Offline Mode tmsh run sys failover offline ## Verify Configuration Integrity tmsh load sys config verify ## Install Image tmsh install sys software image create-volume volume ## Check Installation State tmsh show sys software status cat /var/log/liveinstall.log ## OPTIONAL - Copy Configuration To New Volume ## (Only if you have made changes since installation) clsh --slot=X,Y cpcfg >> FROM VIPRION cpcfg >> FROM NOT VIPRION ## Boot On New Volume tmsh reboot volume ## ONLY VCMP GUEST - Check Boot Up Status >> FROM VCMP HOST vconsole ## Check Logs (LTM, APM, ASM,...) REF - https://support.f5.com/csp/article/K16197 ## Capture Final Config tmsh save sys config file /shared/tmp/$(date '+%Y%m%d')_final.scf no-passphrase ## Compare Initial-Final Config tmsh show sys config-diff /shared/tmp/$(date '+%Y%m%d')_initial.scf /shared/tmp/$(date '+%Y%m%d')_final.scf | egrep -e "\s{3}\|\s{3}" -e "[<]$" -e "^\s*[>]" ## Disable Force Offline tmsh run sys failover online ## ONLY GTM/DNS - Enable Metrics Collection tmsh start sys service big3d ## Capture Final Pool Status /shared/tmp/Check_Pool_Status.sh > /shared/tmp/$(date '+%Y%m%d')_final_pools_output.txt ## Compare Initial-Final Pool Status diff /shared/tmp/$(date '+%Y%m%d')_initial_pools_output.txt /shared/tmp/$(date '+%Y%m%d')_final_pools_output.txt ## ONLY VCMP HOST - Deploy All Guests (Deployed) tmsh show vcmp guest tmsh modify vcmp guest state deployed ## FROM ACTIVE NODE - Check Current Connections tmsh show sys traffic raw ## FROM ACTIVE NODE - Force Failover Event tmsh run sys failover standby ## Check CPU/Memory status tmsh show sys cpu tmsh show sys memory ## Check Current Connections tmsh show sys traffic raw ##Perfom Other Custom Tests Here ... ############################################## ## POST-UPGRADE TASKS (ALL CLUSTER DEVICES) ## ############################################## ## OPTIONAL - Install Big3d daemon in all managed members ## (Only necessary if you upgrade GTM/DNS before its members) big3d_install REF - https://support.f5.com/csp/article/K11661449#update-big3d ## ONLY GTM/DNS - Enable GSLB/ZoneRunner Synchronization tmsh modify gtm global-settings general { synchronization yes synchronize-zone-files yes auto-discovery yes } ## Re-enable Virtual Server Mirroring REF - https://support.f5.com/csp/article/K13478 ## Synchronize HA Cluster tmsh show cm sync-status tmsh run cm config-sync force-full-load-push to-group ## Re-enable Config Auto-Sync (if enabled) tmsh modify cm device-group auto-sync enabled ## Save running config tmsh save sys config ## Create Final UCS (Backup) tmsh save sys ucs /shared/tmp/$(date '+%Y%m%d')_final.ucs ##Delete Unused Images delete sys software image ## Delete Unused Volumes (Mandatory reboot) delete sys software volume Tested this on version: 12.1Knowledge sharing: F5 Software Upgrade/RMA process
Here is quick summary about things should be checked before an F5 upgrade. This is the general F5 support article with clips and there is nice info for VIPRION and VCMP systems: https://support.f5.com/csp/article/K41125752 https://support.f5.com/csp/article/K84554955 https://support.f5.com/csp/article/K84205182 This a great community article 7 Steps Checklist before upgrading your F5 BIG-IP https://support.f5.com/csp/article/K11661449 https://support.f5.com/csp/article/K13081744 Extra addition to the DNS upgrade is that it is better upgrade first the LTM devices that the DNS devices monitor and after the upgrade of 1 or 2 DNS systems till the other DNS systems are also upgraded better upgrade the big3d process on the older DNS systems in the DNS sunc group: https://support.f5.com/csp/article/K15844889 https://support.f5.com/csp/article/K45907236 https://support.f5.com/csp/article/K13734 https://support.f5.com/csp/article/K13312 For BIG-IQ upgrade or for BIG-IQ to upgrade f5 devices: https://support.f5.com/csp/article/K51342220 https://techdocs.f5.com/en-us/bigiq-8-0-0/managing-big-ip-devices-from-big-iq/big-ip-software-upgrades.html For F5 devices with the F5 APM module after upgrade check if the installed F5 Edge Client software needs to be upgraded as it may not work with the new F5 APM TMOS version. https://support.f5.com/csp/article/K13757 An issue I have seen is to install the new version in a volume and transferring the configuration from the old volume to the new but without activating it and then to activate it after a week and there would an old configuration during that week many changes were done on the old volume config, so better before an upgrade so save UCS just in case from the old volume/partition: Some workarounds: https://support.f5.com/csp/article/K82463047 https://support.f5.com/csp/article/K14724 F5 RMA process general articles: F5 general articles for RMA with or withour UCS as without UCS the system and network settings may need to be configured manually and the configuration to be synchronized from the active device to the rma device. https://support.f5.com/csp/article/K12880 For F5 DNS/GTM there are special steps: https://support.f5.com/csp/article/K14083 F5 RMA of VIPRION chassis or a blade as for example when the new blade is installed but the active software version on other blades and vcmp quests is missing then the blade will get stuck in quorum for the chassis or vcmp quest as the primary blade will not be able to update it. If there is single blade in the chassis better hope that there is saved UCS expecially if there are vCMP quests as then for every vcmp quest the system and network need to be manually configured and the other config can be synchronized from the other chassis and vcmp quests that are in HA cluster. https://support.f5.com/csp/article/K14302 https://support.f5.com/csp/article/K16992 https://support.f5.com/csp/article/K23795307?utm_source=f5support&utm_medium=RSS https://support.f5.com/csp/article/K40222952 As the F5 VIPRION chassis is most complex (see K14302) if there is no saved master key as the vCMP quests use keys that are signed by the vCMP host master key and if it is lost then it is really complex, this is a nice F5 devcentral procedure how to generate your own master key that can be the same for the different F5 VIPRION Devices: https://community.f5.com/t5/technical-articles/working-with-masterkeys/ta-p/290454 When loading UCS on the RMA device that has containing encrypted passwords or passphrases, you can check(I have never used the second article but it is nice to have if issues are seen on a vCMP system when a chassis is replaced): https://support.f5.com/csp/article/K9420 Working with MasterKeys https://support.f5.com/csp/article/K13408 The new F5 Joutneys tool can be used for migrating to configuration to the new F5 VELOS and rSeries platforms and maybe in the future the F5 NEXT Operational System. https://community.f5.com/t5/technical-articles/welcome-to-the-f5-big-ip-migration-assistant-now-the-f5-journeys/ta-p/279673 https://www.youtube.com/watch?v=lLm5OkJRicw For the F5 imish/zebos routing module it is good to renember that that the config is not synchronized in a HA pair and before an RMA/upgrade to run the "write" command in the module as this is like the F5 command "save sys config" for CLI made changes as because of the reboot of the devices this changes can be lost. Before the license reactivation I suggest using the tool https://secure.f5.com/validate/validate.jsp to check that you have legitimate license and support contract.
Using BIG-IQ to Address the CVE-2020-5902 Vulnerability
As you’re probably already aware, a critical vulnerability was recently discovered within the BIG-IP Traffic Management User Interface (TMUI). In a nutshell, TMUI—sometimes known as the Configuration Utility—has a Remote Code Execution (RCE) vulnerability that can result in a complete system compromise through the ability to: Execute system commands Create or delete files Disable services Execute arbitrary Java code The most critical cases involve BIG-IP systems whose management port and/or self IPs are exposed to the open internet. In these cases, it’s best to assume a breach/compromise scenario and respond accordingly—refer to your organization’s incident response plan. However, even those BIG-IPs that aren’t internet-facing and running in Appliance mode, are still vulnerable.In short, this is an issue that all BIG-IP customers need to address immediately. Fixing the problem F5 has released several resources to help our customers who’ve been affected by this issue. We recommend getting started here on AskF5. In addition to the resources highlighted in the AskF5 article, our DevCentral team has facilitated some video resources and curated answers to the many questions we have received.The first link provides a matrix that outlines which versions of BIG-IP were affected by the vulnerability. To eliminate this issue completely, the recommended course of action is to update/install a new, fixed version of BIG-IP—these fixed versions are listed in the same matrix. If updates cannot be performed quickly, there are other mitigation techniques that can be employed that are listed on the AskF5 article. Leveraging BIG-IQ A couple of the strategies highlighted in the resources above center around BIG-IQ, F5’s powerful solution for unified visibility and management of BIG-IP. BIG-IQ can be especially useful in the context of addressing the CVE-2020-5902 vulnerability as it makes the management of many BIG-IPs much easier and programmatic. With BIG-IQ’s single UI, you can employ two effective CVE-2020-5902 mitigation strategies: Running a bash script on BIG-IQ managed devices Upgrading/updating managed devices to new BIG-IP software versions The bash approach For those that aren’t ready for a full upgrade of their affected BIG-IPs, leveraging a script that mitigates the vulnerability—at least until such time that a highly recommended upgrade can be performed—is a good strategy. BIG-IQ makes this process simple. You can find the script referenced in the video on github - https://github.com/usrlocalbins/Big-IQ-scripts. The upgrade/update approach As we mentioned before, the recommended method is to update your BIG-IP software to a “fixed” version. Beyond CVE-2020-5902, running updated/upgraded versions of software is good practice as it ensures you: Get the latest features and capabilities Are protected from identified threats, vulnerabilities, and bad actors Remain in compliance Are eligible for support and expert help Keep maintenance costs down Sidestep compatibility issues with legacy software The process for updating managed (many) BIG-IPs with BIG-IQ is very straightforward and greatly reduces the time, effort, and manual errors associated with a piecemeal BIG-IP upgrade approach—especially for those with large BIG-IP portfolios. Further Reading To learn more about upgrading your BIG-IPs via BIG-IQ visit the Knowledge Center on AskF5 . Want to learn more about BIG-IQ? You can find more resources—including a no-install demo—at f5.com/bigiq
Upgrade from 13.1.3.3 to 13.1.3.6
Hi! This weekend, we have encountered some issues with upgrading with 13.1.3.3 to 13.1.3.6 which forced us to cancel the upgrade. When we arrived on 13.1.3.6 the configuration was empty : no VS, no Real Server....empty. One of my guys presumed that the ciphers for SSL on the new version was different than the one of 13.1.3.3 which caused the box to cancel the import of the configuration. Any idea ? thanks!
