BIG-IP Upgrade Procedure Using CLI (vCMP Guest & Host)
Problem this snippet solves: Next article describes an upgrade procedure to perform only using CLI commands. The idea is not to replace an official procedure, but to give a different approach for those guys who love using CLI and they want to execute an upgrade only using commands (without GUI access). The procedure is separated in 4 sections: Data Collection & Planning - for executing some days before the upgrade. Pre-Upgrade Tasks - for executing just before the upgrade (applies to all devices in the cluster). Upgrade Tasks - Only applies for one device in the cluster for each time (normally standby device). Post-Upgrade Tasks - for executing just after the upgrade (applies to all devices in the cluster). This procedure is valid for most of the BIP-IP set-ups: Standalone & clusters vCMP Host & vCMP Guests GTM/DNS Synchronization Groups Everything that helps to fix mistakes is great, so your comments are welcome. OFFICIAL REFERENCES: Release Notes - https://support.f5.com/csp/knowledge-center/software/BIG-IP General Upgrade Procedure - https://support.f5.com/csp/article/K84554955 GTM/DNS Upgrades - https://support.f5.com/csp/article/K11661449 VCMP Host Upgrades - https://support.f5.com/csp/article/K15930#p17 HW Life-Cycle - https://support.f5.com/csp/article/K4309 SW Life-Cycle - https://support.f5.com/csp/article/K5903 HW-SW Compatibility - https://support.f5.com/csp/article/K9476 Upgrade Path - https://support.f5.com/csp/article/K13845 How to use this snippet: >> DATA COLLECTION & PLANNING (ALL CLUSTER DEVICES) >> PRE-UPGRADE TASKS (ALL CLUSTER DEVICES) >> UPGRADE TASKS (ONE DEVICE AT TIME) >> POST-UPGRADE TASKS (ALL CLUSTER DEVICES) Code : ###################################################### ## DATA COLLECTION & PLANNING (ALL CLUSTER DEVICES) ## ###################################################### ## Capture Product Code & Serial Number tmsh show sys hardware ## Capture Management IP & Blade State tmsh show sys cluster ## Capture Provision State tmsh list sys provision ##Capture Release and Volume Info tmsh show sys software ## Capture Master-key tmsh show sys crypto ## Check Relicensing Needed tmsh show sys license | grep -i 'service check date' REF - https://support.f5.com/csp/article/K7727 ##Check Certificate Expiration openssl x509 -noout -text -in /config/httpd/conf/ssl.crt/server.crt | grep Validity -A2 REF - https://support.f5.com/csp/article/K6353 ##Check RAID Integrity tmsh show sys raid tmsh run util platform_check cat /var/log/user.log cat /var/log/kern.log ##Check Mirroring Enabled tmsh show sys connection type mirror tmsh show sys ha-mirror ## Check Upgrade Disk Space (At least 20Gb) vgs ## Check ZebOS Module Running vtysh zebos/rdX/ZebOS.conf >> 'X' REPRESENTS ROUTE DOMAIN ID ## ONLY GTM/DNS - Check Devices Managed by GTM tmsh show gtm iquery all ## ONLY GTM/DNS - Check if DNSSEC keys in FIPS are Synchronized tmsh show sys crypto fips ## Capture QKView (Upload to iHealth) qkview REF - https://ihealth.f5.com/qkview-analyzer/ ##Check Release Notes For Specific Details REF - https://support.f5.com/csp/knowledge-center/software/BIG-IP ## Upload Release Image scp -p / @ :/shared/images/ ## Upload MD5 Hash Image scp -p / @ :/shared/images/ ## Upload Script to Check Pool Status scp -p /Check_Pool_Status.sh @ :/shared/tmp/ REF - https://github.com/DariuSGB/F5_Bash/blob/master/Check_Pool_Status.sh ############################################# ## PRE-UPGRADE TASKS (ALL CLUSTER DEVICES) ## ############################################# ##Disable Virtual Server Mirroring REF - https://support.f5.com/csp/article/K13478 ## Disable Config Auto-Sync (if enabled) tmsh modify cm device-group auto-sync disabled ## ONLY GTM/DNS - Disable GSLB/ZoneRunner Synchronization tmsh modify gtm global-settings general { synchronization no synchronize-zone-files no auto-discovery no } ## Save Running Config tmsh save sys config ##Check HA Cluster Synchronization tmsh show cm sync-status tmsh run cm config-sync to-group ## Check Release Image Integrity cd /shared/images/ md5sum -c ##Create Initial UCS (Backup) tmsh save sys ucs /shared/tmp/$(date '+%Y%m%d')_initial.ucs ## Capture Initial Config tmsh save sys config file /shared/tmp/$(date '+%Y%m%d')_initial.scf no-passphrase ## Capture Initial Pool Status /shared/tmp/Check_Pool_Status.sh > /shared/tmp/$(date '+%Y%m%d')_initial_pools_output.txt ## Check No Upgrade Process Running tmsh show sys software status ## OPTIONAL - Get More Free Disk Space (At least 20Gb) tmsh delete sys software volume vgs ######################################## ## UPGRADE TASKS (ONE DEVICE AT TIME) ## ######################################## ## Restart AOM to Prevent Licensing Problems (iSeries) ipmiutil reset -k REF - https://support.f5.com/csp/article/K00415052 ## ONLY VCMP HOST - Check That All Guests Are In Standby tmsh show vcmp guest >> ACCESS INDIVIDUALLY TO EACH GUEST tmsh show cm sync-status ## ONLY VCMP HOST - Deprovision All Guests (Configured) tmsh show vcmp guest >> EXECUTE FOR EACH GUEST tmsh modify vcmp guest state configured tmsh save sys config ## Re-licensing Device >> BIG-IP WITH INTERNET ACCESS tmsh install sys license registration-key add-on-keys { } REF - https://support.f5.com/csp/article/K15055 >> BIG-IP WITHOUT INTERNET ACCESS cp /config/bigip.license /config/bigip.license.backup get_dossier -b -a ** ACCESS LICENSE ACTIVATION https://activate.f5.com/license/dossier.jsp ** PASTE LICENSE FILE (ENTER 'CTRL+D' AFTER PASTING) cat > /config/bigip.license reloadlic REF - https://support.f5.com/csp/article/K2595 ## Force Offline Mode tmsh run sys failover offline ## Verify Configuration Integrity tmsh load sys config verify ## Install Image tmsh install sys software image create-volume volume ## Check Installation State tmsh show sys software status cat /var/log/liveinstall.log ## OPTIONAL - Copy Configuration To New Volume ## (Only if you have made changes since installation) clsh --slot=X,Y cpcfg >> FROM VIPRION cpcfg >> FROM NOT VIPRION ## Boot On New Volume tmsh reboot volume ## ONLY VCMP GUEST - Check Boot Up Status >> FROM VCMP HOST vconsole ## Check Logs (LTM, APM, ASM,...) REF - https://support.f5.com/csp/article/K16197 ## Capture Final Config tmsh save sys config file /shared/tmp/$(date '+%Y%m%d')_final.scf no-passphrase ## Compare Initial-Final Config tmsh show sys config-diff /shared/tmp/$(date '+%Y%m%d')_initial.scf /shared/tmp/$(date '+%Y%m%d')_final.scf | egrep -e "\s{3}\|\s{3}" -e "[<]$" -e "^\s*[>]" ## Disable Force Offline tmsh run sys failover online ## ONLY GTM/DNS - Enable Metrics Collection tmsh start sys service big3d ## Capture Final Pool Status /shared/tmp/Check_Pool_Status.sh > /shared/tmp/$(date '+%Y%m%d')_final_pools_output.txt ## Compare Initial-Final Pool Status diff /shared/tmp/$(date '+%Y%m%d')_initial_pools_output.txt /shared/tmp/$(date '+%Y%m%d')_final_pools_output.txt ## ONLY VCMP HOST - Deploy All Guests (Deployed) tmsh show vcmp guest tmsh modify vcmp guest state deployed ## FROM ACTIVE NODE - Check Current Connections tmsh show sys traffic raw ## FROM ACTIVE NODE - Force Failover Event tmsh run sys failover standby ## Check CPU/Memory status tmsh show sys cpu tmsh show sys memory ## Check Current Connections tmsh show sys traffic raw ##Perfom Other Custom Tests Here ... ############################################## ## POST-UPGRADE TASKS (ALL CLUSTER DEVICES) ## ############################################## ## OPTIONAL - Install Big3d daemon in all managed members ## (Only necessary if you upgrade GTM/DNS before its members) big3d_install REF - https://support.f5.com/csp/article/K11661449#update-big3d ## ONLY GTM/DNS - Enable GSLB/ZoneRunner Synchronization tmsh modify gtm global-settings general { synchronization yes synchronize-zone-files yes auto-discovery yes } ## Re-enable Virtual Server Mirroring REF - https://support.f5.com/csp/article/K13478 ## Synchronize HA Cluster tmsh show cm sync-status tmsh run cm config-sync force-full-load-push to-group ## Re-enable Config Auto-Sync (if enabled) tmsh modify cm device-group auto-sync enabled ## Save running config tmsh save sys config ## Create Final UCS (Backup) tmsh save sys ucs /shared/tmp/$(date '+%Y%m%d')_final.ucs ##Delete Unused Images delete sys software image ## Delete Unused Volumes (Mandatory reboot) delete sys software volume Tested this on version: 12.1Knowledge sharing: F5 Software Upgrade/RMA process
Here is quick summary about things should be checked before an F5 upgrade. This is the general F5 support article with clips and there is nice info for VIPRION and VCMP systems: https://support.f5.com/csp/article/K41125752 https://support.f5.com/csp/article/K84554955 https://support.f5.com/csp/article/K84205182 This a great community article 7 Steps Checklist before upgrading your F5 BIG-IP https://support.f5.com/csp/article/K11661449 https://support.f5.com/csp/article/K13081744 Extra addition to the DNS upgrade is that it is better upgrade first the LTM devices that the DNS devices monitor and after the upgrade of 1 or 2 DNS systems till the other DNS systems are also upgraded better upgrade the big3d process on the older DNS systems in the DNS sunc group: https://support.f5.com/csp/article/K15844889 https://support.f5.com/csp/article/K45907236 https://support.f5.com/csp/article/K13734 https://support.f5.com/csp/article/K13312 For BIG-IQ upgrade or for BIG-IQ to upgrade f5 devices: https://support.f5.com/csp/article/K51342220 https://techdocs.f5.com/en-us/bigiq-8-0-0/managing-big-ip-devices-from-big-iq/big-ip-software-upgrades.html For F5 devices with the F5 APM module after upgrade check if the installed F5 Edge Client software needs to be upgraded as it may not work with the new F5 APM TMOS version. https://support.f5.com/csp/article/K13757 An issue I have seen is to install the new version in a volume and transferring the configuration from the old volume to the new but without activating it and then to activate it after a week and there would an old configuration during that week many changes were done on the old volume config, so better before an upgrade so save UCS just in case from the old volume/partition: Some workarounds: https://support.f5.com/csp/article/K82463047 https://support.f5.com/csp/article/K14724 F5 RMA process general articles: F5 general articles for RMA with or withour UCS as without UCS the system and network settings may need to be configured manually and the configuration to be synchronized from the active device to the rma device. https://support.f5.com/csp/article/K12880 For F5 DNS/GTM there are special steps: https://support.f5.com/csp/article/K14083 F5 RMA of VIPRION chassis or a blade as for example when the new blade is installed but the active software version on other blades and vcmp quests is missing then the blade will get stuck in quorum for the chassis or vcmp quest as the primary blade will not be able to update it. If there is single blade in the chassis better hope that there is saved UCS expecially if there are vCMP quests as then for every vcmp quest the system and network need to be manually configured and the other config can be synchronized from the other chassis and vcmp quests that are in HA cluster. https://support.f5.com/csp/article/K14302 https://support.f5.com/csp/article/K16992 https://support.f5.com/csp/article/K23795307?utm_source=f5support&utm_medium=RSS https://support.f5.com/csp/article/K40222952 As the F5 VIPRION chassis is most complex (see K14302) if there is no saved master key as the vCMP quests use keys that are signed by the vCMP host master key and if it is lost then it is really complex, this is a nice F5 devcentral procedure how to generate your own master key that can be the same for the different F5 VIPRION Devices: https://community.f5.com/t5/technical-articles/working-with-masterkeys/ta-p/290454 When loading UCS on the RMA device that has containing encrypted passwords or passphrases, you can check(I have never used the second article but it is nice to have if issues are seen on a vCMP system when a chassis is replaced): https://support.f5.com/csp/article/K9420 Working with MasterKeys https://support.f5.com/csp/article/K13408 The new F5 Joutneys tool can be used for migrating to configuration to the new F5 VELOS and rSeries platforms and maybe in the future the F5 NEXT Operational System. https://community.f5.com/t5/technical-articles/welcome-to-the-f5-big-ip-migration-assistant-now-the-f5-journeys/ta-p/279673 https://www.youtube.com/watch?v=lLm5OkJRicw For the F5 imish/zebos routing module it is good to renember that that the config is not synchronized in a HA pair and before an RMA/upgrade to run the "write" command in the module as this is like the F5 command "save sys config" for CLI made changes as because of the reboot of the devices this changes can be lost. Before the license reactivation I suggest using the tool https://secure.f5.com/validate/validate.jsp to check that you have legitimate license and support contract.
How to cancel a installing process
Hi everyone. Are there any way to cancel installing process after typing "tmsh install sys software....." ? One day, during a test I realized that I tried to install unexpected s/w version. But I couldn't find any method. It's waste of time!!. Does anyone have any idea? ----------------------------------------------------------- Sys::Software Status Volume Product Version Build Active Status ----------------------------------------------------------- HD1.1 BIG-IP 12.1.3 8120.0 no installing hotfix <<== Want to cancel the installing process HD1.2 BIG-IP 12.1.0 7135.0 yes complete HD1.3 BIG-IP 11.6.2 1245.0 no complete
SFP+ port don't work after upgrade from 13.1.3 to 14.1.4.6
I have upgraded my BIG-IP 5000s from 13.1.3 to 14.1.4.6. After reboot, the panel display version 14.1.4.6, but all light for SFP+ port is down, and could ping the IP address configured on the port. The IP address of the Management port could be pinged, but could not access the device with ssh or httpd (403 error). Any advice is appreciated ! Best Regards, perlang
BigIP TMOS upgrades through BigIQ
I had a few questions about BigIP TMOS upgrades through BigIQ i.e. upgrading the TMOS ver of the managed devices from BigIQ CM How does BigIQ handle OS upgrade for Active-Standby device pair. We see the below screen while we select the devices , either we can select the Devices or we can select Group/Cluster wherein there is no option to select DSC cluster (which as we understand are Sync pair), we get option to select only DNS sync pair We ended up using the Device option, wherein we add the individual devices. All our deployment are Active- Standby, is BigIQ able to identify a failover pair if I add them as individual devices as I do not have an option to add them as DSC cluster. Presently our OS upgrades are done manually and with zero downtime, we would want to achieve the same with BigIQ with zero touch. Need your feedback Will BigIQ undertake the OS upgrade for the Standby device , failover then upgrade the other device. This is not documented properly we are referring to this article https://techdocs.f5.com/en-us/bigiq-8-0-0/managing-big-ip-devices-from-big-iq/big-ip-software-upgrades.html#GUID-D2C1FCE8-BB1A-4B23-A981-1F03BE47F291
