Forum Discussion
speachey
Cirrus
CirrusReplacing Two HA Platforms with HA VEs
I'm planning to replace two legacy 4200V platform HA peers with a pair of VEs and I'm asking for guidance to ensure success. I have done several RMAs of platforms but never replaced a platform with VE. IP space makes adding new VEs to a sync group with unique self-ips problematic. I plan to replace the platforms one by one in a maintenance window. I'm considering accomplishing this by either configuring the new VEs using UCS archive 'platform-migrate' option (K82540512) or matching network configs and updating the VEs via ConfigSync. Here are two options I'm considering. I'm open to suggestions and any advice is greatly appreciated!
Complete:
- installed same version software on both VE vmware guests
- configured mgmt with unique IPs
Option 1 - Load from UCS:
- set new VEs to FORCED OFFLINE state and disable interfaces other than mgmt (to ensure they do not advertise routes or participate in ConfigSync)
- create UCS files on platforms to be replaced
- load UCS files on VEs with 'platform-migrate' option (K82540512)
- verify configs and add missing trunks, vlans, ConfigSync, Mirroring
- disable Automatic Sync
- shut down standby platform
- change mgmt IP and Host Name of VE to match standby
- enable interfaces
- manually initiate ConfigSync from Active platform to Standby VE
- verify ConfigSync, iQuery, Device Trust, and Logging
- promote VE from Standby to Active
- replace other platform the same way
- restore Automatic Sync
Concerns:
- VE becoming Active before intended (creating Active/Active scenario)
- iQuery comms failing/ lost device trust
- TTL of platform ARP on network devices before the new VEs are cached
- UCS archive might carry over unnecessary legacy cruft (these platforms are very old)
Option 2 - No UCS:
- set new VEs to FORCED OFFLINE state and disable interfaces other than mgmt (to ensure they do not advertise routes or participate in ConfigSync)
- manually add all vlans and self-ips for platform they will replace
- verify ConfigSync and Mirroring configs
- disable Automatic Sync on Active platform
- shut down standby platform
- change mgmt IP and Host Name of VE to match standby
- import device certificate
- enable interfaces
- manually initiate ConfigSync from Active platform to Standby VE
- verify ConfigSync, iQuery, Device Trust, and Logging
- promote VE from Standby to Active
- replace other platform the same way
- restore Automatic Sync
Concerns:
- VE becoming Active before intended (creating Active/Active scenario)
- iQuery comms failing/ lost device trust
- TTL of platform ARP on network devices before the new VEs are cached
Thoughts, concerns, suggestions? Thanks in advance!
Hi speachey ,
I like such these changes.
> Both of options are good ideas for migration , and you covered them well.I want to add some points below :
- After Activating the new licenses on both of New "VEs"
- Take the Master Key from of old F5 4200 appliance , it is enough one appliance as this Key is shared between them as long they are in HA Cluster or single device group.
For retrieving or getting this key issue the below Command on bash shell :
#f5mku -K , Note "K" is a uppercase letter.
Copy it’s output and save it in safe place.
before migration you must re-keying the master key on F5 Virtual edition " VEs" by issuing the below command on bash as well :
#f5mku -r "Key_output_from_the_old_appliance" , Note "r" is a lowercase letter.
> Without Re-Keying the master Key on "VEs" , your migration fails from the first step. - import "UCS" file , or transfer it to " /var/local/ucs " directory.
- Make sure that , the new VEs license is identical with Old device license in Features , I mean for instance if you have on your old devices " ASM , IP intelligence , GTM " you need to request a new license with the new "VEs" contains " ASM , IP intelligence , GTM " which are the same modules or Features.
- it is very very good to power both of VEs on the same version like old devices.
- when loading the ucs on VEs , correct you will use platform-migrate option but you must use the below command on tmsh prompt :
(tmos)# load sys ucs "File_Name" no-license platform-migrate.
> Why " no-licnse option" because F5 licenses is related to Appliances’ serial numbers and other parameters , and of course the license which exists on UCS is related to old appliances and is tied with its serial " mean old devices serials " .
So that you must exclude old license during loading ucs , at the same time you already configured the new license in " step number 1 " - After That , Revise your network design and Flow , and select your new interfaces which you are going to use , and assign again the Vlans to your interfaces " From Network >> Vlan >> (assign your needed interface to your vlan) " this part is depending on your design.
- I assume you will load UCSs on both of VEs
- After finishing " Vlans , Trunks , routes " , you will need to check the HA between both of VEs.
> I recommend that :
- Break up the HA between both of VEs , and remove each other from their device group.
- Re-build HA between both of VEs from scratch.
- Generate a new Certificate for both of them , before building device trust , check the below snap shot :
- Do not forget this step , the trust idea is exchanging certificates between both of appliances and you should not use an old certificate " old certificate will be transferred from old appliance via ucs on the new VEs " so you need to Re-new or re-generate new one.
> Follow this KB to build your HA correctly :
https://support.f5.com/csp/article/K42161405 - After Finishing HA , you will be about step on migration and you can test your services offline before changing the path of traffic from old appliances to new VEs.
- After making sure that all is fine , Take your maintenance window and change traffic Path to VEs to serve your application.
- Do you know , I do not recommend working with VEs over real Appliances , I think you will buy a new 2 physical appliances in the future , as a hardware based is more stable , speedy and efficient than Virtual edition.
- I prefer to choose " load ucs options ".
- I do not think disabling interfaces will add anything.
- you can use forced offline option , it is a very good idea.
- Automatic or manual Sync is not a problem in migration , start with manual sync , after guaranteeing that everything is stable , you can switch to Automatic sync.
I know you have covered most of my points , so I hope the above procedures help you if there are any missing point with you in migration plan.
GoodLuck in your Migration , you will enjoy by taking this Action.
Regards
I've gone with option 2 in the past and it's been successful.
Some additional tips:
- Ensure connectivity on all VLANs (ping self ips between devices). Reason because L2 config might not be correct.
- I don't think it's in the list, but forcing the old passive node to offline and disabling interface is a good way to get a fast rollback.
- Disable the old nodes interfaces before turning the new VM on.
- Run the VM as active at least 1-2 days to ensure that it works as expected before migrating the other device.
I don't think you need to worry as long as you follow your game plan. You've seem to have thought this through properly. Thumbs up!
speacheyThanks Patrik! Good ideas to ping/verify L2 and run one VE as Active for a few days before replacing the other platform. I'm replacing two similar HA platform pairs in different DCs with VEs and I will start with the DC that is not as high profile and test thoroughly. I suggested shutting the platforms down instead of forcing them offline because I plan to use the same mgmt config as the platforms they are replacing. I'll probably actually leave the platforms up and shut all of their interfaces at the nexus switches (mgmt, HA, trunk) so I could bring platforms back up quickly, if needed.
Just curious...have you noticed any performance issues switching from platform to VE? I don't think our load requirements are that high so it looks like VE should work fine ( 1K virtual servers, max of about 50K active connections, 150M throughput[bits], 250 APM Access Session/sec)
"I'll probably actually leave the platforms up and shut all of their interfaces at the nexus switches (mgmt, HA, trunk) so I could bring platforms back up quickly, if needed." - exactly my point. 🙂
As for performance, you should be ok. I have had to back out of a VE migration once because of performance but this was a high intensive sports book betting application. Have you read the docs about VMWare specific settings (if you're using VMware)?
https://clouddocs.f5.com/cloud/public/v1/vmware/vmware_users.html
Migrating the master key is not a bad idea by Mohamed below (if you're not using the ucs path). IIRC it's used to encrypt sensitive stuff in the config such as TLS key passphrases.
Hi speachey ,
I like such these changes.
> Both of options are good ideas for migration , and you covered them well.I want to add some points below :
- After Activating the new licenses on both of New "VEs"
- Take the Master Key from of old F5 4200 appliance , it is enough one appliance as this Key is shared between them as long they are in HA Cluster or single device group.
For retrieving or getting this key issue the below Command on bash shell :
#f5mku -K , Note "K" is a uppercase letter.
Copy it’s output and save it in safe place.
before migration you must re-keying the master key on F5 Virtual edition " VEs" by issuing the below command on bash as well :
#f5mku -r "Key_output_from_the_old_appliance" , Note "r" is a lowercase letter.
> Without Re-Keying the master Key on "VEs" , your migration fails from the first step. - import "UCS" file , or transfer it to " /var/local/ucs " directory.
- Make sure that , the new VEs license is identical with Old device license in Features , I mean for instance if you have on your old devices " ASM , IP intelligence , GTM " you need to request a new license with the new "VEs" contains " ASM , IP intelligence , GTM " which are the same modules or Features.
- it is very very good to power both of VEs on the same version like old devices.
- when loading the ucs on VEs , correct you will use platform-migrate option but you must use the below command on tmsh prompt :
(tmos)# load sys ucs "File_Name" no-license platform-migrate.
> Why " no-licnse option" because F5 licenses is related to Appliances’ serial numbers and other parameters , and of course the license which exists on UCS is related to old appliances and is tied with its serial " mean old devices serials " .
So that you must exclude old license during loading ucs , at the same time you already configured the new license in " step number 1 " - After That , Revise your network design and Flow , and select your new interfaces which you are going to use , and assign again the Vlans to your interfaces " From Network >> Vlan >> (assign your needed interface to your vlan) " this part is depending on your design.
- I assume you will load UCSs on both of VEs
- After finishing " Vlans , Trunks , routes " , you will need to check the HA between both of VEs.
> I recommend that :
- Break up the HA between both of VEs , and remove each other from their device group.
- Re-build HA between both of VEs from scratch.
- Generate a new Certificate for both of them , before building device trust , check the below snap shot :
- Do not forget this step , the trust idea is exchanging certificates between both of appliances and you should not use an old certificate " old certificate will be transferred from old appliance via ucs on the new VEs " so you need to Re-new or re-generate new one.
> Follow this KB to build your HA correctly :
https://support.f5.com/csp/article/K42161405 - After Finishing HA , you will be about step on migration and you can test your services offline before changing the path of traffic from old appliances to new VEs.
- After making sure that all is fine , Take your maintenance window and change traffic Path to VEs to serve your application.
- Do you know , I do not recommend working with VEs over real Appliances , I think you will buy a new 2 physical appliances in the future , as a hardware based is more stable , speedy and efficient than Virtual edition.
- I prefer to choose " load ucs options ".
- I do not think disabling interfaces will add anything.
- you can use forced offline option , it is a very good idea.
- Automatic or manual Sync is not a problem in migration , start with manual sync , after guaranteeing that everything is stable , you can switch to Automatic sync.
I know you have covered most of my points , so I hope the above procedures help you if there are any missing point with you in migration plan.
GoodLuck in your Migration , you will enjoy by taking this Action.
Regards- speachey
Thanks for the response and details Mohamed! These load balancers are LTM + APM. I purchased and already received the LTM and APM VE licenses from F5 and applied them to the VEs and configured Resource Provisioning appropriately. Is there still a need to transfer Master keys? I never heard of Mater Keys so I need to understand/research that.
I mention exporting/importing device certificates because we use device certs signed by a CA and not self-signed. It was an organization requirement that complicates updating device trust between LTMs and separate BIG-IP DNS (formerly GTM) devices in other locations every year. I would not recommend anyone using signed certs for f5 devices if you can avoid it! Would you still advise creating new/unique device certs on the VEs? I could go through the device trust dance, if there is a compelling reason to do so.
You recommend hardware/platform over VE. I know the F5 HW chipsets in the past were always preferred over VE, but everything I have read shows the specs on current VM NICs and specs look sufficient for our needs. I have never put production LTM+APM (SAML/SSO) services on VE and I'm curious about the details of problems you have had. Did increasing VM memory/CPU address your issues? I don't think our load requirements are that high so it looks like VE should work fine ( 1K virtual servers, max of about 50K active connections, 150M throughput[bits], 250 APM Access Session/sec). Are your connections and throughput mush higher than ours?
Thanks again for taking the time to share your knowledge!
Hi speachey ,
> About master Key :- It is the most important thing before Migration , this Key is relevant to UCSs or old appliance Configuration , and it must be printed on the VEs before loading UCS or you definitly fail.
Please read well this KB :
https://techdocs.f5.com/en-us/bigip-13-1-0/big-ip-secure-vault-administration/Working-with-ucs-files/ucs-installation-new-bigip-system/preparing-ucs-install-no-master-key-pw-no-access-bigip-no-object-pw.html
and this as well :
https://www.empirion.co.uk/f5/f5-master-key-rma-migration/
and take a look here Also :
https://securityguy225.wordpress.com/2016/11/11/how-to-migrate-all-configuration-from-2-different-f5-appliance/
____________________________________________________________________
> For Certificates :
- okay very well , you Can proceed without auto generating self sign certficate , but if you got errors in " Device Trust " it would be due to Certificate.
But at for a Certificate signed by CA , I expect that After migration , you will find both of devices "insync" Without issues.
_____________________________________________________________________
> For Physical and Virtual Editions :
I mentioned that as a Concern of Physical appliance Capabilities , but it seems you have done a good sizing with your needs.
- In internet service proiders , it is mandatory to use a very powerful Phyical appliances to process this huge Quantity of traffic.
____________________________________________________________________
Regards.
and GoodLuck with your migration
