on 17-Aug-2011 13:13
Introduction
Ever wanted to “failover” a virtual server to an LTM that is not part of the same HA (high availability) pair? This is a common occurrence for any engineer that has ever had to upgrade their BIG-IP hardware platform or been part of a “consolidation” project. If the virtual is being addressed via DNS this can be as simple as creating the virtual server (using a different IP) and its dependencies on an adjacent unit, then cutting over by updating the A record. Most of us know that it is never that simple. The IP address itself is almost always hardcoded in a number of configurations and therefore we have to move the IP itself with minimal impact on our application. Enter virtual address failover via gratuitous ARP.
There are a number of ways to facilitate this action, but for the sake of expediency and repeatability we’ll be using a script to perform the failover. However, before we start failing things over, lets take a look at what we need to bring our virtual address up on our target unit. It is important at this point that you understand the difference between a virtual address and a virtual server. A virtual server consists of a virtual address and a port, therefore a virtual server can only have one virtual address. A virtual address on the other hand can have any number (up to the number of available ports and protocols) of virtual servers attached to it. The virtual address is often ignored as it is automatically created with the first virtual server referencing that address.
Recreating the Virtual Address And Server on the Target Unit
This method can be fairly labor intensive, but if you’re only moving a few virtual servers it is the most convenient as well as a great way to clean up your configurations during the migration. Before we go into the specific steps, here is what we’re going to be doing: creating a virtual address with ARP disabled, copying the virtual server config(s) and all the dependencies (pools, profiles, health monitors, etc.). Let’s walk through some of the steps involved in this migration.
test-ltm-01(Active)(tmos)# list ltm virtual test_http_virtual => ltm virtual test_http_virtual { destination 10.0.0.200:http ip-protocol tcp mask 255.255.255.255 mirror enabled pool test_http_pool profiles { http { } tcp-lan-optimized { } } snat automap }
test-ltm-01(Active)(tmos)# list ltm pool test_http_pool => ltm pool test_http_pool { members { 10.0.0.131:http { state up } 10.0.0.132:http { state up } 10.0.0.133:http { session disabled state down } } monitor http }
create ltm virtual-address 10.0.0.200 arp disabled*Make sure that you create the virtual address with ARP disabled prior to creating any virtual servers otherwise the virtual address will be created automatically as will be an IP conflict (/var/log/ltm will report and IP conflicts). This action must be done from the CLI as there is no GUI option to disable ARP on virtual address creation.
test-ltm-02(Active)(tmos)# edit ltm pool test_http_pool => create pool test_http_pool { members add { 10.0.0.131:http { state up } 10.0.0.132:http { state up } 10.0.0.133:http { session disabled state down } } monitor http }*Also node the 'add' statement after the members component. This is necessary when adding new pool members.
test-ltm-02(Active)(tmos)# edit ltm virtual test_http_virtual => create virtual test_http_virtual { destination 10.0.0.200:http ip-protocol tcp mask 255.255.255.255 mirror enabled pool test_http_pool profiles add { http { } tcp-lan-optimized { } } snat automap }*Note the 'add' statement after profiles.
Alternative Approach – Moving The Entire Configuration
As was mentioned at the beginning of this section, this approach may not be feasible if you are dealing with hundreds or thousands of virtual addresses and servers. In that case you’ll probably want to go about dumping the entire configuration, moving it to the new unit(s), changing the unit-specific values (self-IPs, hostname, etc.) in bigip_base.conf, working out any kinks, loading it (with the interface cables removed), then disabling ARP for all the virtual addresses using a TMSH script, cabling it up, then cutting over one virtual at a time. Let’s walk through the individual steps:
test-ltm-01(Active)(tmos)# save sys ucs test-ltm-01-final-20110817 => Saving active configuration...
test-ltm-01(Active)(tmos)# run util bash [root@test-ltm-01:Active] ~ # scp /var/local/ucs/test-ltm-01-final-20110817.ucs test-ltm-02:/var/local/ucs/.
test-ltm-02(Active)(tmos)# load sys ucs test-ltm-01-final-20110817.ucs => Saving active configuration... Current configuration backed up to /var/local/ucs/cs_backup.ucs. Product : BIG-IP Version : 10.2.1 Hostname: test-ltm-02.f5.com Installing --full-- configuration on host test-ltm-02.f5.com Installing configuration... Installing the license file... Post-processing... Reloading License and configuration - this may take a few minutes... Full configuration has been loaded successfully.
test-ltm-02(Active)(tmos)# run util bash [root@test-ltm-02:Active] ~ # vi /config/bigip_base.conf
[root@test-ltm-02:Active] ~ # edit cli script toggle-virtual-address-arp.tclCode:
1: proc script::run {} {
2: set state [lindex $tmsh::argv 1]
3:
4: if { $state ne "disabled" && $state ne "enabled" } {
5: puts "Requires one argument: \[enabled/disabled\] "
6: } else {
7: foreach {address} [tmsh::get_config /ltm virtual-address] {
8: tmsh::modify /ltm virtual-address [tmsh::get_name $address] arp $state
9: puts "ARP $state for virtual address [tmsh::get_name $address]"
10: }
11: }
12: }
13:
14: proc script::help {} {
15: tmsh::add_help "Enable/Disable ARP for all virtual addresses on this unit\n\ntoggle-virtual-address-arp \[enabled/disabled\]"
16: }
17:
test-ltm-02(Active)(tmos)# run cli script toggle-virtual-address-arp.tcl disabled => ARP disabled for virtual address 10.0.0.202 ARP disabled for virtual address 10.0.0.205 ARP disabled for virtual address 10.0.0.206 ARP disabled for virtual address 10.0.0.240 ARP disabled for virtual address 10.0.0.241 ARP disabled for virtual address 10.0.0.245 ARP disabled for virtual address 10.0.0.249
The Cutover
While you probably could do the cutover by running the toggle-virtual address-arp script on both the source and target units, disabling and enabling respectively, this is probably not such a great idea. The chance of something going haywire is just to great to take that risk. We prefer to go through the virtual servers one-by-one (or at most, just a few at a time) in order to perform the necessary testing and validation during each cutover. Using an iControl script written in Ruby will also help us ensure that our mistakes will be limited and easily reversible in the event of a misstep.
virtual-address-arp-switcher [OPTIONS] -h, --help: show help --bigip-address-source, -b [hostname]: specify the hostname or IP address for source BIG-IP --bigip-user-source, -u [username]: username for source BIG-IP --bigip-pass-source, -p [password]: password for source BIG-IP --bigip-address-target [hostname]: specify the target BIG-IP address --bigip-user-target [username]: username for target BIG-IP, by default assumes same as source BIG-IP --bigip-pass-target [password]: password for target BIG-IP, by default assumes same as source BIG-IP --virtual-address, -v [ip address]: virtual address for which to disable and enable ARP --delay, -d [seconds]: delay between disabling ARP on the source BIG-IP and enabling on target
virtual-address-arp-switcher.rb --bigip-address-source test-ltm-01 --bigip-address-target test-ltm-02 -u admin -p admin -v 10.0.0.202Pretty easy, eh? Once you execute the command, everything will be verified and you'll be provided with a confirmation before performing the failover. When the failover occurs, ARP will be disabled on the source and enabled on the target. As ARP is enabled on the target, it will send a gratuitous ARP in the same fashion as a normal HA failover.
Conclusion
While we can automate a number of the steps involved in the cutover we just demonstrated, there is no substitute for the due diligence involved in testing and validating each step and service as you go. Regardless of how much of this process is automated, it will always remain somewhat tedious and time consuming. When we have done this in the past, we would do 5 – 10 services per maintenance window, let them bake for a few days, then move to the next set. Slow and steady does the trick. Until next time, happy ARPing!
Resources
Toggle Virtual Address ARP script (TMSH Codeshare) - toggles ARP state for all the virtual addresses on a unit
Virtual Address ARP Switcher (iControl Codeshare) - quickly swap virtual addresses between BIG-IPs