Ruby and iControl: Migrating Virtual Addresses Using ARP

Introduction

Ever wanted to “failover” a virtual server to an LTM that is not part of the same HA (high availability) pair? This is a common occurrence for any engineer that has ever had to upgrade their BIG-IP hardware platform or been part of a “consolidation” project. If the virtual is being addressed via DNS this can be as simple as creating the virtual server (using a different IP) and its dependencies on an adjacent unit, then cutting over by updating the A record. Most of us know that it is never that simple. The IP address itself is almost always hardcoded in a number of configurations and therefore we have to move the IP itself with minimal impact on our application. Enter virtual address failover via gratuitous ARP.

There are a number of ways to facilitate this action, but for the sake of expediency and repeatability we’ll be using a script to perform the failover. However, before we start failing things over, lets take a look at what we need to bring our virtual address up on our target unit. It is important at this point that you understand the difference between a virtual address and a virtual server. A virtual server consists of a virtual address and a port, therefore a virtual server can only have one virtual address. A virtual address on the other hand can have any number (up to the number of available ports and protocols) of virtual servers attached to it. The virtual address is often ignored as it is automatically created with the first virtual server referencing that address.

Recreating the Virtual Address And Server on the Target Unit

This method can be fairly labor intensive, but if you’re only moving a few virtual servers it is the most convenient as well as a great way to clean up your configurations during the migration. Before we go into the specific steps, here is what we’re going to be doing: creating a virtual address with ARP disabled, copying the virtual server config(s) and all the dependencies (pools, profiles, health monitors, etc.). Let’s walk through some of the steps involved in this migration.

1. Identify the virtual address, virtual server and dependencies to be moved. Open up an SSH connection to your source BIG-IP and fire up TMSH (if TMSH is not your default shell, it should be). If you’re in BASH by default, type ‘tmsh’ to start the Traffic Management Shell.
2. Use the ‘list’ command to list the virtual server config:
   

   test-ltm-01(Active)(tmos)# list ltm virtual test_http_virtual =>
   
   ltm virtual test_http_virtual {
       destination 10.0.0.200:http
       ip-protocol tcp
       mask 255.255.255.255
       mirror enabled
       pool test_http_pool
       profiles {
           http { }
           tcp-lan-optimized { }
       }
       snat automap
   }

3. In this sample configuration, our migration should be pretty straightforward. Most of the dependencies are built-in to the LTM by default. The only element that needs further examination will be the ‘test_http_pool’ pool. We can list its configuration in much the same way that we did the virtual server:
   

   test-ltm-01(Active)(tmos)# list ltm pool test_http_pool =>
   
   ltm pool test_http_pool {
       members {
           10.0.0.131:http {
               state up
           }
           10.0.0.132:http {
               state up
           }
           10.0.0.133:http {
               session disabled
               state down
           }
       }
       monitor http
   }

4. Now that we’ve identified the resources we’ll need to create on our target LTM, we can SSH over that unit and get started. Once again, you’ll want to fire up TMSH to get started.
5. Create the virtual address with ARP disabled:
   

   create ltm virtual-address 10.0.0.200 arp disabled

   *Make sure that you create the virtual address with ARP disabled prior to creating any virtual servers otherwise the virtual address will be created automatically as will be an IP conflict (/var/log/ltm will report and IP conflicts). This action must be done from the CLI as there is no GUI option to disable ARP on virtual address creation.
6. Next we'll create the ‘test_http_pool’ pool on the target LTM. Note the 'edit' command here, it will spawn a text editor:
   

   test-ltm-02(Active)(tmos)# edit ltm pool test_http_pool =>
   
   create pool test_http_pool {
       members add {
           10.0.0.131:http {
               state up
           }
           10.0.0.132:http {
               state up
           }
           10.0.0.133:http {
               session disabled
               state down
           }
       }
       monitor http
   }

   *Also node the 'add' statement after the members component. This is necessary when adding new pool members.
7. Finally, we’ll create our virtual server:
   

   test-ltm-02(Active)(tmos)# edit ltm virtual test_http_virtual =>
   
   create virtual test_http_virtual {
       destination 10.0.0.200:http
       ip-protocol tcp
       mask 255.255.255.255
       mirror enabled
       pool test_http_pool
       profiles add {
           http { }
           tcp-lan-optimized { }
       }
       snat automap
   }

   *Note the 'add' statement after profiles.
8. We’ll address the next series of steps in “The Cutover” section.

Alternative Approach – Moving The Entire Configuration

As was mentioned at the beginning of this section, this approach may not be feasible if you are dealing with hundreds or thousands of virtual addresses and servers. In that case you’ll probably want to go about dumping the entire configuration, moving it to the new unit(s), changing the unit-specific values (self-IPs, hostname, etc.) in bigip_base.conf, working out any kinks, loading it (with the interface cables removed), then disabling ARP for all the virtual addresses using a TMSH script, cabling it up, then cutting over one virtual at a time. Let’s walk through the individual steps:

1. Save a UCS archive from your source unit(s):
   

   test-ltm-01(Active)(tmos)# save sys ucs test-ltm-01-final-20110817 =>
   
   Saving active configuration...

2. Breakout to BASH and SCP the archive to the target unit:
   

   test-ltm-01(Active)(tmos)# run util bash
   
   [root@test-ltm-01:Active] ~ # scp /var/local/ucs/test-ltm-01-final-20110817.ucs test-ltm-02:/var/local/ucs/.

3. Make sure that all of the interface cables are removed for the time being, then load the UCS on the target unit:
   

   test-ltm-02(Active)(tmos)# load sys ucs test-ltm-01-final-20110817.ucs =>
   
   Saving active configuration...
   Current configuration backed up to /var/local/ucs/cs_backup.ucs.
   Product : BIG-IP
   Version : 10.2.1
   Hostname: test-ltm-02.f5.com
   Installing --full-- configuration on host test-ltm-02.f5.com
   Installing configuration...
   Installing the license file...
   Post-processing...
   Reloading License and configuration - this may take a few minutes...
   Full configuration has been loaded successfully.

4. Make necessary changes to bigip_base.conf, save, and load it:
   

   test-ltm-02(Active)(tmos)# run util bash
   
   [root@test-ltm-02:Active] ~ # vi /config/bigip_base.conf

5. Use the toggle-virtual-address-arp TMSH script to disable ARP for all of the virtual addresses on the unit. Copy the code from the script (may be easier to copy from the Codeshare) and paste it into the text editor via TMSH:
   

   [root@test-ltm-02:Active] ~ # edit cli script toggle-virtual-address-arp.tcl

   Code:
   

      1: proc script::run {} {

      2:     set state [lindex $tmsh::argv 1]

      3: 

      4:     if { $state ne "disabled" && $state ne "enabled" } {

      5:         puts "Requires one argument: \[enabled/disabled\] "

      6:     } else {

      7:         foreach {address} [tmsh::get_config /ltm virtual-address] {

      8:             tmsh::modify /ltm virtual-address [tmsh::get_name $address] arp $state

      9:             puts "ARP $state for virtual address [tmsh::get_name $address]"

     10:         }

     11:     }

     12: }

     13: 

     14: proc script::help {} {

     15:     tmsh::add_help "Enable/Disable ARP for all virtual addresses on this unit\n\ntoggle-virtual-address-arp \[enabled/disabled\]"

     16: }

     17: 

6. Once you have toggle-virtual-address-arp.tcl script in place on your target LTM, run it to disable ARP for all the virtual addresses:
   

   test-ltm-02(Active)(tmos)# run cli script toggle-virtual-address-arp.tcl disabled =>
   
   ARP disabled for virtual address 10.0.0.202
   ARP disabled for virtual address 10.0.0.205
   ARP disabled for virtual address 10.0.0.206
   ARP disabled for virtual address 10.0.0.240
   ARP disabled for virtual address 10.0.0.241
   ARP disabled for virtual address 10.0.0.245
   ARP disabled for virtual address 10.0.0.249

7. Re-attach all the interface cables and closely monitor ‘/var/log/ltm’ on the source unit for any duplicate IP addresses. If any appear, immediately remove the interface cable and resolve the issue. All traffic at this point should still be flowing through the original source unit. Check your virtual server statistics to verify this.

The Cutover

While you probably could do the cutover by running the toggle-virtual address-arp script on both the source and target units, disabling and enabling respectively, this is probably not such a great idea. The chance of something going haywire is just to great to take that risk. We prefer to go through the virtual servers one-by-one (or at most, just a few at a time) in order to perform the necessary testing and validation during each cutover. Using an iControl script written in Ruby will also help us ensure that our mistakes will be limited and easily reversible in the event of a misstep.

1. Start by getting the F5 iControl for Ruby library installed on your workstation. Check out Getting Started With Ruby and iControl.
2. Grab the code for the Virtual Address ARP Switcher from the Codeshare and copy it to your workstation.
3. The virtual-address-arp-switcher script takes a number of command line options. Familiarize yourself with them:
   

   virtual-address-arp-switcher [OPTIONS]
   
   -h, --help:
   
      show help
   
   --bigip-address-source, -b [hostname]:
   
      specify the hostname or IP address for source BIG-IP
   
   --bigip-user-source, -u [username]:
   
      username for source BIG-IP
   
   --bigip-pass-source, -p [password]:
   
      password for source BIG-IP
   
   --bigip-address-target [hostname]:
   
      specify the target BIG-IP address
   
   --bigip-user-target [username]:
   
      username for target BIG-IP, by default assumes same as source BIG-IP
   
   --bigip-pass-target [password]:
   
      password for target BIG-IP, by default assumes same as source BIG-IP
   
   --virtual-address, -v [ip address]:
   
      virtual address for which to disable and enable ARP
   
   --delay, -d [seconds]:
   
      delay between disabling ARP on the source BIG-IP and enabling on target

4. Pick a virtual that you would like to relocate from your source unit to your target LTM. In this case, we want to move the virtual address 10.0.0.202 from test-ltm-01 to test-ltm-02. The ‘admin’ credentials are the same on both boxes, so we only need to specify them once. This is what our input would look like:
   

   virtual-address-arp-switcher.rb --bigip-address-source test-ltm-01 --bigip-address-target test-ltm-02 -u admin -p admin -v 10.0.0.202

   Pretty easy, eh? Once you execute the command, everything will be verified and you'll be provided with a confirmation before performing the failover. When the failover occurs, ARP will be disabled on the source and enabled on the target. As ARP is enabled on the target, it will send a gratuitous ARP in the same fashion as a normal HA failover.
5. At this point the virtual address and the associated virtual servers should have moved to the target unit. Check your statistics and make sure they are incrementing as expected. Once you have moved everything over, reset your statistics counters on the source box and let them run for a few days to ensure that nothing was left behind.

Conclusion

While we can automate a number of the steps involved in the cutover we just demonstrated, there is no substitute for the due diligence involved in testing and validating each step and service as you go. Regardless of how much of this process is automated, it will always remain somewhat tedious and time consuming. When we have done this in the past, we would do 5 – 10 services per maintenance window, let them bake for a few days, then move to the next set. Slow and steady does the trick. Until next time, happy ARPing!

Resources

Toggle Virtual Address ARP script (TMSH Codeshare) - toggles ARP state for all the virtual addresses on a unit

Virtual Address ARP Switcher (iControl Codeshare) - quickly swap virtual addresses between BIG-IPs