Forum Discussion

Khamari_2736's avatar
Khamari_2736
Icon for Nimbostratus rankNimbostratus
Nov 28, 2011

Global load balance between two servers (failover), with no preemption

Hello Group,

 

 

We would like to global load balance between two servers (failover), with no preemption. Meaning that the WideIP will resolve always to the same server until that server in question goes down, and regardless of the status of the other server; so we can avoid any flapping issues.

 

We tried persistence with (Static Persist CIDR IPv4 = 0) but it doesn't cut it, especially if requests are not coming from all the clients at the time of the failover. This causes the persistence table records (of the same wideip) to point to different servers and not to the same one. Not good.

 

For now I’m using the global availability with fallback to primary pool member and manual resume enabled on the pools. This ensures that every request resolves to the same pool member until it goes down. The down side of this is that we are talking about hundreds of servers. Checking all pool memebers, adn manualy taking them from the manual resume status is over kill. we just don't have the man power for this tidious task, and it opens us to human errors/ issues ..etc. so automating this process is vital to us.

 

That being said, I would like assistance in enabling all pool members (of a specific pool )that went to a manual resume status using an iRule, when all of them (the pool members of this specific pool) are all in the manual resume status.

 

Something like:

 

when DNS_REQUEST {

 

if { {[active_members Smarts] < 1} } {

 

set [LB::status pool Smarts member 172.17.187.167 426 up]

 

set [LB::status pool Smarts member 10.24.180.1 426 up]

 

}

 

}

 

Or

 

when LB_FAILED {

 

pool Smarts

 

set [LB::status pool Smarts member 172.17.187.167 426 up]

 

set [LB::status pool Smarts member 10.24.180.1 426 up]

 

}

 

 

Not sure about the syntax of the Global Traffic I don’t see many example for GTMs iRules and set [LB::status pool Smarts member ip port up] seems to be ineffective on the GTM irule.

 

Thank you

 

  • (honestly it was frustrating to not have this feature on the GTMS, and have to reinvent the wheel, for something as simple as enabling/disabling preemption on the pool members)

     

     

    Anyway if you are ruining into the same issue this is what I did:

     

     

    I end up building an auto resume script,that runs on all of the GTMs on a round robin every minute (on the cron job) or so for redenedancy.

     

     

    The script uses snmpwalk to check the pools status on the localhost, and uses the GTM native tmsh command to bring only the pools that are completely down (on manual resume status) up.

     

     

    Here it is:

     

     

    !/usr/bin/perl

     

    ================================================================

     

    $Id:$

     

    $Revision:$ v1.0.0

     

    $Author:$ Ahmed Khamari

     

    $Date:$ 12-06-2011

     

    ================================================================

     

    Description: Enable pool members that are manually disabled when their pool

     

    is completely down, due to a manual resume feature.

     

    Combining this with global availability, to Globally failover between two servers will

     

    prevent possible flapping issues, if one of the servers goes unstable.

     

    ================================================================

     

    Disclaimer:

     

    This script is provided "AS IS", for non-commercial use, and you, its user, assume all risks when using it.

     

    ================================================================

     

     

    my $snmpwalk_prog = '/usr/bin/snmpwalk';

     

    my $snmpwalk_cmd =

     

    $snmpwalk_prog . ' -v2c -c public localhost gtmPoolStatusAvailState';

     

    verify SNMP walk is available

     

    die "Program $snmpwalk_prog is not available" unless -x $snmpwalk_prog;

     

    open( SNMPWALK, '-|', $snmpwalk_cmd )

     

    or die "Failed to run command $snmpwalk_cmd : $!";

     

    while () {

     

    skip green

     

    next if /INTEGER: green/;

     

    get pool

     

    my ($pool) = /"(.*?)"/;

     

    construct command

     

    my $cmd = "tmsh modify gtm pool $pool members modify { all { enabled }}";

     

    run command

     

    print "Running command: $cmd\n";

     

    system($cmd) == 0 or die "$cmd failed";

     

    }

     

    exit(0);

     

  • Awesome!!!

     

     

    Thanks a lot Khamari, I had the same issue, and the GTM persistence issue was driving me crazy, until I read your post! Your solution and script worked like a charm, Thank you! Thank you! Thank you.

     

     

    Peter
  • This is another version of the perl script, that adds a feature that make sure that the backup server is out-of-the-manual resume at all time.

     

     

    !/usr/bin/perl

     

     

    ================================================================

     

    Description: Enable pool members that are manually disabled when their pool

     

    is completely down, due to a manual resume feature.

     

    Combining this with global availability, to Globally failover between two servers will

     

    prevent possible flapping issues, if one of the servers goes unstable.

     

    ================================================================

     

    Disclaimer:

     

    This script is provided "AS IS", for non-commercial use, and you, its user, assume all risks when using it.

     

    ================================================================

     

     

     

    use strict;

     

    use warnings;

     

     

    define system commands

     

    my $snmpwalk_prog = '/usr/bin/snmpwalk -v2c -c baltimore localhost ';

     

    my $pool_status = $snmpwalk_prog . 'gtmPoolStatusAvailState';

     

    my $pool_member_order = $snmpwalk_prog . 'gtmPoolMbrOrder';

     

    my $pool_member_enabled = $snmpwalk_prog . 'gtmPoolMbrEnabled';

     

    my %pools;

     

     

    get pool member order

     

    open( my $pool_member_order_out, '-|', $pool_member_order )

     

    or die "Failed to run command $pool_member_order : $!";

     

    while (<$pool_member_order_out>) {

     

    my ( $pool, $member ) = / "(.*?)" [.]ipv4[.] "(.*?)" /x;

     

    my ($member_order) = / (\d) $ /x;

     

    $pools{$pool}{MEMBER}{$member} = $member_order;

     

    }

     

     

    get pool member enable status

     

    open( my $pool_member_enabled_out, '-|', $pool_member_enabled )

     

    or die "Failed to run command $pool_member_enabled : $!";

     

    while (<$pool_member_enabled_out>) {

     

    my ( $pool, $member ) = / "(.*?)" [.]ipv4[.] "(.*?)" /x;

     

    my ($status) = / INTEGER: \s+ ( enable | disable ) /x;

     

    my $member_order = $pools{$pool}{MEMBER}{$member};

     

    $pools{$pool}{$member_order} = $status;

     

    }

     

     

    get pool status

     

    open( my $pool_status_out, '-|', $pool_status )

     

    or die "Failed to run command $pool_status : $!";

     

    while (<$pool_status_out>) {

     

     

    get pool

     

    my ($pool) = /"(.*?)"/;

     

     

    pool status

     

    my ($pool_status) = / INTEGER: \s+ ( green | red ) /x;

     

     

    pool status is red or member 1 is disabled

     

    if ( ( $pool_status eq 'red' )

     

    or ( exists $pools{$pool}{1} and $pools{$pool}{1} eq 'disable' ) )

     

    {

     

    construct command

     

    my $cmd =

     

    "tmsh modify gtm pool $pool members modify { all { enabled }}";

     

     

    run command

     

    print "Running command: $cmd\n";

     

    system($cmd) == 0 or die "$cmd failed";

     

    }

     

    }
  • !/usr/bin/perl

     

    ================================================================

     

    Description: Enable pool members that are manually disabled when their pool

     

    is completely down, due to a manual resume feature.

     

    Applies only to pools with Global Availability and Manual Resume

     

    Combining this with global availability, to Globally failover between two servers will

     

    prevent possible flapping issues, if one of the servers goes unstable.

     

    ================================================================

     

    Disclaimer:

     

    This script is provided "AS IS", for non-commercial use, and you, its user, assume all risks when using it.

     

    ================================================================

     

     

    Logic for running command:

     

    mode is "global availability"

     

    AND

     

    manual resume is "enable" )

     

    AND

     

    (

     

    pool status = "red"

     

    OR

     

    ( A member is pool has order 1 AND same member is "disabled" )

     

    )

     

     

    ==================================================

     

    use strict;

     

    use warnings;

     

    use Getopt::Long;

     

     

    my $usage = <<"END_USAGE";

     

    $0 [--help | --?] [--debug]

     

    Where: --help - Displays usage

     

    --debug - Display debug information

     

    END_USAGE

     

     

    get command line options

     

    my $help;

     

    my $debug;

     

    GetOptions(

     

    "help|?" => \$help,

     

    "debug" => \$debug

     

    );

     

     

    ( print $usage and exit(0) ) if $help;

     

     

    define system commands

     

    my $snmpwalk_prog = '/usr/bin/snmpwalk -v2c -c baltimore localhost ';

     

    my $pool_status = $snmpwalk_prog . 'gtmPoolStatusAvailState';

     

    my $pool_order = $snmpwalk_prog . 'gtmPoolMbrOrder';

     

    my $pool_enabled = $snmpwalk_prog . 'gtmPoolMbrEnabled';

     

    my $pool_mode = $snmpwalk_prog . 'gtmPoolLbMode';

     

    my $pool_manual_resume = $snmpwalk_prog . 'gtmPoolManualResume';

     

    my %pools;

     

     

    get pool member order number

     

    print ("Command: $pool_order\n") if $debug;

     

    open( my $pool_order_out, '-|', $pool_order )

     

    or die "Failed to run command $pool_order : $!";

     

    while (<$pool_order_out>) {

     

    my ( $pool, $ip, $port ) = / "(.*?)" [.]ipv4[.] "(.*?)" [.] (\d)+ /x;

     

    my $member = $ip . '.' . $port;

     

    my ($member_order) = / (\d) $ /x;

     

    $pools{$pool}{MEMBER}{$member} = $member_order;

     

    }

     

     

    determine if pool member is enabled

     

    print ("Command: $pool_enabled\n") if $debug;

     

    open( my $pool_enabled_out, '-|', $pool_enabled )

     

    or die "Failed to run command $pool_enabled : $!";

     

    while (<$pool_enabled_out>) {

     

    my ( $pool, $ip, $port ) = / "(.*?)" [.]ipv4[.] "(.*?)" [.] (\d)+ /x;

     

    my $member = $ip . '.' . $port;

     

    my $is_enabled = / INTEGER: \s+ enable /x ? 1 : 0;

     

    my $member_order = $pools{$pool}{MEMBER}{$member};

     

    $pools{$pool}{MEMBER_ORDER}{$member_order} = $is_enabled;

     

    }

     

     

    get pool status

     

    print ("Command: $pool_status\n") if $debug;

     

    open( my $pool_status_out, '-|', $pool_status )

     

    or die "Failed to run command $pool_status : $!";

     

    while (<$pool_status_out>) {

     

     

    get pool

     

    my ($pool) = /"(.*?)"/;

     

     

    pool status

     

    my ($pool_status) = / INTEGER: \s+ ( green | red ) /x;

     

    $pools{$pool}{STATUS} = $pool_status;

     

    }

     

     

    get pool mode

     

    print ("Command: $pool_mode\n") if $debug;

     

    open( my $pool_mode_out, '-|', $pool_mode )

     

    or die "Failed to run command $pool_mode : $!";

     

    while (<$pool_mode_out>) {

     

     

    get pool

     

    my ($pool) = /"(.*?)"/;

     

     

    pool status

     

    my ($pool_mode) = / INTEGER: \s+ ( \w+ ) /x;

     

    $pools{$pool}{MODE} = $pool_mode;

     

    }

     

     

    get manual resume

     

    print ("Command: $pool_manual_resume\n") if $debug;

     

    open( my $pool_manual_resume_out, '-|', $pool_manual_resume )

     

    or die "Failed to run command $pool_manual_resume : $!";

     

    while (<$pool_manual_resume_out>) {

     

     

    get pool

     

    my ($pool) = /"(.*?)"/;

     

     

    is pool manual resume

     

    my $is_manual_resume = / INTEGER: \s+ enable /x ? 1 : 0;

     

    $pools{$pool}{IS_MANUAL_RESUME} = $is_manual_resume;

     

    }

     

     

    loop through pools and run command as required

     

    POOL:

     

    for my $pool ( keys %pools ) {

     

     

    my $status = $pools{$pool}{STATUS};

     

    my $mode = $pools{$pool}{MODE};

     

    my $is_manual_resume = $pools{$pool}{IS_MANUAL_RESUME};

     

    my $is_member_one_disabled =

     

    exists $pools{$pool}{MEMBER_ORDER}{1}

     

    && $pools{$pool}{MEMBER_ORDER}{1} == 0

     

    ? 1

     

    : 0;

     

     

    print(

     

    "\n", $pool,

     

    " mode:", $mode,

     

    " is manual resume:", $is_manual_resume,

     

    " status:", $status,

     

    " is member 1 disabled:", $is_member_one_disabled,

     

    "\n"

     

    ) if $debug;

     

     

    run test

     

    if ( $mode eq 'ga'

     

    and $is_manual_resume

     

    and ( $status eq 'red' or $is_member_one_disabled ) )

     

    {

     

     

    construct command

     

    my $cmd =

     

    "tmsh modify gtm pool $pool members modify { all { enabled }}";

     

     

    if ($debug) {

     

    print("\n===\nCommand: $cmd\n===\n");

     

    next POOL;

     

    }

     

     

    run command

     

    system($cmd) == 0 or die "$cmd failed";

     

    }

     

    }

     

     

     

    exit(0);