Global load balance between two servers (failover), with no preemption

(honestly it was frustrating to not have this feature on the GTMS, and have to reinvent the wheel, for something as simple as enabling/disabling preemption on the pool members)

 

 

Anyway if you are ruining into the same issue this is what I did:

 

 

I end up building an auto resume script,that runs on all of the GTMs on a round robin every minute (on the cron job) or so for redenedancy.

 

 

The script uses snmpwalk to check the pools status on the localhost, and uses the GTM native tmsh command to bring only the pools that are completely down (on manual resume status) up.

 

 

Here it is:

 

 

!/usr/bin/perl

 

================================================================

 

$Id:$

 

$Revision:$ v1.0.0

 

$Author:$ Ahmed Khamari

 

$Date:$ 12-06-2011

 

================================================================

 

Description: Enable pool members that are manually disabled when their pool

 

is completely down, due to a manual resume feature.

 

Combining this with global availability, to Globally failover between two servers will

 

prevent possible flapping issues, if one of the servers goes unstable.

 

================================================================

 

Disclaimer:

 

This script is provided "AS IS", for non-commercial use, and you, its user, assume all risks when using it.

 

================================================================

 

 

my $snmpwalk_prog = '/usr/bin/snmpwalk';

 

my $snmpwalk_cmd =

 

$snmpwalk_prog . ' -v2c -c public localhost gtmPoolStatusAvailState';

 

verify SNMP walk is available

 

die "Program $snmpwalk_prog is not available" unless -x $snmpwalk_prog;

 

open( SNMPWALK, '-|', $snmpwalk_cmd )

 

or die "Failed to run command $snmpwalk_cmd : $!";

 

while () {

 

skip green

 

next if /INTEGER: green/;

 

get pool

 

my ($pool) = /"(.*?)"/;

 

construct command

 

my $cmd = "tmsh modify gtm pool $pool members modify { all { enabled }}";

 

run command

 

print "Running command: $cmd\n";

 

system($cmd) == 0 or die "$cmd failed";

 

}

 

exit(0);

 

Awesome!!!

 

 

Thanks a lot Khamari, I had the same issue, and the GTM persistence issue was driving me crazy, until I read your post! Your solution and script worked like a charm, Thank you! Thank you! Thank you.

 

 

Peter

You're welcome! Glad that I could help!

This is another version of the perl script, that adds a feature that make sure that the backup server is out-of-the-manual resume at all time.

 

 

!/usr/bin/perl

 

 

================================================================

 

Description: Enable pool members that are manually disabled when their pool

 

is completely down, due to a manual resume feature.

 

Combining this with global availability, to Globally failover between two servers will

 

prevent possible flapping issues, if one of the servers goes unstable.

 

================================================================

 

Disclaimer:

 

This script is provided "AS IS", for non-commercial use, and you, its user, assume all risks when using it.

 

================================================================

 

 

 

use strict;

 

use warnings;

 

 

define system commands

 

my $snmpwalk_prog = '/usr/bin/snmpwalk -v2c -c baltimore localhost ';

 

my $pool_status = $snmpwalk_prog . 'gtmPoolStatusAvailState';

 

my $pool_member_order = $snmpwalk_prog . 'gtmPoolMbrOrder';

 

my $pool_member_enabled = $snmpwalk_prog . 'gtmPoolMbrEnabled';

 

my %pools;

 

 

get pool member order

 

open( my $pool_member_order_out, '-|', $pool_member_order )

 

or die "Failed to run command $pool_member_order : $!";

 

while (<$pool_member_order_out>) {

 

my ( $pool, $member ) = / "(.*?)" [.]ipv4[.] "(.*?)" /x;

 

my ($member_order) = / (\d) $ /x;

 

$pools{$pool}{MEMBER}{$member} = $member_order;

 

}

 

 

get pool member enable status

 

open( my $pool_member_enabled_out, '-|', $pool_member_enabled )

 

or die "Failed to run command $pool_member_enabled : $!";

 

while (<$pool_member_enabled_out>) {

 

my ( $pool, $member ) = / "(.*?)" [.]ipv4[.] "(.*?)" /x;

 

my ($status) = / INTEGER: \s+ ( enable | disable ) /x;

 

my $member_order = $pools{$pool}{MEMBER}{$member};

 

$pools{$pool}{$member_order} = $status;

 

}

 

 

get pool status

 

open( my $pool_status_out, '-|', $pool_status )

 

or die "Failed to run command $pool_status : $!";

 

while (<$pool_status_out>) {

 

 

get pool

 

my ($pool) = /"(.*?)"/;

 

 

pool status

 

my ($pool_status) = / INTEGER: \s+ ( green | red ) /x;

 

 

pool status is red or member 1 is disabled

 

if ( ( $pool_status eq 'red' )

 

or ( exists $pools{$pool}{1} and $pools{$pool}{1} eq 'disable' ) )

 

{

 

construct command

 

my $cmd =

 

"tmsh modify gtm pool $pool members modify { all { enabled }}";

 

 

run command

 

print "Running command: $cmd\n";

 

system($cmd) == 0 or die "$cmd failed";

 

}

 

}

!/usr/bin/perl

 

================================================================

 

Description: Enable pool members that are manually disabled when their pool

 

is completely down, due to a manual resume feature.

 

Applies only to pools with Global Availability and Manual Resume

 

Combining this with global availability, to Globally failover between two servers will

 

prevent possible flapping issues, if one of the servers goes unstable.

 

================================================================

 

Disclaimer:

 

This script is provided "AS IS", for non-commercial use, and you, its user, assume all risks when using it.

 

================================================================

 

 

Logic for running command:

 

mode is "global availability"

 

AND

 

manual resume is "enable" )

 

AND

 

(

 

pool status = "red"

 

OR

 

( A member is pool has order 1 AND same member is "disabled" )

 

)

 

 

==================================================

 

use strict;

 

use warnings;

 

use Getopt::Long;

 

 

my $usage = <<"END_USAGE";

 

$0 [--help | --?] [--debug]

 

Where: --help - Displays usage

 

--debug - Display debug information

 

END_USAGE

 

 

get command line options

 

my $help;

 

my $debug;

 

GetOptions(

 

"help|?" => \$help,

 

"debug" => \$debug

 

);

 

 

( print $usage and exit(0) ) if $help;

 

 

define system commands

 

my $snmpwalk_prog = '/usr/bin/snmpwalk -v2c -c baltimore localhost ';

 

my $pool_status = $snmpwalk_prog . 'gtmPoolStatusAvailState';

 

my $pool_order = $snmpwalk_prog . 'gtmPoolMbrOrder';

 

my $pool_enabled = $snmpwalk_prog . 'gtmPoolMbrEnabled';

 

my $pool_mode = $snmpwalk_prog . 'gtmPoolLbMode';

 

my $pool_manual_resume = $snmpwalk_prog . 'gtmPoolManualResume';

 

my %pools;

 

 

get pool member order number

 

print ("Command: $pool_order\n") if $debug;

 

open( my $pool_order_out, '-|', $pool_order )

 

or die "Failed to run command $pool_order : $!";

 

while (<$pool_order_out>) {

 

my ( $pool, $ip, $port ) = / "(.*?)" [.]ipv4[.] "(.*?)" [.] (\d)+ /x;

 

my $member = $ip . '.' . $port;

 

my ($member_order) = / (\d) $ /x;

 

$pools{$pool}{MEMBER}{$member} = $member_order;

 

}

 

 

determine if pool member is enabled

 

print ("Command: $pool_enabled\n") if $debug;

 

open( my $pool_enabled_out, '-|', $pool_enabled )

 

or die "Failed to run command $pool_enabled : $!";

 

while (<$pool_enabled_out>) {

 

my ( $pool, $ip, $port ) = / "(.*?)" [.]ipv4[.] "(.*?)" [.] (\d)+ /x;

 

my $member = $ip . '.' . $port;

 

my $is_enabled = / INTEGER: \s+ enable /x ? 1 : 0;

 

my $member_order = $pools{$pool}{MEMBER}{$member};

 

$pools{$pool}{MEMBER_ORDER}{$member_order} = $is_enabled;

 

}

 

 

get pool status

 

print ("Command: $pool_status\n") if $debug;

 

open( my $pool_status_out, '-|', $pool_status )

 

or die "Failed to run command $pool_status : $!";

 

while (<$pool_status_out>) {

 

 

get pool

 

my ($pool) = /"(.*?)"/;

 

 

pool status

 

my ($pool_status) = / INTEGER: \s+ ( green | red ) /x;

 

$pools{$pool}{STATUS} = $pool_status;

 

}

 

 

get pool mode

 

print ("Command: $pool_mode\n") if $debug;

 

open( my $pool_mode_out, '-|', $pool_mode )

 

or die "Failed to run command $pool_mode : $!";

 

while (<$pool_mode_out>) {

 

 

get pool

 

my ($pool) = /"(.*?)"/;

 

 

pool status

 

my ($pool_mode) = / INTEGER: \s+ ( \w+ ) /x;

 

$pools{$pool}{MODE} = $pool_mode;

 

}

 

 

get manual resume

 

print ("Command: $pool_manual_resume\n") if $debug;

 

open( my $pool_manual_resume_out, '-|', $pool_manual_resume )

 

or die "Failed to run command $pool_manual_resume : $!";

 

while (<$pool_manual_resume_out>) {

 

 

get pool

 

my ($pool) = /"(.*?)"/;

 

 

is pool manual resume

 

my $is_manual_resume = / INTEGER: \s+ enable /x ? 1 : 0;

 

$pools{$pool}{IS_MANUAL_RESUME} = $is_manual_resume;

 

}

 

 

loop through pools and run command as required

 

POOL:

 

for my $pool ( keys %pools ) {

 

 

my $status = $pools{$pool}{STATUS};

 

my $mode = $pools{$pool}{MODE};

 

my $is_manual_resume = $pools{$pool}{IS_MANUAL_RESUME};

 

my $is_member_one_disabled =

 

exists $pools{$pool}{MEMBER_ORDER}{1}

 

&& $pools{$pool}{MEMBER_ORDER}{1} == 0

 

? 1

 

: 0;

 

 

print(

 

"\n", $pool,

 

" mode:", $mode,

 

" is manual resume:", $is_manual_resume,

 

" status:", $status,

 

" is member 1 disabled:", $is_member_one_disabled,

 

"\n"

 

) if $debug;

 

 

run test

 

if ( $mode eq 'ga'

 

and $is_manual_resume

 

and ( $status eq 'red' or $is_member_one_disabled ) )

 

{

 

 

construct command

 

my $cmd =

 

"tmsh modify gtm pool $pool members modify { all { enabled }}";

 

 

if ($debug) {

 

print("\n===\nCommand: $cmd\n===\n");

 

next POOL;

 

}

 

 

run command

 

system($cmd) == 0 or die "$cmd failed";

 

}

 

}

 

 

 

exit(0);