Geolocation data auto-update

I wrote this last night, and it works for us -- feel free to use it -- though I recommend you test it in your environment first.

!/usr/bin/perl -w
 Download the latest GeoIP files for F5 LTM and install them. This script runs
 from a supporting host.

 Requires:
  * perl (WWW::Mechanize and its dependencies)
  * A ssh suite with passwordless authentication via key to the load balancers
  * unzip
  * md5sum

 This script will be fragile, since it relies upon minimal changes to the F5
 website. It should be reasonably safe, given it checks the md5sum of the
 downloaded file, but use at your own risk.

 Configure the $user and $pass variables for your login to the F5 site.
 Configure the @servers list for the servers to SSH to and update the DBs on

use WWW::Mechanize;

 Configuration section - Update these

my $user = 'F5_LOGIN_USERNAME';
my $pass = 'F5_PASSWORD';
my $uri = 'https://downloads.f5.com/esd/ecc.sv?sw=BIG-IP&pro=big-ip_v11.x&ver=11.6.0&container=GeoLocationUpdates';
my @servers = qw(SERVER1 SERVER2);
my $dir = '/tmp/f5geoip';

 End configuration - changes shouldn't be necessary below unless the F5 site changes.

die "\$dir needs set to something that doesn't exist\n" if($dir eq '' || $dir eq '/' || -e $dir);  Sanity check....
mkdir($dir);

 Login to F5, download the GeoIP update and md5sum
my $mech = WWW::Mechanize->new( cookie_jar => {} );
$mech->get($uri);
$mech->submit_form(
  form_name => 'login',
  fields => {
    userid => $user,
    passwd => $pass
  }
);

$mech->submit_form(
  form_name => 'LicenseAgreement'
);

$mech->follow_link(url_regex => qr/\.zip$/);
$mech->follow_link(url_regex => qr/\.zip/);
$mech->save_content("$dir/f5geoip.zip");
$mech->get($uri);
$mech->follow_link(url_regex => qr/\.zip\.md5$/);
$mech->follow_link(url_regex => qr/\.zip\.md5/);
$mech->save_content("$dir/f5geoip.zip.md5");

 Validate the file contents against md5sum and extract
my $sum = `cat $dir/f5geoip.zip.md5 | cut -d ' ' -f 1`;
my $filesum = `md5sum $dir/f5geoip.zip | cut -d ' ' -f 1`;
chomp $sum;
chomp $filesum;
die "Invalid checksum on GeoIP Database\n" if ($sum ne $filesum);

my $ret = system("unzip $dir/f5geoip.zip -d $dir >/dev/null") >> 8;
die "zip didn't properly extract\n" if($ret != 0);

 Generate a list of the RPMs
my @files = split("\n", `ls -1 $dir | grep rpm`);

 Copy the files to the LB via SCP and then run the updater
foreach my $server (@servers) {
  foreach my $file (@files) {
    system("scp $dir/$file $server:/shared/tmp > /dev/null");
    system("ssh $server 'geoip_update_data -f /shared/tmp/$file; rm -f /shared/tmp/$file'");
  }
}

 Cleanup
system("rm -f $dir/*; rmdir $dir");

Has anything changed here where F5 supports auto-updates for this? If attack signature updates can be automated for ASM, doing the same for geo-location on GTM shouldn't be a stretch.

This is a problem for me too. If the bigIP cannot update the database automatically, is there a list we can subscribe to so that we know when there is a new version? That alone would help a bunch!

Cheers, Mike

The above Perl script didn't work for me, and since I know PHP better, I made a CLI script to do the job. Its been tested on 12.1.2 and it works great!!

It takes 1 param, which is the https file link. An optional 2nd param is for the download link for the MD5 hash file. without this, it doesn't do the check.

Here is a link to the script on my GitHub. I know you rather have the code here, but this way i can update it later if problems are found.

Github - C2theG - GeoIP_Updater

Here is bash script that automatically downloads and installs the latest Geolocation database:

F5 Geolocation Database update script

This script will sign into downloads.f5.com (using existing credentials), and download the latest appropriate geolocation database for the BIG-IP release and install the update. This should work on all current supported BIG-IP releases.

Place the script in /etc/cron.weekly or /etc/cron.daily and ensure that it is executable. It stores the name of the last update installed in /var/tmp/geo and only downloads if a new file is available.

If you need to use a proxy to get to the download site, you can set a proxy options variable.

As the credentials used to access downloads.f5.com are stored in the script, I recommend creating a new set of credentials for this purpose, and only using those credentials for this purpose.

Set the variables

downloads_user=""

downloads_password=""

downloads_from="USA - WEST COAST"

proxy_opts=""

The script logs into /var/log/ltm when it runs, downloads and installs the database. It also logs errors, such as login failure and database backup errors.

If the F5 Downloads site changes or restructures, the script will probably fail.

#! /bin/bash
# f5 geolocation updater
# linuxtech@mail.com
# free for use 
logger -p local0.notice "Geolocation update file check - checking for updates"
base_dir="/var/tmp/geo"
if [[ ! -e $base_dir ]]; then
    mkdir -p $base_dir
fi
cd $base_dir
if [[ -e "$base_dir/geo_cookies.txt" ]]; then
  /bin/rm -f "$base_dir/geo_cookies.txt"
fi
fullversion=$(tmsh show sys version | grep " Version" | awk '{ print $2 }')
# echo $fullversion
baseversion=$(echo $fullversion | awk -F. '{ print $1 }')
# echo $baseversion
containerversion=$(echo $fullversion | awk -F. '{ print $1"."$2"."$3 }')
# backup the current Geolocation database
dir="/shared/GeoIP_backup"
if [[ ! -e $dir ]]; then
    mkdir -p $dir
elif [[ ! -d $dir ]]; then
	logger -p local0.err "Geolocation update file check - error backing up Geolocation database: $dir already exists but is not a directory"
	exit
fi
if [ $baseversion -ge 15 ]; then
  /bin/cp -R /shared/GeoIP/* /shared/GeoIP_backup/
else
  /bin/cp -R /usr/share/GeoIP/* /shared/GeoIP_backup/
fi 
# echo $containerversion
container="sw=BIG-IP&pro=big-ip_v$baseversion.x&ver=$containerversion&container=GeoLocationUpdates"
# echo $container
downloads_user=""
downloads_password=""
downloads_from="USA - WEST COAST"
# specify any curl proxy options as required
# eg --proxy http://user:password@host:port/
# or
# "" for direct connect
proxy_opts=""
# get the login page
loginpage=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent https://downloads.f5.com/esd/ecc.sv?$container 2>&1 | grep "action=" | awk -F'[=\"|\">]' '{ print $3 }' )
# echo $loginpage
# submit the credentials
afterlogin=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent $loginpage -X POST --data-urlencode "userid=$downloads_user" --data-urlencode "passwd=$downloads_password" 2>&1 | grep "F5 Networks - My Account" | awk -F'[="|">]' '{ print $6 }' )
# echo $afterlogin
if [[ $afterlogin == "" ]]; then
  logger -p local0.err "Geolocation update file check - login failure"
  /bin/rm -f "$base_dir/geo_cookies.txt"
  exit
fi
# back to the geolocation container
target_container="https://downloads.f5.com/esd/ecc.sv?$container"
# echo $target_container
mycontainer=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent $target_container 2>&1 )
# send the EULA accept
eula_path="https://downloads.f5.com/esd/eula.sv?$container&path=&file=&B1=I+Accept"
# echo $eula_path
servedownload=$(curl -m 5 --connect-timeout 2 --no-keepalive -kvLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent --ignore-content-length "$eula_path" 2>&1 | grep -e "href\=.*zip\'" | awk -F"[<|>]" '{print $2}' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F"'" '{ print $2 }' )
# echo $servedownload
# get the AWS zip location
target_zip="https://downloads.f5.com/esd/$servedownload"
selected_zip=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent "$target_zip" 2>&1 | grep -e "href.*${downloads_from}" | awk -F'[<|>]' '{ print $6 } ' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F'"' '{ print $2 }' )
# echo $selected_zip
zip_file_name=$( echo $selected_zip | awk -F'[?]' '{ print $1 }' | awk -F'[/]' '{ print $4 }' | awk -FF '{ print $6 }' )
if [[ ! -e $zip_file_name ]]; then
	logger -p local0.notice "Geolocation update file check - downloading update $zip_file_name"
	curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent -o "$base_dir/$zip_file_name" "$selected_zip" 2>&1
	md5servedownload=$(curl -m 5 --connect-timeout 2 --no-keepalive -kvLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent --ignore-content-length "$eula_path" 2>&1 | grep -e "href\=.*zip.md5\'" | awk -F"[<|>]" '{print $2}' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F"'" '{ print $2 }' )
	target_md5="https://downloads.f5.com/esd/$md5servedownload"
	selected_md5=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent "$target_md5" 2>&1 | grep -e "href.*${downloads_from}" | awk -F'[<|>]' '{ print $6 } ' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F'"' '{ print $2 }' )
	md5_file_name=$( echo $selected_md5 | awk -F'[?]' '{ print $1 }' | awk -F'[/]' '{ print $4 }' | awk -FF '{ print $6 }' )
	curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent -o "$base_dir/$md5_file_name" "$selected_md5" 2>&1
	if md5sum --status -c $md5_file_name; then
	  logger -p local0.notice "Geolocation update file check - installing update $zip_file_name"
	  unzip -qq "$base_dir/$zip_file_name" 2>&1 > /dev/null
	  for rpm in *.rpm
	  do
		# echo $rpm
		geoip_update_data -l -f $rpm 2>&1 > /dev/null
		/bin/rm -f $rpm
	  done
	  /bin/rm -f "$base_dir/geo_cookies.txt"
	  /bin/rm -f "$base_dir/README.txt"
	  /bin/rm -f "$base_dir/$zip_file_name"
	  /bin/rm -f "$base_dir/$md5_file_name"
	  for last_zip in "*.zip"
	  do
		rm -f $last_zip
	  done
	  touch "$base_dir/$zip_file_name"
	else
		logger -p local0.err "Geolocation update file check - download failed verification"
		/bin/rm -f "$base_dir/geo_cookies.txt"
	    /bin/rm -f "$base_dir/$zip_file_name"
	    /bin/rm -f "$base_dir/$md5_file_name"
	fi
else
	logger -p local0.notice "Geolocation update file check - latest database currently installed"
    /bin/rm -f "$base_dir/geo_cookies.txt"
fi

Hi, does the script work also on Rel. 15.1.x?

My F5-credentials are fine and I verified connectivity to downloads.f5.com via curl through Forward-Proxy. In  /var/log/ltm I see these messages, but geolocation-files will not be downloaded.

Nov 14 21:18:17 lb-test notice root[8027]: Geolocation update file check - checking for updates
Nov 14 21:18:34 lb-test notice root[8077]: Geolocation update file check - downloading update

----------

Doing some debugging on script shows the following output:

base_dir = /var/tmp/geo
fullversion = 15.1.6.1
baseversion=15
containerversion=15.1.6
container = sw=BIG-IP&pro=big-ip_v15.x&ver=15.1.6&container=GeoLocationUpdates
downloads_user = username
downloads_from=USA - WEST COAST
proxy_opts = --proxy http://10.14.38.3:3128/
loginpage=https://api-u.f5.com/auth/pub/sso/login/user
afterlogin= F5 - My Account
target_container=https://downloads.f5.com/esd/ecc.sv?sw=BIG-IP&pro=big-ip_v15.x&ver=15.1.6&container=GeoLocationUpdates
eula_path=https://downloads.f5.com/esd/eula.sv?sw=BIG-IP&pro=big-ip_v15.x&ver=15.1.6&container=GeoLocationUpdates&path=&file=&B1=I+ Accept
servedownload=
target_zip=https://downloads.f5.com/esd/
selected_zip=
zip_file_name=
md5servedownload=
target_md5=https://downloads.f5.com/esd/
selected_md5=
md5_file_name=

----------

Many Thanks!

Josef