Geolocation data auto-update

[Sean_29304]

Hello,

Is there a way to automate Geolocation data updates?

Thank you,

Sean

[hooleylist]

Hi Sean, Currently there isn't an option. You could potentially script this off of LTM by downloading the latest GeoIP update file and install it. The manual process is described here: sol11176: Downloading and installing updates to the IP geolocation database https://support.f5.com/kb/en-us/solutions/public/11000/100/sol11176.html I also suggest opening a case with F5 Support to request this type of functionality be added to LTM. It might be useful to also request an iControl hook for doing this. Aaron

[Sean_29304]

Hi Aaron, I'm very new to F5, therefore dumb questions. Could you please point me to a tutorial for scripting that kind of action? Thank you. Sean

[hooleylist]

I would probably write the script to run on an external host that has internet access and access to the BIG-IP(s). You could use whatever scripting or programming language you're comfortable with to download the file from downloads.f5.com. You could then have your script copy the file to the BIG-IP using scp, ssh to the BIG-IP command line unpack the file and run the installation. There isn't anything specific to BIG-IP in this process. Or you could just set a calendar reminder to do this manually once a month when a new GeoIP file is published 🙂 Aaron

[Sean_29304]

Thank you, I'll see what I can do about this. Manual is not an option (imagine 10 other products that would have the same solution). Would be great if the appliance could auto update 🙂 Thanks again, Sean

[David_Burley_20]

I wrote this last night, and it works for us -- feel free to use it -- though I recommend you test it in your environment first.

!/usr/bin/perl -w
 Download the latest GeoIP files for F5 LTM and install them. This script runs
 from a supporting host.

 Requires:
  * perl (WWW::Mechanize and its dependencies)
  * A ssh suite with passwordless authentication via key to the load balancers
  * unzip
  * md5sum

 This script will be fragile, since it relies upon minimal changes to the F5
 website. It should be reasonably safe, given it checks the md5sum of the
 downloaded file, but use at your own risk.

 Configure the $user and $pass variables for your login to the F5 site.
 Configure the @servers list for the servers to SSH to and update the DBs on

use WWW::Mechanize;

 Configuration section - Update these

my $user = 'F5_LOGIN_USERNAME';
my $pass = 'F5_PASSWORD';
my $uri = 'https://downloads.f5.com/esd/ecc.sv?sw=BIG-IP&pro=big-ip_v11.x&ver=11.6.0&container=GeoLocationUpdates';
my @servers = qw(SERVER1 SERVER2);
my $dir = '/tmp/f5geoip';

 End configuration - changes shouldn't be necessary below unless the F5 site changes.

die "\$dir needs set to something that doesn't exist\n" if($dir eq '' || $dir eq '/' || -e $dir);  Sanity check....
mkdir($dir);

 Login to F5, download the GeoIP update and md5sum
my $mech = WWW::Mechanize->new( cookie_jar => {} );
$mech->get($uri);
$mech->submit_form(
  form_name => 'login',
  fields => {
    userid => $user,
    passwd => $pass
  }
);

$mech->submit_form(
  form_name => 'LicenseAgreement'
);

$mech->follow_link(url_regex => qr/\.zip$/);
$mech->follow_link(url_regex => qr/\.zip/);
$mech->save_content("$dir/f5geoip.zip");
$mech->get($uri);
$mech->follow_link(url_regex => qr/\.zip\.md5$/);
$mech->follow_link(url_regex => qr/\.zip\.md5/);
$mech->save_content("$dir/f5geoip.zip.md5");

 Validate the file contents against md5sum and extract
my $sum = `cat $dir/f5geoip.zip.md5 | cut -d ' ' -f 1`;
my $filesum = `md5sum $dir/f5geoip.zip | cut -d ' ' -f 1`;
chomp $sum;
chomp $filesum;
die "Invalid checksum on GeoIP Database\n" if ($sum ne $filesum);

my $ret = system("unzip $dir/f5geoip.zip -d $dir >/dev/null") >> 8;
die "zip didn't properly extract\n" if($ret != 0);

 Generate a list of the RPMs
my @files = split("\n", `ls -1 $dir | grep rpm`);

 Copy the files to the LB via SCP and then run the updater
foreach my $server (@servers) {
  foreach my $file (@files) {
    system("scp $dir/$file $server:/shared/tmp > /dev/null");
    system("ssh $server 'geoip_update_data -f /shared/tmp/$file; rm -f /shared/tmp/$file'");
  }
}

 Cleanup
system("rm -f $dir/*; rmdir $dir");

[johnnypizzy]

Thanks for the script David. Works great!

[prt1969]

Has anything changed here where F5 supports auto-updates for this? If attack signature updates can be automated for ASM, doing the same for geo-location on GTM shouldn't be a stretch.

[Nova_201357]

This is a problem for me too. If the bigIP cannot update the database automatically, is there a list we can subscribe to so that we know when there is a new version? That alone would help a bunch!

Cheers, Mike

[Chris_Gray_2395]

The above Perl script didn't work for me, and since I know PHP better, I made a CLI script to do the job. Its been tested on 12.1.2 and it works great!!

It takes 1 param, which is the https file link. An optional 2nd param is for the download link for the MD5 hash file. without this, it doesn't do the check.

Here is a link to the script on my GitHub. I know you rather have the code here, but this way i can update it later if problems are found.

Github - C2theG - GeoIP_Updater

[Patrik_Jonsson]

Thanks for sharing Chris! Add a link in the code share too?

[Chris_Gray_2395]

Thanks, Done! https://devcentral.f5.com/s/articles/auto-updates-geoip-database-on-big-ip-1061

 

[Linux_Tech]

Here is bash script that automatically downloads and installs the latest Geolocation database:

F5 Geolocation Database update script

This script will sign into downloads.f5.com (using existing credentials), and download the latest appropriate geolocation database for the BIG-IP release and install the update. This should work on all current supported BIG-IP releases.

Place the script in /etc/cron.weekly or /etc/cron.daily and ensure that it is executable. It stores the name of the last update installed in /var/tmp/geo and only downloads if a new file is available.

If you need to use a proxy to get to the download site, you can set a proxy options variable.

As the credentials used to access downloads.f5.com are stored in the script, I recommend creating a new set of credentials for this purpose, and only using those credentials for this purpose.

Set the variables

downloads_user=""

downloads_password=""

downloads_from="USA - WEST COAST"

proxy_opts=""

The script logs into /var/log/ltm when it runs, downloads and installs the database. It also logs errors, such as login failure and database backup errors.

If the F5 Downloads site changes or restructures, the script will probably fail.

#! /bin/bash
# f5 geolocation updater
# linuxtech@mail.com
# free for use 
logger -p local0.notice "Geolocation update file check - checking for updates"
base_dir="/var/tmp/geo"
if [[ ! -e $base_dir ]]; then
    mkdir -p $base_dir
fi
cd $base_dir
if [[ -e "$base_dir/geo_cookies.txt" ]]; then
  /bin/rm -f "$base_dir/geo_cookies.txt"
fi
fullversion=$(tmsh show sys version | grep " Version" | awk '{ print $2 }')
# echo $fullversion
baseversion=$(echo $fullversion | awk -F. '{ print $1 }')
# echo $baseversion
containerversion=$(echo $fullversion | awk -F. '{ print $1"."$2"."$3 }')
# backup the current Geolocation database
dir="/shared/GeoIP_backup"
if [[ ! -e $dir ]]; then
    mkdir -p $dir
elif [[ ! -d $dir ]]; then
	logger -p local0.err "Geolocation update file check - error backing up Geolocation database: $dir already exists but is not a directory"
	exit
fi
if [ $baseversion -ge 15 ]; then
  /bin/cp -R /shared/GeoIP/* /shared/GeoIP_backup/
else
  /bin/cp -R /usr/share/GeoIP/* /shared/GeoIP_backup/
fi 
# echo $containerversion
container="sw=BIG-IP&pro=big-ip_v$baseversion.x&ver=$containerversion&container=GeoLocationUpdates"
# echo $container
downloads_user=""
downloads_password=""
downloads_from="USA - WEST COAST"
# specify any curl proxy options as required
# eg --proxy http://user:password@host:port/
# or
# "" for direct connect
proxy_opts=""
# get the login page
loginpage=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent https://downloads.f5.com/esd/ecc.sv?$container 2>&1 | grep "action=" | awk -F'[=\"|\">]' '{ print $3 }' )
# echo $loginpage
# submit the credentials
afterlogin=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent $loginpage -X POST --data-urlencode "userid=$downloads_user" --data-urlencode "passwd=$downloads_password" 2>&1 | grep "F5 Networks - My Account" | awk -F'[="|">]' '{ print $6 }' )
# echo $afterlogin
if [[ $afterlogin == "" ]]; then
  logger -p local0.err "Geolocation update file check - login failure"
  /bin/rm -f "$base_dir/geo_cookies.txt"
  exit
fi
# back to the geolocation container
target_container="https://downloads.f5.com/esd/ecc.sv?$container"
# echo $target_container
mycontainer=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent $target_container 2>&1 )
# send the EULA accept
eula_path="https://downloads.f5.com/esd/eula.sv?$container&path=&file=&B1=I+Accept"
# echo $eula_path
servedownload=$(curl -m 5 --connect-timeout 2 --no-keepalive -kvLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent --ignore-content-length "$eula_path" 2>&1 | grep -e "href\=.*zip\'" | awk -F"[<|>]" '{print $2}' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F"'" '{ print $2 }' )
# echo $servedownload
# get the AWS zip location
target_zip="https://downloads.f5.com/esd/$servedownload"
selected_zip=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent "$target_zip" 2>&1 | grep -e "href.*${downloads_from}" | awk -F'[<|>]' '{ print $6 } ' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F'"' '{ print $2 }' )
# echo $selected_zip
zip_file_name=$( echo $selected_zip | awk -F'[?]' '{ print $1 }' | awk -F'[/]' '{ print $4 }' | awk -FF '{ print $6 }' )
if [[ ! -e $zip_file_name ]]; then
	logger -p local0.notice "Geolocation update file check - downloading update $zip_file_name"
	curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent -o "$base_dir/$zip_file_name" "$selected_zip" 2>&1
	md5servedownload=$(curl -m 5 --connect-timeout 2 --no-keepalive -kvLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent --ignore-content-length "$eula_path" 2>&1 | grep -e "href\=.*zip.md5\'" | awk -F"[<|>]" '{print $2}' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F"'" '{ print $2 }' )
	target_md5="https://downloads.f5.com/esd/$md5servedownload"
	selected_md5=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent "$target_md5" 2>&1 | grep -e "href.*${downloads_from}" | awk -F'[<|>]' '{ print $6 } ' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F'"' '{ print $2 }' )
	md5_file_name=$( echo $selected_md5 | awk -F'[?]' '{ print $1 }' | awk -F'[/]' '{ print $4 }' | awk -FF '{ print $6 }' )
	curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent -o "$base_dir/$md5_file_name" "$selected_md5" 2>&1
	if md5sum --status -c $md5_file_name; then
	  logger -p local0.notice "Geolocation update file check - installing update $zip_file_name"
	  unzip -qq "$base_dir/$zip_file_name" 2>&1 > /dev/null
	  for rpm in *.rpm
	  do
		# echo $rpm
		geoip_update_data -l -f $rpm 2>&1 > /dev/null
		/bin/rm -f $rpm
	  done
	  /bin/rm -f "$base_dir/geo_cookies.txt"
	  /bin/rm -f "$base_dir/README.txt"
	  /bin/rm -f "$base_dir/$zip_file_name"
	  /bin/rm -f "$base_dir/$md5_file_name"
	  for last_zip in "*.zip"
	  do
		rm -f $last_zip
	  done
	  touch "$base_dir/$zip_file_name"
	else
		logger -p local0.err "Geolocation update file check - download failed verification"
		/bin/rm -f "$base_dir/geo_cookies.txt"
	    /bin/rm -f "$base_dir/$zip_file_name"
	    /bin/rm -f "$base_dir/$md5_file_name"
	fi
else
	logger -p local0.notice "Geolocation update file check - latest database currently installed"
    /bin/rm -f "$base_dir/geo_cookies.txt"
fi

[Curious1]

Hi, does this script work on v14.x?

Just tried it and I get a login failure yet my creds are ok and verified connectivity to downloads.f5.com from the BigIP with curl. Result is a 302 to the login page.

Nov 2 16:17:44 host notice : Geolocation update file check - checking for updates

Nov 2 16:17:48 host err : Geolocation update file check - login failure (my creds are ok though, confirmed with a manual download)

Thanks

[Linux_Tech]

There seem to have been some download website changes that prevented the script from working.

I have made some changes to the script which I tested on 14.1.

Give it a go:

#! /bin/bash
# f5 geolocation updater
# linuxtech@mail.com
# free for use 
logger -p local0.notice "Geolocation update file check - checking for updates"
base_dir="/var/tmp/geo"
if [[ ! -e $base_dir ]]; then
    mkdir -p $base_dir
fi
cd $base_dir
if [[ -e "$base_dir/geo_cookies.txt" ]]; then
  /bin/rm -f "$base_dir/geo_cookies.txt"
fi
fullversion=$(tmsh show sys version | grep " Version" | awk '{ print $2 }')
# echo $fullversion
baseversion=$(echo $fullversion | awk -F. '{ print $1 }')
# echo $baseversion
containerversion=$(echo $fullversion | awk -F. '{ print $1"."$2"."$3 }')
# backup the current Geolocation database
dir="/shared/GeoIP_backup"
if [[ ! -e $dir ]]; then
    mkdir -p $dir
elif [[ ! -d $dir ]]; then
	logger -p local0.err "Geolocation update file check - error backing up Geolocation database: $dir already exists but is not a directory"
	exit
fi
if [ $baseversion -ge 15 ]; then
  /bin/cp -R /shared/GeoIP/* /shared/GeoIP_backup/
else
  /bin/cp -R /usr/share/GeoIP/* /shared/GeoIP_backup/
fi 
# echo $containerversion
container="sw=BIG-IP&pro=big-ip_v$baseversion.x&ver=$containerversion&container=GeoLocationUpdates"
# echo $container
downloads_user=""
downloads_password=""
downloads_from="USA - WEST COAST"
# specify any curl proxy options as required
# eg --proxy http://user:password@host:port/
# or
# "" for direct connect
proxy_opts=""
# get the login page
loginpage=$(curl -m 10 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent https://downloads.f5.com/esd/ecc.sv?$container 2>&1 | grep "action=" | awk -F'[=\"|\">]' '{ print $3 }' )
# echo $loginpage
# submit the credentials
afterlogin=$(curl -m 10 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent $loginpage -X POST --data-urlencode "userid=$downloads_user" --data-urlencode "passwd=$downloads_password" 2>&1 | grep "F5 - My Account" | awk -F'[="|">]' '{ print $6 }' )
# echo $afterlogin
if [[ $afterlogin == "" ]]; then
  logger -p local0.err "Geolocation update file check - login failure"
  /bin/rm -f "$base_dir/geo_cookies.txt"
  exit
fi
# back to the geolocation container
target_container="https://downloads.f5.com/esd/ecc.sv?$container"
# echo $target_container
mycontainer=$(curl -m 10 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent $target_container 2>&1 )
# send the EULA accept
eula_path="https://downloads.f5.com/esd/eula.sv?$container&path=&file=&B1=I+Accept"
# echo $eula_path
servedownload=$(curl -m 10 --connect-timeout 2 --no-keepalive -kvLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent --ignore-content-length "$eula_path" 2>&1 | grep -e "href\=.*zip\'" | awk -F"[<|>]" '{print $2}' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F"'" '{ print $2 }' )
# echo $servedownload
# get the AWS zip location
target_zip="https://downloads.f5.com/esd/$servedownload"
# echo $target_zip
selected_zip=$(curl -m 10 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent "$target_zip" 2>&1 | grep -e "href.*${downloads_from}" | awk -F'[<|>]' '{ print $6 } ' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F'"' '{ print $2 }' )
# echo $selected_zip
zip_file_name=$( echo $selected_zip | awk -F'[?]' '{ print $1 }' | awk -F'[/]' '{ print $NF }' )
# echo $zip_file_name
if [[ ! -e $zip_file_name ]]; then
	logger -p local0.notice "Geolocation update file check - downloading update $zip_file_name"
	curl -m 30 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent -o "$base_dir/$zip_file_name" "$selected_zip" 2>&1
	md5servedownload=$(curl -m 5 --connect-timeout 2 --no-keepalive -kvLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent --ignore-content-length "$eula_path" 2>&1 | grep -e "href\=.*zip.md5\'" | awk -F"[<|>]" '{print $2}' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F"'" '{ print $2 }' )
	target_md5="https://downloads.f5.com/esd/$md5servedownload"
	selected_md5=$(curl -m 5 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent "$target_md5" 2>&1 | grep -e "href.*${downloads_from}" | awk -F'[<|>]' '{ print $6 } ' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F'"' '{ print $2 }' )
	md5_file_name=$( echo $selected_md5 | awk -F'[?]' '{ print $1 }' | awk -F'[/]' '{ print $NF }' )
	# echo $md5_file_name
	curl -m 30 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent -o "$base_dir/$md5_file_name" "$selected_md5" 2>&1
	if md5sum --status -c $md5_file_name; then
	  logger -p local0.notice "Geolocation update file check - installing update $zip_file_name"
	  unzip -qq "$base_dir/$zip_file_name" 2>&1 > /dev/null
	  for rpm in *.rpm
	  do
		# echo $rpm
		geoip_update_data -l -f $rpm 2>&1 > /dev/null
		/bin/rm -f $rpm
	  done
	  /bin/rm -f "$base_dir/geo_cookies.txt"
	  /bin/rm -f "$base_dir/README.txt"
	  /bin/rm -f "$base_dir/$zip_file_name"
	  /bin/rm -f "$base_dir/$md5_file_name"
	  for last_zip in "*.zip"
	  do
		rm -f $last_zip
	  done
	  touch "$base_dir/$zip_file_name"
	else
		logger -p local0.err "Geolocation update file check - download failed verification"
		/bin/rm -f "$base_dir/geo_cookies.txt"
	    /bin/rm -f "$base_dir/$zip_file_name"
	    /bin/rm -f "$base_dir/$md5_file_name"
	fi
else
	logger -p local0.notice "Geolocation update file check - latest database currently installed"
    /bin/rm -f "$base_dir/geo_cookies.txt"
fi

[kridsana]

Code isn't working. Maybe due to downloads.f5 change to myf5 right now

[braeutjo]

Hi, does the script work also on Rel. 15.1.x?

My F5-credentials are fine and I verified connectivity to downloads.f5.com via curl through Forward-Proxy. In  /var/log/ltm I see these messages, but geolocation-files will not be downloaded.

Nov 14 21:18:17 lb-test notice root[8027]: Geolocation update file check - checking for updates
Nov 14 21:18:34 lb-test notice root[8077]: Geolocation update file check - downloading update

----------

Doing some debugging on script shows the following output:

base_dir = /var/tmp/geo
fullversion = 15.1.6.1
baseversion=15
containerversion=15.1.6
container = sw=BIG-IP&pro=big-ip_v15.x&ver=15.1.6&container=GeoLocationUpdates
downloads_user = username
downloads_from=USA - WEST COAST
proxy_opts = --proxy http://10.14.38.3:3128/
loginpage=https://api-u.f5.com/auth/pub/sso/login/user
afterlogin= F5 - My Account
target_container=https://downloads.f5.com/esd/ecc.sv?sw=BIG-IP&pro=big-ip_v15.x&ver=15.1.6&container=GeoLocationUpdat...
eula_path=https://downloads.f5.com/esd/eula.sv?sw=BIG-IP&pro=big-ip_v15.x&ver=15.1.6&container=GeoLocationUpda... Accept
servedownload=
target_zip=https://downloads.f5.com/esd/
selected_zip=
zip_file_name=
md5servedownload=
target_md5=https://downloads.f5.com/esd/
selected_md5=
md5_file_name=

----------

Many Thanks!

Josef

[kridsana]

Code isn't working. Maybe due to downloads.f5 change to myf5 right now