Forum Discussion
Geolocation data auto-update
Is there a way to automate Geolocation data updates?
Thank you,
Sean
- hooleylistCirrostratusHi Sean,
- Sean_29304NimbostratusHi Aaron,
- hooleylistCirrostratusI would probably write the script to run on an external host that has internet access and access to the BIG-IP(s). You could use whatever scripting or programming language you're comfortable with to download the file from downloads.f5.com. You could then have your script copy the file to the BIG-IP using scp, ssh to the BIG-IP command line unpack the file and run the installation. There isn't anything specific to BIG-IP in this process.
- Sean_29304NimbostratusThank you, I'll see what I can do about this. Manual is not an option (imagine 10 other products that would have the same solution).
- David_Burley_20Nimbostratus
I wrote this last night, and it works for us -- feel free to use it -- though I recommend you test it in your environment first.
!/usr/bin/perl -w Download the latest GeoIP files for F5 LTM and install them. This script runs from a supporting host. Requires: * perl (WWW::Mechanize and its dependencies) * A ssh suite with passwordless authentication via key to the load balancers * unzip * md5sum This script will be fragile, since it relies upon minimal changes to the F5 website. It should be reasonably safe, given it checks the md5sum of the downloaded file, but use at your own risk. Configure the $user and $pass variables for your login to the F5 site. Configure the @servers list for the servers to SSH to and update the DBs on use WWW::Mechanize; Configuration section - Update these my $user = 'F5_LOGIN_USERNAME'; my $pass = 'F5_PASSWORD'; my $uri = 'https://downloads.f5.com/esd/ecc.sv?sw=BIG-IP&pro=big-ip_v11.x&ver=11.6.0&container=GeoLocationUpdates'; my @servers = qw(SERVER1 SERVER2); my $dir = '/tmp/f5geoip'; End configuration - changes shouldn't be necessary below unless the F5 site changes. die "\$dir needs set to something that doesn't exist\n" if($dir eq '' || $dir eq '/' || -e $dir); Sanity check.... mkdir($dir); Login to F5, download the GeoIP update and md5sum my $mech = WWW::Mechanize->new( cookie_jar => {} ); $mech->get($uri); $mech->submit_form( form_name => 'login', fields => { userid => $user, passwd => $pass } ); $mech->submit_form( form_name => 'LicenseAgreement' ); $mech->follow_link(url_regex => qr/\.zip$/); $mech->follow_link(url_regex => qr/\.zip/); $mech->save_content("$dir/f5geoip.zip"); $mech->get($uri); $mech->follow_link(url_regex => qr/\.zip\.md5$/); $mech->follow_link(url_regex => qr/\.zip\.md5/); $mech->save_content("$dir/f5geoip.zip.md5"); Validate the file contents against md5sum and extract my $sum = `cat $dir/f5geoip.zip.md5 | cut -d ' ' -f 1`; my $filesum = `md5sum $dir/f5geoip.zip | cut -d ' ' -f 1`; chomp $sum; chomp $filesum; die "Invalid checksum on GeoIP Database\n" if ($sum ne $filesum); my $ret = system("unzip $dir/f5geoip.zip -d $dir >/dev/null") >> 8; die "zip didn't properly extract\n" if($ret != 0); Generate a list of the RPMs my @files = split("\n", `ls -1 $dir | grep rpm`); Copy the files to the LB via SCP and then run the updater foreach my $server (@servers) { foreach my $file (@files) { system("scp $dir/$file $server:/shared/tmp > /dev/null"); system("ssh $server 'geoip_update_data -f /shared/tmp/$file; rm -f /shared/tmp/$file'"); } } Cleanup system("rm -f $dir/*; rmdir $dir");
- johnnypizzyNimbostratusThanks for the script David. Works great!
- prt1969Nimbostratus
Has anything changed here where F5 supports auto-updates for this? If attack signature updates can be automated for ASM, doing the same for geo-location on GTM shouldn't be a stretch.
- Nova_201357Nimbostratus
This is a problem for me too. If the bigIP cannot update the database automatically, is there a list we can subscribe to so that we know when there is a new version? That alone would help a bunch!
Cheers, Mike
- Chris_Gray_2395Nimbostratus
The above Perl script didn't work for me, and since I know PHP better, I made a CLI script to do the job. Its been tested on 12.1.2 and it works great!!
It takes 1 param, which is the https file link. An optional 2nd param is for the download link for the MD5 hash file. without this, it doesn't do the check.
Here is a link to the script on my GitHub. I know you rather have the code here, but this way i can update it later if problems are found.
Github - C2theG - GeoIP_Updater
Thanks for sharing Chris! Add a link in the code share too?
- Chris_Gray_2395Nimbostratus
- Linux_TechNimbostratus
Here is bash script that automatically downloads and installs the latest Geolocation database:
F5 Geolocation Database update script
This script will sign into downloads.f5.com (using existing credentials), and download the latest appropriate geolocation database for the BIG-IP release and install the update. This should work on all current supported BIG-IP releases.
Place the script in /etc/cron.weekly or /etc/cron.daily and ensure that it is executable. It stores the name of the last update installed in /var/tmp/geo and only downloads if a new file is available.
If you need to use a proxy to get to the download site, you can set a proxy options variable.
As the credentials used to access downloads.f5.com are stored in the script, I recommend creating a new set of credentials for this purpose, and only using those credentials for this purpose.
Set the variables
downloads_user=""
downloads_password=""
downloads_from="USA - WEST COAST"
proxy_opts=""
The script logs into /var/log/ltm when it runs, downloads and installs the database. It also logs errors, such as login failure and database backup errors.
If the F5 Downloads site changes or restructures, the script will probably fail.
#! /bin/bash # f5 geolocation updater # linuxtech@mail.com # free for use logger -p local0.notice "Geolocation update file check - checking for updates" base_dir="/var/tmp/geo" if [[ ! -e $base_dir ]]; then mkdir -p $base_dir fi cd $base_dir if [[ -e "$base_dir/geo_cookies.txt" ]]; then /bin/rm -f "$base_dir/geo_cookies.txt" fi fullversion=$(tmsh show sys version | grep " Version" | awk '{ print $2 }') # echo $fullversion baseversion=$(echo $fullversion | awk -F. '{ print $1 }') # echo $baseversion containerversion=$(echo $fullversion | awk -F. '{ print $1"."$2"."$3 }') # backup the current Geolocation database dir="/shared/GeoIP_backup" if [[ ! -e $dir ]]; then mkdir -p $dir elif [[ ! -d $dir ]]; then logger -p local0.err "Geolocation update file check - error backing up Geolocation database: $dir already exists but is not a directory" exit fi if [ $baseversion -ge 15 ]; then /bin/cp -R /shared/GeoIP/* /shared/GeoIP_backup/ else /bin/cp -R /usr/share/GeoIP/* /shared/GeoIP_backup/ fi # echo $containerversion container="sw=BIG-IP&pro=big-ip_v$baseversion.x&ver=$containerversion&container=GeoLocationUpdates" # echo $container downloads_user="" downloads_password="" downloads_from="USA - WEST COAST" # specify any curl proxy options as required # eg --proxy http://user:password@host:port/ # or # "" for direct connect proxy_opts="" # get the login page loginpage=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent https://downloads.f5.com/esd/ecc.sv?$container 2>&1 | grep "action=" | awk -F'[=\"|\">]' '{ print $3 }' ) # echo $loginpage # submit the credentials afterlogin=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent $loginpage -X POST --data-urlencode "userid=$downloads_user" --data-urlencode "passwd=$downloads_password" 2>&1 | grep "F5 Networks - My Account" | awk -F'[="|">]' '{ print $6 }' ) # echo $afterlogin if [[ $afterlogin == "" ]]; then logger -p local0.err "Geolocation update file check - login failure" /bin/rm -f "$base_dir/geo_cookies.txt" exit fi # back to the geolocation container target_container="https://downloads.f5.com/esd/ecc.sv?$container" # echo $target_container mycontainer=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent $target_container 2>&1 ) # send the EULA accept eula_path="https://downloads.f5.com/esd/eula.sv?$container&path=&file=&B1=I+Accept" # echo $eula_path servedownload=$(curl -m 5 --connect-timeout 2 --no-keepalive -kvLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent --ignore-content-length "$eula_path" 2>&1 | grep -e "href\=.*zip\'" | awk -F"[<|>]" '{print $2}' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F"'" '{ print $2 }' ) # echo $servedownload # get the AWS zip location target_zip="https://downloads.f5.com/esd/$servedownload" selected_zip=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent "$target_zip" 2>&1 | grep -e "href.*${downloads_from}" | awk -F'[<|>]' '{ print $6 } ' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F'"' '{ print $2 }' ) # echo $selected_zip zip_file_name=$( echo $selected_zip | awk -F'[?]' '{ print $1 }' | awk -F'[/]' '{ print $4 }' | awk -FF '{ print $6 }' ) if [[ ! -e $zip_file_name ]]; then logger -p local0.notice "Geolocation update file check - downloading update $zip_file_name" curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent -o "$base_dir/$zip_file_name" "$selected_zip" 2>&1 md5servedownload=$(curl -m 5 --connect-timeout 2 --no-keepalive -kvLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent --ignore-content-length "$eula_path" 2>&1 | grep -e "href\=.*zip.md5\'" | awk -F"[<|>]" '{print $2}' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F"'" '{ print $2 }' ) target_md5="https://downloads.f5.com/esd/$md5servedownload" selected_md5=$(curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent "$target_md5" 2>&1 | grep -e "href.*${downloads_from}" | awk -F'[<|>]' '{ print $6 } ' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F'"' '{ print $2 }' ) md5_file_name=$( echo $selected_md5 | awk -F'[?]' '{ print $1 }' | awk -F'[/]' '{ print $4 }' | awk -FF '{ print $6 }' ) curl -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent -o "$base_dir/$md5_file_name" "$selected_md5" 2>&1 if md5sum --status -c $md5_file_name; then logger -p local0.notice "Geolocation update file check - installing update $zip_file_name" unzip -qq "$base_dir/$zip_file_name" 2>&1 > /dev/null for rpm in *.rpm do # echo $rpm geoip_update_data -l -f $rpm 2>&1 > /dev/null /bin/rm -f $rpm done /bin/rm -f "$base_dir/geo_cookies.txt" /bin/rm -f "$base_dir/README.txt" /bin/rm -f "$base_dir/$zip_file_name" /bin/rm -f "$base_dir/$md5_file_name" for last_zip in "*.zip" do rm -f $last_zip done touch "$base_dir/$zip_file_name" else logger -p local0.err "Geolocation update file check - download failed verification" /bin/rm -f "$base_dir/geo_cookies.txt" /bin/rm -f "$base_dir/$zip_file_name" /bin/rm -f "$base_dir/$md5_file_name" fi else logger -p local0.notice "Geolocation update file check - latest database currently installed" /bin/rm -f "$base_dir/geo_cookies.txt" fi
- Curious1Cirrus
Hi, does this script work on v14.x?
Just tried it and I get a login failure yet my creds are ok and verified connectivity to downloads.f5.com from the BigIP with curl. Result is a 302 to the login page.
Nov 2 16:17:44 host notice : Geolocation update file check - checking for updates
Nov 2 16:17:48 host err : Geolocation update file check - login failure (my creds are ok though, confirmed with a manual download)
Thanks
- Linux_TechNimbostratus
There seem to have been some download website changes that prevented the script from working.
I have made some changes to the script which I tested on 14.1.
Give it a go:
#! /bin/bash # f5 geolocation updater # linuxtech@mail.com # free for use logger -p local0.notice "Geolocation update file check - checking for updates" base_dir="/var/tmp/geo" if [[ ! -e $base_dir ]]; then mkdir -p $base_dir fi cd $base_dir if [[ -e "$base_dir/geo_cookies.txt" ]]; then /bin/rm -f "$base_dir/geo_cookies.txt" fi fullversion=$(tmsh show sys version | grep " Version" | awk '{ print $2 }') # echo $fullversion baseversion=$(echo $fullversion | awk -F. '{ print $1 }') # echo $baseversion containerversion=$(echo $fullversion | awk -F. '{ print $1"."$2"."$3 }') # backup the current Geolocation database dir="/shared/GeoIP_backup" if [[ ! -e $dir ]]; then mkdir -p $dir elif [[ ! -d $dir ]]; then logger -p local0.err "Geolocation update file check - error backing up Geolocation database: $dir already exists but is not a directory" exit fi if [ $baseversion -ge 15 ]; then /bin/cp -R /shared/GeoIP/* /shared/GeoIP_backup/ else /bin/cp -R /usr/share/GeoIP/* /shared/GeoIP_backup/ fi # echo $containerversion container="sw=BIG-IP&pro=big-ip_v$baseversion.x&ver=$containerversion&container=GeoLocationUpdates" # echo $container downloads_user="" downloads_password="" downloads_from="USA - WEST COAST" # specify any curl proxy options as required # eg --proxy http://user:password@host:port/ # or # "" for direct connect proxy_opts="" # get the login page loginpage=$(curl -m 10 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent https://downloads.f5.com/esd/ecc.sv?$container 2>&1 | grep "action=" | awk -F'[=\"|\">]' '{ print $3 }' ) # echo $loginpage # submit the credentials afterlogin=$(curl -m 10 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent $loginpage -X POST --data-urlencode "userid=$downloads_user" --data-urlencode "passwd=$downloads_password" 2>&1 | grep "F5 - My Account" | awk -F'[="|">]' '{ print $6 }' ) # echo $afterlogin if [[ $afterlogin == "" ]]; then logger -p local0.err "Geolocation update file check - login failure" /bin/rm -f "$base_dir/geo_cookies.txt" exit fi # back to the geolocation container target_container="https://downloads.f5.com/esd/ecc.sv?$container" # echo $target_container mycontainer=$(curl -m 10 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent $target_container 2>&1 ) # send the EULA accept eula_path="https://downloads.f5.com/esd/eula.sv?$container&path=&file=&B1=I+Accept" # echo $eula_path servedownload=$(curl -m 10 --connect-timeout 2 --no-keepalive -kvLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent --ignore-content-length "$eula_path" 2>&1 | grep -e "href\=.*zip\'" | awk -F"[<|>]" '{print $2}' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F"'" '{ print $2 }' ) # echo $servedownload # get the AWS zip location target_zip="https://downloads.f5.com/esd/$servedownload" # echo $target_zip selected_zip=$(curl -m 10 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent "$target_zip" 2>&1 | grep -e "href.*${downloads_from}" | awk -F'[<|>]' '{ print $6 } ' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F'"' '{ print $2 }' ) # echo $selected_zip zip_file_name=$( echo $selected_zip | awk -F'[?]' '{ print $1 }' | awk -F'[/]' '{ print $NF }' ) # echo $zip_file_name if [[ ! -e $zip_file_name ]]; then logger -p local0.notice "Geolocation update file check - downloading update $zip_file_name" curl -m 30 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent -o "$base_dir/$zip_file_name" "$selected_zip" 2>&1 md5servedownload=$(curl -m 5 --connect-timeout 2 --no-keepalive -kvLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent --ignore-content-length "$eula_path" 2>&1 | grep -e "href\=.*zip.md5\'" | awk -F"[<|>]" '{print $2}' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F"'" '{ print $2 }' ) target_md5="https://downloads.f5.com/esd/$md5servedownload" selected_md5=$(curl -m 5 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent "$target_md5" 2>&1 | grep -e "href.*${downloads_from}" | awk -F'[<|>]' '{ print $6 } ' | awk -F'=' '{ st = index($0,"="); print substr($0,st+1) }' | awk -F'"' '{ print $2 }' ) md5_file_name=$( echo $selected_md5 | awk -F'[?]' '{ print $1 }' | awk -F'[/]' '{ print $NF }' ) # echo $md5_file_name curl -m 30 --connect-timeout 2 --no-keepalive -kLb $base_dir/geo_cookies.txt -c $base_dir/geo_cookies.txt $proxy_opts --silent -o "$base_dir/$md5_file_name" "$selected_md5" 2>&1 if md5sum --status -c $md5_file_name; then logger -p local0.notice "Geolocation update file check - installing update $zip_file_name" unzip -qq "$base_dir/$zip_file_name" 2>&1 > /dev/null for rpm in *.rpm do # echo $rpm geoip_update_data -l -f $rpm 2>&1 > /dev/null /bin/rm -f $rpm done /bin/rm -f "$base_dir/geo_cookies.txt" /bin/rm -f "$base_dir/README.txt" /bin/rm -f "$base_dir/$zip_file_name" /bin/rm -f "$base_dir/$md5_file_name" for last_zip in "*.zip" do rm -f $last_zip done touch "$base_dir/$zip_file_name" else logger -p local0.err "Geolocation update file check - download failed verification" /bin/rm -f "$base_dir/geo_cookies.txt" /bin/rm -f "$base_dir/$zip_file_name" /bin/rm -f "$base_dir/$md5_file_name" fi else logger -p local0.notice "Geolocation update file check - latest database currently installed" /bin/rm -f "$base_dir/geo_cookies.txt" fi
- braeutjoNimbostratus
Hi, does the script work also on Rel. 15.1.x?
My F5-credentials are fine and I verified connectivity to downloads.f5.com via curl through Forward-Proxy. In /var/log/ltm I see these messages, but geolocation-files will not be downloaded.
Nov 14 21:18:17 lb-test notice root[8027]: Geolocation update file check - checking for updates
Nov 14 21:18:34 lb-test notice root[8077]: Geolocation update file check - downloading update----------
Doing some debugging on script shows the following output:
base_dir = /var/tmp/geo
fullversion = 15.1.6.1
baseversion=15
containerversion=15.1.6
container = sw=BIG-IP&pro=big-ip_v15.x&ver=15.1.6&container=GeoLocationUpdates
downloads_user = username
downloads_from=USA - WEST COAST
proxy_opts = --proxy http://10.14.38.3:3128/
loginpage=https://api-u.f5.com/auth/pub/sso/login/user
afterlogin= F5 - My Account
target_container=https://downloads.f5.com/esd/ecc.sv?sw=BIG-IP&pro=big-ip_v15.x&ver=15.1.6&container=GeoLocationUpdates
eula_path=https://downloads.f5.com/esd/eula.sv?sw=BIG-IP&pro=big-ip_v15.x&ver=15.1.6&container=GeoLocationUpdates&path=&file=&B1=I+ Accept
servedownload=
target_zip=https://downloads.f5.com/esd/
selected_zip=
zip_file_name=
md5servedownload=
target_md5=https://downloads.f5.com/esd/
selected_md5=
md5_file_name=----------
Many Thanks!
Josef
Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com