APM Optimisation Script
Problem this snippet solves: With the current Covid-19 lockdown, many workers are now working from home which is putting stress on existing APM VPN devices. This script looks through the config and suggests some changes to be made to reduce CPU usage, based on https://support.f5.com/csp/article/K46161759 Matthieu Dierick has created a YouTube video showing how to use this at https://youtu.be/F0Z1AnM3L54 Let me know if you have any questions or requirements. Source code is held at https://github.com/pwhitef5/apm-vpn-optimisation/tree/master How to use this snippet: Copy the file to the /var/tmp directory as apm-optimisation Give it permissions with `chmod +x /var/tmp/apm-optimisation` Run with `/var/tmp/apm-optimisation`. Output is to stdout Example: [root@apm-1:Active:Standalone] ~ # ./apm-optimisation APM Optimisation Visibility CPU Usage -------------------------------- Current Average Maximum 52% 30% 93% -------------------------------- Compression -------------------------------- Licensed Hardware unlimited None -------------------------------- --- Partition /Common --- Connectivity Profile Compression -------------------------------- Profile Name Status -------------------------------- myConnectivity Disabled myConnectivity2 Disabled -------------------------------- Network Access Profile Compression ----------------------------------------------------------------------------------------------------------- Name | Compression | Split-Tunneling | Client Traffic Classifier | DTLS ----------------------------------------------------------------------------------------------------------- networkAccess | Enabled | Enabled | Disabled | Enabled networkAccess2 | Disabled | Enabled | Disabled | Disabled ----------------------------------------------------------------------------------------------------------- --- Optimisation Suggestions --- - CPU rate is LOW. Go down the Winchester and wait for it all to blow over - Hardware Compression is not included so consider turning off the feature ------- Partition /Common ------- - To turn off compression in the connectivity profile, run the command 'tmsh modify apm profile connectivity /Common/myConnectivity compression disabled' - To turn off compression in the NA profile, run the command 'tmsh modify apm resource network-access /Common/networkAccess compression none' - To turn on Client Traffic Classifier, run the commands below: tmsh create apm resource client-rate-class /Common/rate_class_2M { rate 2000000 } tmsh create apm resource client-rate-class /Common/rate_class_1M { rate 1000000 } tmsh create apm resource client-traffic-classifier /Common/client-traffic-classifier-1 { entries add { entry { client-rate-class rate_class_1M dst-ip any dst-mask any dst-port https src-ip any src-mask any } } } tmsh modify apm resource network-access /Common/networkAccess client-traffic-classifier client-traffic-classifier-1 - Network Access profile /Common/networkAccess is using SNAT automap. Consider using a SNAT pool - To turn on Client Traffic Classifier, run the commands below: tmsh create apm resource client-rate-class /Common/rate_class_2M { rate 2000000 } tmsh create apm resource client-rate-class /Common/rate_class_1M { rate 1000000 } tmsh create apm resource client-traffic-classifier /Common/client-traffic-classifier-1 { entries add { entry { client-rate-class rate_class_1M dst-ip any dst-mask any dst-port https src-ip any src-mask any } } } tmsh modify apm resource network-access /Common/networkAccess2 client-traffic-classifier client-traffic-classifier-1 - To turn on DTLS, create a duplicate virtual server listening on UDP and enabled DTLS in the Network Access List Network Settings ( see https://devcentral.f5.com/s/articles/APM-DTLS-Virtual-Server-iApp ) - Network Access profile /Common/networkAccess2 is using SNAT automap. Consider using a SNAT pool ----------------------------------------------------------------------------------------------------------- Code : #!/bin/bash # Version 5 8/4/2020 P.White # This is a script to check your APM system and give suggestions to reduce CPU usage # Taken from suggestions at https://support.f5.com/csp/article/K46161759 # v2 - small typo fix line 119 create changed to modify # v3 - updated classifier to only include https as it was causing an error # v4 - loops through admin partitions and prints out for each # v5 - added DTLS check and suggestion suggestions="--- Optimisation Suggestions ---\n" getLicensedCompression () { # Show the licensed compression comp=`tmsh -q show sys license detail|grep perf_http_compression|awk '{print $2}'|sed 's/\[\(.*\)\]/\1/g'` if [ x$comp != "x" ];then echo -n "$comp" else echo -n "Error!" fi } getHardwareCompression () { # Show hardware compression hcomp=`tmsh -q show sys license detail|grep "HTTP Hardware Compression"` if [ x$hcomp = "x" ];then # Hardware compression is not enabled echo -n "None" else echo -n "$hcomp" fi } clear echo "APM Optimisation Visibility" # CPU usage cur=`tmsh -q show sys cpu |grep "Utilization"|awk '{print $2}'` avg=`tmsh -q show sys cpu |grep "Utilization"|awk '{print $3}'` max=`tmsh -q show sys cpu |grep "Utilization"|awk '{print $4}'` if [ $avg -gt 90 ];then suggestions+=" - CPU rate is VERY HIGH! Turn off compression, implement split tunneling and consider more processing\n" elif [ $avg -gt 60 ];then suggestions+=" - CPU rate is HIGH! Turn off compression and consider split tunneling for non-internal traffic\n" elif [ $avg -gt 40 ];then suggestions+=" - CPU rate is MEDIUM. Consider turning off compression where required\n" else suggestions+=" - CPU rate is LOW. Go down the Winchester and wait for it all to blow over\n" fi echo echo "CPU Usage" echo "--------------------------------" echo -e "Current\tAverage\tMaximum" echo -e "$cur%\t$avg%\t$max%" echo "--------------------------------" echo # Compression clic=`getLicensedCompression` chw=`getHardwareCompression` if [ $chw = "None" ];then suggestions+=" - Hardware Compression is not included so consider turning off the feature\n" fi echo "Compression" echo "--------------------------------" echo -e "Licensed\tHardware" echo -e "$clic\t$chw" echo "--------------------------------" # loop through adminstrative partitions for partition in `tmsh -q list auth partition one-line|awk '{print $3}'`;do suggestions+="\n------- Partition /$partition -------\n" echo " --- Partition /$partition ---" echo echo "Connectivity Profile Compression" echo "--------------------------------" echo -e "Profile Name\t\tStatus" echo "--------------------------------" for profile in `tmsh -q -c "cd /$partition;list apm profile connectivity one-line"|awk '{print $4}'`;do if [ $profile = "connectivity" ];then continue fi if [ `tmsh -q -c "cd /$partition;list apm profile connectivity $profile one-line"|grep "compress-gzip-level 0"|wc -l` -gt 0 ];then echo -e "$profile\t\tDisabled" else suggestions+=" - To turn off compression in the connectivity profile, run the command 'tmsh modify apm profile connectivity /$partition/$profile compress-gzip-level 0'\n" echo -e "$profile\t\tEnabled" fi done echo "--------------------------------" echo echo "Network Access Profile Compression" echo "-----------------------------------------------------------------------------------------------------------" echo -e " Name\t\t\t| Compression\t| Split-Tunneling\t| Client Traffic Classifier\t| DTLS" echo "-----------------------------------------------------------------------------------------------------------" for profile in `tmsh -q -c "cd /$partition;list apm resource network-access one-line"|awk '{print $4}'`;do # Compression if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "compression gzip"|wc -l` -gt 0 ];then echo -en "$profile\t\t| Enabled" suggestions+=" - To turn off compression in the NA profile, run the command 'tmsh modify apm resource network-access /$partition/$profile compression none'\n" else echo -en "$profile\t\t| Disabled" fi if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "split-tunneling true"|wc -l` -gt 0 ];then echo -en "\t| Enabled" else echo -en "\t| Disabled" suggestions+=" - To turn on split-tunneling, run the command 'tmsh modify apm resource network-access /$partition/$profile split-tunneling true'\n" suggestions+=" - To configure split-tunneling exclude traffic by DNS name, run the command 'tmsh modify apm resource network-access /$partition/$profile address-space-exclude-dns-name add { office.com microsoftonline.com google.com gmail.com facebook.com }'\n" suggestions+=" - To configure split-tunneling exclude traffic by IP address, run the command 'tmsh modify apm resource network-access /$partition/$profile address-space-include-subnet add { { subnet 10.0.0.0/8 } { subnet 172.16.0.0/16 } { subnet 192.168.0.0/16 } }'\n" fi if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "client-traffic-classifier "|wc -l` -gt 0 ];then echo -en "\t\t| Enabled" else echo -en "\t\t| Disabled" suggestions+=" - To turn on Client Traffic Classifier, run the commands below:\n" suggestions+="tmsh create apm resource client-rate-class /$partition/rate_class_2M { rate 2000000 }\n" suggestions+="tmsh create apm resource client-rate-class /$partition/rate_class_1M { rate 1000000 }\n" suggestions+="tmsh create apm resource client-traffic-classifier /$partition/client-traffic-classifier-1 { entries add { entry { client-rate-class rate_class_1M dst-ip any dst-mask any dst-port https src-ip any src-mask any } } }\n" suggestions+="tmsh modify apm resource network-access /$partition/$profile client-traffic-classifier client-traffic-classifier-1\n" fi if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "dtls true"|wc -l` -gt 0 ];then echo -en "\t\t\t| Enabled" else echo -en "\t\t\t| Disabled" suggestions+=" - To turn on DTLS, create a duplicate virtual server listening on UDP and enabled DTLS in the Network Access List Network Settings ( see https://devcentral.f5.com/s/articles/APM-DTLS-Virtual-Server-iApp )\n" fi # Check for SNAT automap if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line all-properties"|grep "snat automap"|wc -l` -gt 0 ];then suggestions+=" - Network Access profile /$partition/$profile is using SNAT automap. Consider using a SNAT pool\n" fi echo "" done echo "-----------------------------------------------------------------------------------------------------------" # Check VSs for mirroring for vs in `tmsh list ltm virtual one-line|awk '{print $3}'`;do if [ `tmsh -q -c "cd /$partition;list ltm virtual $vs mirror"|grep "mirror enabled"|wc -l` -gt 0 ];then echo echo "WARNING! Virtual Server /$partition/$vs has mirroring enabled\n" echo suggestions+="Consider disabling Connection Mirroring for virtual server /$partition/$vs with the command 'tmsh modify ltm virtual /$partition/$vs mirror disabled'\n" fi done done echo echo -e "$suggestions" echo "-----------------------------------------------------------------------------------------------------------" Tested this on version: 13.0SSL VPN Tunnel and Office 365
Problem this snippet solves: This implements Regan Anderson's script from https://devcentral.f5.com/s/articles/SSL-VPN-Split-Tunneling-and-Office-365 as an iApp for simple deployment. This will install and configure the script and also create an iCall which runs every day at midnight. Also available at https://github.com/pwhitef5/apm-vpn-optimisation/blob/master/o365_iapp.tmpl How to use this snippet: Add the iApp via iApps>Templates Create a new application via iApps>Application Services>Applications, use the o365_optimisation iApp template Select your requirements ( remember to delete additional URLs and IPs if not used ) and hit finished Code : cli admin-partitions { update-partition Common } sys application template /Common/o365_optimisation { actions { definition { html-help { Office 365 Optimisation iCall Version 1 To see this in a wider window, hit the Launch button above For help on the various options, go to https://github.com/f5regan/o365-apm-split-tunnel This script fetches Office 365 URLs and IPs (IPv4 and/or IPv6) from Microsoft's Office 365 IP Address and URL web service, dynamically updates Network Access List "Exclude" properties for one or more Network Access Lists, and applies changes to the affected Access Policies. If the script is running on an HA pair of BIG-IPs then the script will also initiate a ConfigSync to push the updated configuration from the active BIG-IP to the standby BIG-IP. Script Requirements TMOS 12.1.0 or higher BIG-IP must be capable of resolving internet DNS names (ex. via DNS Lookup Server configuration) BIG-IP must be able to reach endpoints.office.com via TCP 443 (via Management or TMM interface) Administrative rights on the BIG-IP(s) Bash shell access on the BIG-IP(s) Things to Note This software is supplied "AS IS" without any warranties or support. This script does not enable “split tunneling” or make any other modifications, other than those mentioned, to the Network Access List(s) that may be required to enable the desired functionality. For guidance relating to Network Access List / Split Tunnelling configuration refer to the BIG-IP APM Knowledge Center. Some split tunneling guidance: Allow Local DNS Servers should be enabled to allow client access to Office 365 when VPN is disconnected DNS Exclude Address Space is not supported on macOS IPV6 Exclude Address Space doesn't currently work on macOS This script should not be used with BIG-IP Edge Client's Always Connected Mode: if Stonewall is configured to block traffic, then the excluded resources are not reachable (this is by design). This script tracks the version of the Office 365 service instance and will only update the exclusion lists if a newer version is detected. If modifications to the script's operating parameters (ex. Network Access Lists, O365 Service Areas, Additional URLs/IPs to Exclude) are made, they will NOT take effect until the script detects a new service instance version. To force the script to run with the updated parameters earlier, remove the o365_version.txt file from the script's working directory OR temporarily set force_o365_record_refresh = 1, then manually execute the script (python /shared/o365/apm_o365_update.py). This script overwrites the contents of the following fields in the defined Network Access Lists when an update is detected: If "use_url" is set to 1: DNS Exclude Address Space If "use_ipv4" is set to 1: IPV4 Exclude Address Space If "use_ipv6" is set to 1: IPV6 Exclude Address Space The aforementioned fields / properties should not be managed via TMSH or the GUI after implementing this script as the script will overwrite any manual changes to these fields when it detects the next Office 365 service instance update. To add non-Office 365 URLs/IPs to or remove any URL/IP (Office 365 or otherwise) from the "Exclude Address Space" properties of the Network Access Lists, use the noimport_* and additional_* fields in the script (see User Options in the Appendix for usage details) . Be sure to apply these changes to both units of an HA pair! Usage of DNS Address Exclusions requires the installation of the DNS Relay Proxy service on the VPN client K9694: Overview of the Windows DNS Relay Proxy service K49720803: BIG-IP Edge Client operations guide | Chapter 3: Common approaches to configuring VPN While the endpoints retrieved by this script handle the vast majority of Office 365 traffic, it is possible some traffic related to Office 365 is not summarized in the endpoint lists and will still go through the VPN. HA: This script must be implemented on both members of an HA pair HA: Updates made to the Python script are NOT synchronized between peers - updates must be made manually to each instance of the Python script HA: Exclusion list updates will only take place on the Active member - changes are synced from the Active to Standby by the script This script relies on iCall to periodically run. See What is iCall? on DevCentral for an overview of iCall. iCall scripts and periodic handlers are documented in greater detail in the F5 TMSH Reference. } implementation { set app_dir [tmsh::pwd] set app_name $tmsh::app_name set script {#!/usr/bin/env python # -*- coding: utf-8 -*- # O365 URL/IP update automation for BIG-IP # Version: 1.1 # Last Modified: 01 April 2020 # Original author: Makoto Omura, F5 Networks Japan G.K. # # Modified for APM Network Access "Exclude Address Space" by Regan Anderson, F5 Networks # Modified for iCall by Peter White, F5 Networks # # This Sample Software provided by the author is for illustrative # purposes only which provides customers with programming information # regarding the products. This software is supplied "AS IS" without any # warranties and support. # # The author assumes no responsibility or liability for the use of the # software, conveys no license or title under any patent, copyright, or # mask work right to the product. # # The author reserves the right to make changes in the software without # notification. The author also make no representation or warranty that # such application will be suitable for the specified use without # further testing or modification. #----------------------------------------------------------------------- import httplib import urllib import uuid import os import re import json import commands import datetime import sys #----------------------------------------------------------------------- # User Options - Configure as desired #----------------------------------------------------------------------- # Access Profile Name(s) - ex. SINGLE ["AP1"] OR MULTIPLE ["AP1", "AP2", "AP3"] access_profiles = [} set accessProfiles {} set naList {} foreach {row} $::main__access_profile { array set cols [lindex $row 0] lappend accessProfiles "\"$cols(profile)\" " lappend naLists "\"$cols(nalist)\" " } append script [ join $accessProfiles , ] append script {] # Network Access List Name(s) - ex. SINGLE ["NAL1"] OR MULTIPLE ["NAL1", "NAL2", "NAL3"] na_lists = [} append script [ join $naLists , ] append script {] # Microsoft Web Service Customer endpoints (ENABLE ONLY ONE ENDPOINT) # These are the set of URLs defined by customer endpoints as described here: https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges customer_endpoint = } append script "\"$::main__endpoint\"" append script { # O365 "SeviceArea" (O365 endpoints) to consume, as described here: https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges care_exchange = } append script $config__include_exchange append script { # "Exchange Online": 0=do not care, 1=care care_sharepoint = } append script $config__include_sharepoint append script { # "SharePoint Online and OneDrive for Business": 0=do not care, 1=care care_skype = } append script $config__include_skype append script { # "Skype for Business Online and Microsoft Teams": 0=do not care, 1=care care_common = } append script $config__include_common append script { # "Microsoft 365 Common and Office Online": 0=do not care, 1=care # O365 Record types to download & update use_url = } append script $config__use_url append script { # DNS/URL exclusions: 0=do not use, 1=use use_ipv4 = } append script $config__use_ipv4 append script { # IPv4 exclusions: 0=do not use, 1=use use_ipv6 = } append script $config__use_ipv6 append script { # IPv6 exclusions: 0=do not use, 1=use # O365 Categories to download & update o365_categories = } append script $config__o365_categories append script { # 0=Optimize only, 1= Optimize & Allow, 2 = Optimize, Allow, and Default # O365 Endpoints to import - O365 required endpoints or all endpoints # WARNING: "import all" includes non-O365 URLs that one may not want to bypass (ex. www.youtube.com) only_required = } append script $config__only_required append script { # 0=import all, 1=O365 required only # Don't import these O365 URLs (URL must be exact or ends_with match to URL as it exists in JSON record - pattern matching not supported) # Provide URLs in list format - ex. [".facebook.com", "*.itunes.apple.com", "bit.ly"] #noimport_urls = [] noimport_urls = [".symcd.com",".symcb.com",".entrust.net",".digicert.com",".identrust.com",".verisign.net",".globalsign.net",".globalsign.com",".geotrust.com",".omniroot.com",".letsencrypt.org",".public-trust.com","platform.linkedin.com"] # Don't import these O365 IPs (IP must be exact match to IP as it exists in JSON record - IP/CIDR mask cannot be modified) # Provide IPs (IPv4 and IPv6) in list format - ex. ["191.234.140.0/22", "2620:1ec:a92::152/128"] noimport_ips = [] # Non-O365 URLs to add to DNS Exclude List # Provide URLs in list format - ex. ["m.facebook.com", "*.itunes.apple.com", "bit.ly"] additional_urls = [} set additional_urls {} foreach {row} $::config__additional_urls { array set cols [lindex $row 0] lappend additional_urls "\"$cols(url)\"" } append script [ join $additional_urls , ] append script {] # Non-O365 IPs to add to IPV4 Exclude List # Provide IPs in list format - ex. ["191.234.140.0/22", "131.253.33.215/32"] additional_ipv4 = [} set additional_ips {} foreach {row} $::config__additional_ips { array set cols [lindex $row 0] lappend additional_ips "\"$cols(ip)\"" } append script [ join $additional_ips , ] append script {] # Non-O365 IPs to add to IPV6 Exclude List # Provide IPs in list format - ex. ["2603:1096:400::/40", "2620:1ec:a92::152/128"] additional_ipv6 = [] # Action if O365 endpoint list is not updated force_o365_record_refresh = 0 # 0=do not update, 1=update (for test/debug purpose) # BIG-IP HA Configuration device_group_name = "} append script $config__dg append script {" # Name of Sync-Failover Device Group. Required for HA paired BIG-IP. ha_config = } if { [llength [tmsh::get_config cm device]] > 1 } { append script "1" } else { append script "0" } append script { # 0=stand alone, 1=HA paired # Log configuration log_level = } append script $main__debug append script { # 0=none, 1=normal, 2=verbose #----------------------------------------------------------------------- # System Options - Modify only when necessary #----------------------------------------------------------------------- # Working directory, file name for guid & version management work_directory = "/shared/o365/" file_name_guid = "/shared/o365/guid.txt" file_ms_o365_version = "/shared/o365/o365_version.txt" log_dest_file = "/var/log/o365_update" # Microsoft Web Service URLs url_ms_o365_endpoints = "endpoints.office.com" url_ms_o365_version = "endpoints.office.com" uri_ms_o365_version = "/version?ClientRequestId=" #----------------------------------------------------------------------- # Implementation - Please do not modify #----------------------------------------------------------------------- list_urls_to_exclude = [] list_ipv4_to_exclude = [] list_ipv6_to_exclude = [] def log(lev, msg): if log_level >= lev: log_string = "{0:%Y-%m-%d %H:%M:%S}".format(datetime.datetime.now()) + " " + msg + "\n" f = open(log_dest_file, "a") f.write(log_string) f.flush() f.close() return def main(): # ----------------------------------------------------------------------- # Check if this BIG-IP is ACTIVE for the traffic group (= traffic_group_name) # ----------------------------------------------------------------------- result = commands.getoutput("tmsh show /cm failover-status field-fmt") if ("status ACTIVE" in result) or (ha_config == 0): log(1, "This BIG-IP is standalone or HA ACTIVE. Initiating O365 update.") else: log(1, "This BIG-IP is HA STANDBY. Aborting O365 update.") sys.exit(0) # ----------------------------------------------------------------------- # GUID management # ----------------------------------------------------------------------- # Create guid file if not existent if not os.path.isdir(work_directory): os.mkdir(work_directory) log(1, "Created work directory " + work_directory + " because it did not exist.") if not os.path.exists(file_name_guid): f = open(file_name_guid, "w") f.write("\n") f.flush() f.close() log(1, "Created GUID file " + file_name_guid + " because it did not exist.") # Read guid from file and validate. Create one if not existent f = open(file_name_guid, "r") f_content = f.readline() f.close() if re.match('[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', f_content): guid = f_content log(2, "Valid GUID is read from local file " + file_name_guid + ".") else: guid = str(uuid.uuid4()) f = open(file_name_guid, "w") f.write(guid) f.flush() f.close() log(1, "Generated a new GUID, and saved it to " + file_name_guid + ".") # ----------------------------------------------------------------------- # O365 endpoints list version check # ----------------------------------------------------------------------- # Read version of previously received record if os.path.isfile(file_ms_o365_version): f = open(file_ms_o365_version, "r") f_content = f.readline() f.close() # Check if the VERSION record format is valid if re.match('[0-9]{10}', f_content): ms_o365_version_previous = f_content log(2, "Valid previous VERSION found in " + file_ms_o365_version + ".") else: ms_o365_version_previous = "1970010200" f = open(file_ms_o365_version, "w") f.write(ms_o365_version_previous) f.flush() f.close() log(1, "Valid previous VERSION was not found. Wrote dummy value in " + file_ms_o365_version + ".") else: ms_o365_version_previous = "1970010200" f = open(file_ms_o365_version, "w") f.write(ms_o365_version_previous) f.flush() f.close() log(1, "Valid previous VERSION was not found. Wrote dummy value in " + file_ms_o365_version + ".") # ----------------------------------------------------------------------- # O365 endpoints list VERSION check # ----------------------------------------------------------------------- request_string = uri_ms_o365_version + guid conn = httplib.HTTPSConnection(url_ms_o365_version) conn.request('GET', request_string) res = conn.getresponse() if not res.status == 200: # MS O365 version request failed log(1, "VERSION request to MS web service failed. Assuming VERSIONs did not match, and proceed.") dict_o365_version = {} else: # MS O365 version request succeeded log(2, "VERSION request to MS web service was successful.") dict_o365_version = json.loads(res.read()) ms_o365_version_latest = "" for record in dict_o365_version: if record.has_key('instance'): if record["instance"] == customer_endpoint and record.has_key("latest"): latest = record["latest"] if re.match('[0-9]{10}', latest): ms_o365_version_latest = latest f = open(file_ms_o365_version, "w") f.write(ms_o365_version_latest) f.flush() f.close() log(2, "Previous VERSION is " + ms_o365_version_previous) log(2, "Latest VERSION is " + ms_o365_version_latest) if ms_o365_version_latest == ms_o365_version_previous and force_o365_record_refresh == 0: log(1, "You already have the latest MS O365 URL/IP Address list: " + ms_o365_version_latest + ". Aborting operation.") sys.exit(0) # ----------------------------------------------------------------------- # Request O365 endpoints list & put it in dictionary # ----------------------------------------------------------------------- request_string = "/endpoints/" + customer_endpoint + "?ClientRequestId=" + guid conn = httplib.HTTPSConnection(url_ms_o365_endpoints) conn.request('GET', request_string) res = conn.getresponse() if not res.status == 200: log(1, "ENDPOINTS request to MS web service failed. Aborting operation.") sys.exit(0) else: log(2, "ENDPOINTS request to MS web service was successful.") dict_o365_all = json.loads(res.read()) # Process for each record(id) of the endpoint JSON data for dict_o365_record in dict_o365_all: service_area = str(dict_o365_record['serviceArea']) category = str(dict_o365_record['category']) if (o365_categories == 0 and category == "Optimize") \ or (o365_categories == 1 and (category == "Optimize" or category == "Allow")) \ or (o365_categories == 2): if (only_required == 0) or (only_required and str(dict_o365_record['required']) == "True"): if (care_common and service_area == "Common") \ or (care_exchange and service_area == "Exchange") \ or (care_sharepoint and service_area == "SharePoint") \ or (care_skype and service_area == "Skype"): if use_url: # Append "urls" if existent in each record if dict_o365_record.has_key('urls'): list_urls = list(dict_o365_record['urls']) for url in list_urls: list_urls_to_exclude.append(url) # Append "allowUrls" if existent in each record if dict_o365_record.has_key('allowUrls'): list_allow_urls = list(dict_o365_record['allowUrls']) for url in list_allow_urls: list_urls_to_exclude.append(url) # Append "defaultUrls" if existent in each record if dict_o365_record.has_key('defaultUrls'): list_default_urls = dict_o365_record['defaultUrls'] for url in list_default_urls: list_urls_to_exclude.append(url) if use_ipv4 or use_ipv6: # Append "ips" if existent in each record if dict_o365_record.has_key('ips'): list_ips = list(dict_o365_record['ips']) for ip in list_ips: if re.match('^.+:', ip): list_ipv6_to_exclude.append(ip) else: list_ipv4_to_exclude.append(ip) log(1, "Number of unique ENDPOINTS to import...") # Add administratively defined URLs/IPs and (Re)process to remove duplicates and excluded values if use_url: # Combine lists and remove duplicate URLs urls_undup = list(set(list_urls_to_exclude + additional_urls)) ## Remove set of excluded URLs from the list of collected URLs for x_url in noimport_urls: urls_undup = [x for x in urls_undup if not x.endswith(x_url)] log(1, "URL: " + str(len(urls_undup))) if use_ipv4: # Combine lists and remove duplicate IPv4 addresses ipv4_undup = list(set(list_ipv4_to_exclude + additional_ipv4)) ## Remove set of excluded IPv4 addresses from the list of collected IPv4 addresses for x_ip in noimport_ips: ipv4_undup = [x for x in ipv4_undup if not x.endswith(x_ip)] log(1, "IPv4 host/net: " + str(len(ipv4_undup))) if use_ipv6: # Combine lists and duplicate IPv6 addresses ipv6_undup = list(set(list_ipv6_to_exclude + additional_ipv6)) ## Remove set of excluded IPv6 addresses from the list of collected IPv6 addresses for x_ip in noimport_ips: ipv6_undup = [x for x in ipv6_undup if not x.endswith(x_ip)] log(1, "IPv6 host/net: " + str(len(ipv6_undup))) # ----------------------------------------------------------------------- # URLs, IPv4 & IPv6 addresses formatted for TMSH # ----------------------------------------------------------------------- if use_url: # Initialize the URL string url_exclude_list = "" # Write URLs to string for url in urls_undup: url_exclude_list = url_exclude_list + " " + url.lower() if use_ipv4: # Initialize the IPv4 string ipv4_exclude_list = "" # Write IPv4 addresses to string for ip4 in (list(sorted(ipv4_undup))): ipv4_exclude_list = ipv4_exclude_list + "{subnet " + ip4 + " } " if use_ipv6: # Initialize the IPv6 string ipv6_exclude_list = "" # Write IPv6 addresses to string for ip6 in (list(sorted(ipv6_undup))): ipv6_exclude_list = ipv6_exclude_list + "{subnet " + ip6 + " } " # ----------------------------------------------------------------------- # Load URL and/or IPv4 and/or IPv6 lists into Network Access resource # ----------------------------------------------------------------------- if use_url: for na in na_lists: result = commands.getoutput("tmsh modify /apm resource network-access " + na + " address-space-exclude-dns-name replace-all-with { " + url_exclude_list + " }") log(2, "Updated " + na + " with latest O365 URL list.") if use_ipv4: for na in na_lists: result = commands.getoutput("tmsh modify /apm resource network-access " + na + " address-space-exclude-subnet { " + ipv4_exclude_list + " }") log(2, "Updated " + na + " with latest IPv4 O365 address list.") if use_ipv6: for na in na_lists: result = commands.getoutput("tmsh modify /apm resource network-access " + na + " ipv6-address-space-exclude-subnet { " + ipv6_exclude_list + " }") log(2, "Updated " + na + " with latest IPv6 O365 address list.") #----------------------------------------------------------------------- # Apply Access Policy and Initiate Config Sync: Device to Group #----------------------------------------------------------------------- for ap in access_profiles: result = commands.getoutput("tmsh modify /apm profile access " + ap + " generation-action increment") if ha_config == 1: log(1, "Initiating Config-Sync.") result = commands.getoutput("tmsh run cm config-sync to-group " + device_group_name) log(2, result + "\n") log(1, "Completed O365 URL/IP address update process.") if __name__=='__main__': main() } # Create directory catch { exec /bin/mkdir -p /shared/o365 # Create the script set scriptname "/shared/o365/apm_o365_update_${app_name}.py" set fh [ open $scriptname w+ ] puts $fh $script close $fh # Make the script executable exec /bin/chmod +x $scriptname } set scriptText "script_${app_name} definition \{ catch \{ exec python $scriptname \} \}" tmsh::create sys icall script $scriptText tmsh::create sys icall handler periodic handler_${app_name} script script_${app_name} interval 86400 first-occurrence [ clock format [clock seconds] -format %Y-%m-%d:23:59:59 ] } presentation { section main { # The entry below creates a large text box that must be filled out with a valid IP Address # For details of APL, look at the iApps developers guide: # https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-iapps-developer-11-4-0.html choice debug display "medium" default "Off" {"Off" => "0", "Low" => "1","High" => "2"} optional (debug != "0") { message message1 "Note: logging output goes to /var/log/o365_update" } table access_profile { choice profile display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items apm profile access]" } choice nalist display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items apm resource network-access]" } } choice endpoint display "medium" default "Worldwide" {"Worldwide" => "Worldwide", "USGovDoD" => "USGovDoD","USGovGCCHigh" => "USGovGCCHigh","China" => "China","Germany" => "Germany"} } section config { choice include_exchange display "medium" default "1" {"No" => "0", "Yes" => "1"} choice include_sharepoint display "medium" default "0" {"No" => "0", "Yes" => "1"} choice include_skype display "medium" default "0" {"No" => "0", "Yes" => "1"} choice include_common display "medium" default "0" {"No" => "0", "Yes" => "1"} choice use_url display "medium" default "1" {"No" => "0", "Yes" => "1"} choice use_ipv4 display "medium" default "1" {"No" => "0", "Yes" => "1"} choice use_ipv6 display "medium" default "0" {"No" => "0", "Yes" => "1"} choice o365_categories display "medium" default "0" {"Optimize Only" => "0", "Optimize and Allow" => "1", "Optimize, Allow and Default" => "2"} choice only_required display "medium" default "1" {"No" => "0", "Yes" => "1"} table additional_urls { string url required display "medium" validator "IpOrFqdn" } table additional_ips { string ip required display "medium" validator "IpAddress" } choice dg display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items cm device-group]" } } text { # Entities below set the text for the questions and section names, etc. Make them simple and relevant. main "Main" main.debug "Logging Level" main.message1 "" main.access_profile "Access Profiles / NA List" main.access_profile.profile "Access Profile" main.access_profile.nalist "NA List" main.endpoint "Endpoint" config "Configuration" config.include_exchange "Exchange" config.include_sharepoint "Sharepoint" config.include_skype "Skype" config.include_common "Common" config.use_url "Use URLs" config.use_ipv4 "Use IPv4 subnets" config.use_ipv6 "Use IPv6 subnets" config.o365_categories "O365 Categories" config.only_required "Required Endpoints Only" config.additional_urls "Additional URLs" config.additional_urls.url "FQDN" config.additional_ips "Additional IPs" config.additional_ips.ip "IP/Mask" config.dg "High Availability Device Group" } } role-acl none run-as none } } description "o365 split tunnel update iCall v1" ignore-verification false requires-bigip-version-max none requires-bigip-version-min none requires-modules { apm } signing-key none tmpl-checksum none tmpl-signature none } Tested this on version: 13.0APM VPN Optimisation iApp
Problem this snippet solves: This is an iApp which creates an iCall that runs every 5 minutes and checks the 5-minute average CPU rate averaged across all CPUs. Depending on the CPU rate, features are enabled or disabled such as compression. Note that this will make changes to all of your connectivity profiles ( except the built-in `connectivity` profile ) and Network Access profiles so you should have a backup before deploying. Changes are auto-applied to all SSL-VPN APM access profiles Logging is done to /var/log/ltm as shown below. CPU > 90% Compression Off Split-tunneling On Default Rate Class 100K Client-traffic-classifier Enabled CPU > 60% Compression Off Split-tunneling On Default Rate Class 500K Client-traffic-classifier Enabled CPU > 40% Compression Off Split-tunneling Off Default Rate Class 1M Client-traffic-classifier Enabled CPU > 20% Compression On Split-tunneling Off Default Rate Class 1M Client-traffic-classifier Enabled CPU < 20% Compression On Split-tunneling Off Default Rate Class 1M Client-traffic-classifier Disabled I have tested the basic workings of this but it has NOT been tested in a production environment. I would be happy to have some pilot customers to try it out and help develop. Source code is held at https://github.com/pwhitef5/apm-vpn-optimisation/tree/master How to use this snippet: Copy and paste the text below into a document on your PC or jump server Navigate to the BIG-IP GUI iApps>Templates. Click on Import Tick 'Overwrite Existing Templates' and select the file you created in step 1. Click Upload Create a service by navigating to iApps>Application Services>Applications. Click Create Call the service 'APM-VPN-Optimisation' or a suitable name, select the 'apm-vpn-optimisation_icall' template Hit Finished To view the changes made by the iCall, login to the BIG-IP via ssh and run the command `tailf /var/log/ltm` Example logs: Mar 20 14:55:00 apm-1 notice scriptd[9780]: 01420004:5: apm-vpn-optimisation:CPU rate: 1 Mar 20 14:55:00 apm-1 notice scriptd[9780]: 01420004:5: apm-vpn-optimisation:Turning on compression for profile myConnectivity: compress-gzip-level 6 Mar 20 14:55:00 apm-1 notice scriptd[9780]: 01420004:5: apm-vpn-optimisation:Turning on compression for profile networkAccess: compression gzip Mar 20 14:55:00 apm-1 notice scriptd[9780]: 01420004:5: apm-vpn-optimisation:Turning off split-tunneling for profile networkAccess: split-tunneling false Mar 20 14:55:00 apm-1 notice scriptd[9780]: 01420004:5: apm-vpn-optimisation:Creating client-rate-classes and client-traffic-classifier-1. rate: rate_class_1M Code : cli admin-partitions { update-partition Common } sys application template /Common/apm-vpn-optimisation_icall { actions { definition { html-help { } implementation { set app_dir [tmsh::pwd] set app_name $tmsh::app_name set icallTemplate {# Retrieve the CPU usage set cpuStatus [tmsh::get_status sys cpu] set numCpus 0 set totalUsage 0 foreach {cpu} $cpuStatus { incr numCpus set name [tmsh::get_name $cpu] set value [tmsh::get_field_value $cpu cpu-info.${name}.five-min-avg-system ] incr totalUsage $value } set cpuRate [ expr { $totalUsage / $numCpus } ] tmsh::log "apm-vpn-optimisation:CPU rate: $cpuRate" # Set features on or off if { $cpuRate > 90 } { set compression 0 set split-tunneling 1 set rate-class "rate_class_100K" set client-traffic-classifier 1 } elseif { $cpuRate > 60 } { set compression 0 set split-tunneling 1 set rate-class "rate_class_500K" set client-traffic-classifier 1 } elseif { $cpuRate > 40 } { set compression 0 set split-tunneling 0 set rate-class "rate_class_1M" set client-traffic-classifier 1 } elseif { $cpuRate > 20 } { set compression 1 set split-tunneling 0 set rate-class "rate_class_1M" set client-traffic-classifier 1 } else { set compression 1 set split-tunneling 0 set rate-class "rate_class_1M" set client-traffic-classifier 0 } set changed 0 # Set compression set connectivityProfiles [tmsh::get_config apm profile connectivity all-properties] foreach {profile} $connectivityProfiles { set name [tmsh::get_name $profile] if { $name == "connectivity" } { continue } # Get current status set currentStatus [tmsh::get_field_value $profile compress-gzip-level] if { $currentStatus < 1 && $compression > 0 } { # If it is turned off and should be on then turn on # Turn on tmsh::log "apm-vpn-optimisation:Turning on compression for profile $name: compress-gzip-level 6" tmsh::modify apm profile connectivity $name compress-gzip-level 6 } elseif { $currentStatus > 0 && $compression < 1 } { # Turn off tmsh::log "apm-vpn-optimisation:Turning off compression for profile $name: compress-gzip-level 0" tmsh::modify apm profile connectivity $name compress-gzip-level 0 } } set networkAccessProfiles [tmsh::get_config apm resource network-access all-properties] foreach {profile} $networkAccessProfiles { set name [tmsh::get_name $profile] set currentStatus [tmsh::get_field_value $profile compression] if { $currentStatus == "none" && $compression > 0 } { # Turn on tmsh::log "apm-vpn-optimisation:Turning on compression for profile $name: compression gzip" tmsh::modify apm resource network-access $name compression gzip set changed 1 } elseif { $currentStatus == "gzip" && $compression < 1} { # Turn off tmsh::log "apm-vpn-optimisation:Turning off compression for profile $name: compression none" tmsh::modify apm resource network-access $name compression none set changed 1 } } # Set split-tunneling set networkAccessProfiles [tmsh::get_config apm resource network-access all-properties] foreach {profile} $networkAccessProfiles { set name [tmsh::get_name $profile] set currentStatus [tmsh::get_field_value $profile split-tunneling] tmsh::begin_transaction if { $currentStatus != "true" && ${split-tunneling} > 0 } { tmsh::log "apm-vpn-optimisation:Turning on split-tunneling for profile $name: split-tunneling true" tmsh::modify apm resource network-access $name address-space-exclude-dns-name add \{ office.com microsoftonline.com google.com gmail.com facebook.com \} tmsh::modify apm resource network-access $name address-space-include-subnet \{\{ subnet 10.0.0.0/8 \} \{ subnet 172.16.0.0/16 \} \{ subnet 192.168.0.0/16 \}\} tmsh::modify apm resource network-access $name split-tunneling true set changed 1 } elseif { $currentStatus == "true" && ${split-tunneling} < 1 } { tmsh::log "apm-vpn-optimisation:Turning off split-tunneling for profile $name: split-tunneling false" tmsh::modify apm resource network-access $name split-tunneling false set changed 1 } tmsh::commit_transaction } # Create rate class tmsh::log "apm-vpn-optimisation:Creating client-rate-classes and client-traffic-classifier-1. rate: ${rate-class}" tmsh::stateless enabled tmsh::begin_transaction tmsh::create apm resource client-rate-class rate_class_4M \{ rate 4000000 \} tmsh::create apm resource client-rate-class rate_class_2M \{ rate 2000000 \} tmsh::create apm resource client-rate-class rate_class_1M \{ rate 1000000 \} tmsh::create apm resource client-rate-class rate_class_500K \{ rate 500000 \} tmsh::create apm resource client-rate-class rate_class_100K \{ rate 100000 \} tmsh::create apm resource client-traffic-classifier client-traffic-classifier-1 \{ entries add \{ \ entry \{ client-rate-class ${rate-class} dst-ip any dst-mask any dst-port https src-ip any src-mask any \} \ entry0 \{ client-rate-class rate_class_2M dst-ip any dst-mask any dst-port stun protocol 17 src-ip any src-mask any \} \ entry1 \{ client-rate-class rate_class_2M dst-ip any dst-mask any dst-port twrpc protocol 17 src-ip any src-mask any \} \ entry2 \{ client-rate-class rate_class_2M dst-ip any dst-mask any dst-port plethora protocol 17 src-ip any src-mask any \} \ entry3 \{ client-rate-class rate_class_2M dst-ip any dst-mask any dst-port cleanerliverc protocol 17 src-ip any src-mask any \} \ \} \} tmsh::commit_transaction tmsh::stateless disabled set networkAccessProfiles [tmsh::get_config apm resource network-access all-properties] foreach {profile} $networkAccessProfiles { set name [tmsh::get_name $profile] set currentStatus [tmsh::get_field_value $profile client-traffic-classifier] tmsh::begin_transaction if { $currentStatus != "client-traffic-classifier-1" && ${client-traffic-classifier} > 0 } { # Turn on tmsh::log "apm-vpn-optimisation:Turning on client-traffic-classifier for profile $name: client-traffic-classifier client-traffic-classifier-1" tmsh::modify apm resource network-access $name client-traffic-classifier client-traffic-classifier-1 set changed 1 } elseif { $currentStatus == "client-traffic-classifier-1" && ${client-traffic-classifier} < 1} { # Turn off tmsh::log "apm-vpn-optimisation:Turning off client-traffic-classifier for profile $name: client-traffic-classifier none" tmsh::modify apm resource network-access $name client-traffic-classifier none set changed 1 } tmsh::commit_transaction } # Apply profiles if { $changed > 0 } { set accessProfiles [tmsh::get_config apm profile type] foreach {profile} $accessProfiles { set name [tmsh::get_name $profile] # Check type of profile is ssl-vpn if { [tmsh::get_field_value $profile type] == "ssl-vpn" } { tmsh::log "apm-vpn-optimisation: Applying SSL-VPN access profile $name" tmsh::modify apm profile access $name generation-action increment } } } # Left blank } tmsh::create sys icall script "${app_name}_avo_script definition { [tmsh::expand_macro $icallTemplate ] }" tmsh::create sys icall handler periodic ${app_name}_avo_handler interval 300 script ${app_name}_avo_script } presentation { section main { # The entry below creates a large text box that must be filled out with a valid IP Address # For details of APL, look at the iApps developers guide: # https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-iapps-developer-11-4-0.html message intro "APM VPN Optimisation Version 1 20/3/2020" message usage "Note that this iApp will create an iCall which runs every 5 mins and changes your connectivity and network-access profiles automatically. You should backup your configuration before use to allow rollback to original configuration" } text { # Entities below set the text for the questions and section names, etc. Make them simple and relevant. main "Main" main.intro "" main.usage "" } } role-acl none run-as none } } description "APM VPN Optimisation iApp v2" ignore-verification false requires-bigip-version-max none requires-bigip-version-min none requires-modules { apm } signing-key none tmpl-checksum none tmpl-signature none } Tested this on version: 13.0APM VPN Bandwidth Controller iApp
Problem this snippet solves: Overview This iApp will create a set of virtual servers to apply a Bandwidth Controller policy to VPN tunnel traffic. Example iperf without the iApp: $ iperf -c 10.20.20.3 ------------------------------------------------------------ Client connecting to 10.20.20.3, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.20.20.131 port 5957 connected with 10.20.20.3 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 184 MBytes 154 Mbits/sec iperf with 10Mbps dynamic policy $ iperf -c 10.20.20.3 ------------------------------------------------------------ Client connecting to 10.20.20.3, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.20.20.131 port 6066 connected with 10.20.20.3 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.2 sec 12.1 MBytes 9.98 Mbits/sec iperf with 1Gbps dynamic policy $ iperf -c 10.20.20.3 ------------------------------------------------------------ Client connecting to 10.20.20.3, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.20.20.131 port 6569 connected with 10.20.20.3 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 190 MBytes 159 Mbits/sec Function This creates a set of virtual servers listening on the VPN tunnel with an iRule assigned which applies the BWC policy to both upload and download traffic. You can change your BWC rate as you require, it will be applied to new flows ie users don't have to reconnect. This has been tested that it deploys and works on v13 but I have not tested this in a production environment, therefore you should test its usage prior to implementation in a production environment. If you have successfully tested it then please PM with details and/or suggestions. How to use this snippet: Usage Instructions This assumes that you already have a VPN configured Create a Bandwidth Controller Policy with the overall bandwidth for the WHOLE VPN, and enable Dynamic if you want to specify the bandwidth for each user. In the example below, the Maximum Rate of 20Mbps is applied to the whole VPN and 10Mbps is applied to each flow. Load the iApp template at iApps>Templates and Import Deploy a new iApp service at iApps>Application Services>Applications and select the apm_bwc_iapp template Select the VPN tunnel and the Default BWC policy Select any SNAT requirements If you want to apply different rates to different traffic then add entries in the Protocol-specific Bandwidth Controller table. If you don't want to add these then click the X to remove the default entry. Hit Finished Objects created: Code : cli admin-partitions { update-partition Common } sys application template /Common/apm_bwc_iapp { actions { definition { html-help { } implementation { set app_dir [tmsh::pwd] set app_name $tmsh::app_name # https://support.f5.com/csp/article/K54955814 set rule_tcp {when CLIENT_ACCEPTED { BWC::policy attach <%=$bwc_policy%> "[IP::remote_addr]:[TCP::remote_port]" } when SERVER_CONNECTED { BWC::policy attach <%=$bwc_policy%> "[IP::remote_addr]:[TCP::remote_port]" } } set rule_udp {when CLIENT_ACCEPTED { BWC::policy attach <%=$bwc_policy%> "[IP::remote_addr]:[UDP::remote_port]" } when SERVER_CONNECTED { BWC::policy attach <%=$bwc_policy%> "[IP::remote_addr]:[UDP::remote_port]" } } if { $::main__use_snat == "Automap" } { set snat "source-address-translation \{ type automap \} " } elseif { $::main__use_snat == "SNAT Pool" } { set snat "source-address-translation \{ type snat pool $::main__snatpool \} " } else { set snat "" } # Create default iRule tmsh::create ltm rule rule_bwc_${app_name}_udp_default [ tmsh::expand_macro $rule_udp -vars "bwc_policy \"$::main__bwc_policy\"" ] tmsh::create ltm rule rule_bwc_${app_name}_tcp_default [ tmsh::expand_macro $rule_tcp -vars "bwc_policy \"$::main__bwc_policy\"" ] # Create default VS tmsh::create ltm virtual vs_bwc_${app_name}_udp_default ip-protocol udp vlans-enabled vlans replace-all-with \{ $::main__tunnel \} destination 0.0.0.0:any mask any $snat profiles replace-all-with \{ udp \} rules \{ rule_bwc_${app_name}_udp_default \} source 0.0.0.0/0 translate-address disabled translate-port disabled tmsh::create ltm virtual vs_bwc_${app_name}_tcp_default ip-protocol tcp vlans-enabled vlans replace-all-with \{ $::main__tunnel \} destination 0.0.0.0:any mask any $snat profiles replace-all-with \{ tcp \} rules \{ rule_bwc_${app_name}_tcp_default \} source 0.0.0.0/0 translate-address disabled translate-port disabled # Create custom ports and iRules foreach {row} $::main__entries { array set cols [lindex $row 0] # protocol, port and bwc_policy set rulename "rule_bwc_${app_name}_$cols(protocol)_$cols(port)" set vsname "vs_bwc_${app_name}_$cols(protocol)_$cols(port)" if { $cols(protocol) == "tcp" } { tmsh::create ltm rule $rulename [tmsh::expand_macro $rule_tcp -vars "bwc_policy \"$cols(bwc_policy)\"" ] tmsh::create ltm virtual $vsname ip-protocol tcp vlans-enabled vlans replace-all-with \{ $::main__tunnel \} destination 0.0.0.0:$cols(port) mask any $snat profiles replace-all-with \{ $cols(protocol) \} rules \{ $rulename \} source 0.0.0.0/0 translate-address disabled translate-port disabled } else { tmsh::create ltm rule $rulename [tmsh::expand_macro $rule_udp -vars "bwc_policy \"$cols(bwc_policy)\"" ] tmsh::create ltm virtual $vsname ip-protocol udp vlans-enabled vlans replace-all-with \{ $::main__tunnel \} destination 0.0.0.0:$cols(port) mask any $snat profiles replace-all-with \{ $cols(protocol) \} rules \{ $rulename \} source 0.0.0.0/0 translate-address disabled translate-port disabled } } } macro { } presentation { section main { # The entry below creates a large text box that must be filled out with a valid IP Address # For details of APL, look at the iApps developers guide: # https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-iapps-developer-11-4-0.html message intro "This iApp will create a forwarding virtual server on the specified VPN tunnel which intercepts the traffic and assigns a BWC policy" choice tunnel display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items net tunnel]" } choice bwc_policy display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items -norecursive net bwc policy]" } choice use_snat display "large" default "None" { "None" => "None", "Automap" => "Automap", "SNAT Pool" => "SNAT Pool" } optional (use_snat == "SNAT Pool") { choice snatpool display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items ltm snatpool]" } } table entries { choice protocol display "large" default "tcp" { "tcp" => "tcp", "udp" => "udp" } string port display "large" required validator "PortNumber" default "443" choice bwc_policy display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items -norecursive net bwc policy]" } } } text { # Entities below set the text for the questions and section names, etc. Make them simple and relevant. main "Main" main.intro "Usage" main.tunnel "VPN Tunnel" main.bwc_policy "Default BWC Policy" main.use_snat "Source Address Translation" main.snatpool "SNAT Pool" main.entries "Protocol-specific Bandwidth Controller" main.entries.protocol "Protocol" main.entries.port "Port" main.entries.bwc_policy "BWC Policy" } } role-acl none run-as none } } description "iApp to create an outgoing VS to apply a BWC policy to VPN user traffic v2" ignore-verification false requires-bigip-version-max none requires-bigip-version-min none requires-modules { apm } signing-key none tmpl-checksum none tmpl-signature none } Tested this on version: 13.0APM DTLS Virtual Server iApp
Problem this snippet solves: TCP VPN tunnels suffer from an issue known as "TCP meltdown" where packets dropped on the tunnel cause both TCP tunnels to backoff and we see a sawtooth throughput pattern. See https://en.wikipedia.org/wiki/Tunneling_protocol This iApp will clone your existing TCP VPN virtual server and create a UDP DTLS virtual server on port 4433. It will also enable DTLS on your selected Network Access profile. Tested on v13.1 but not used in a production deployment Available on Github at https://github.com/pwhitef5/apm-vpn-optimisation/blob/master/apm_dtls_iapp.tmpl How to use this snippet: Install the iApp as normal Deploy a service and select the VPN virtual server and Network Access profile in use The newly created virtual server will be a clone of the original called <VS name>_dtls and listening on UDP Code : cli admin-partitions { update-partition Common } sys application template /Common/apm_dtls_iapp { actions { definition { html-help { } implementation { set app_dir [tmsh::pwd] set app_name $tmsh::app_name # https://support.f5.com/csp/article/K54955814 set vs [lindex [ tmsh::get_config ltm virtual $::main__vsname ] 0] set profiles [ tmsh::get_field_value $vs profiles ] set connectivityProfile "" set clientsslProfile "" foreach profile $profiles { set name [ lindex [ split $profile ] 1 ] set newname "${name}-dtls" # Retrieve the connectivity profile if { ! [ catch { tmsh::get_config apm profile connectivity $name } ] } { set connectivityProfile $name } # Retrieve the client-ssl profile if { ! [ catch { tmsh::get_config ltm profile client-ssl $name } ] } { set clientsslProfile $name } } if { $connectivityProfile == "" } { error "Error! Virtual server $::main__vsname does not have a connectivity profile assigned" } if { $clientsslProfile == "" } { error "Error! Virtual server $::main__vsname does not have a client SSL profile assigned" } set vip "[lindex [split [ tmsh::get_field_value $vs destination ] : ] 0 ]:4433" # Check if VLANs are set if { [catch {set vlans [ tmsh::get_field_value $vs vlans]} err ] } { # VLANS are not configured set vlans "" } else { # VLANS are configured if { $vlans != "" } { set vlans " vlans-enabled vlans replace-all-with \{ $vlans \}" } else { set vlans "" } } # Check if SNAT is set if { [ catch {set snatType [ tmsh::get_field_value $vs source-address-translation.type ] } err ]} { # No SNAT pool set snat "" } elseif { $snatType == "automap" } { # Automap set snat " source-address-translation \{ type automap \}" } elseif { $snatType == "snat" } { # SNAT pool is set set snatpool [ tmsh::get_field_value $vs source-address-translation.pool ] set snat " source-address-translation \{ type snat pool $snatpool \}" } else { # Should never hit this set snat "" } # Create VS tmsh::create ltm virtual $newname $vlans $snat ip-protocol udp destination $vip profiles replace-all-with \{ $connectivityProfile \{ context clientside \} $clientsslProfile \{ context clientside \} \} # Change network access profile tmsh::stateless enabled tmsh::modify apm resource network-access $::main__networkAccessProfile dtls true tmsh::stateless disabled } presentation { section main { # The entry below creates a large text box that must be filled out with a valid IP Address # For details of APL, look at the iApps developers guide: # https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-iapps-developer-11-4-0.html message intro "This iApp will create a DTLS virtual server based on your existing VPN virtual server, and add DTLS support to the network access profile" choice vsname display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items ltm virtual]" } choice networkAccessProfile display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items apm resource network-access]" } } text { # Entities below set the text for the questions and section names, etc. Make them simple and relevant. main "Main" main.intro "Usage" main.vsname "VPN Virtual Server" main.networkAccessProfile "Network Access Profile" } } role-acl none run-as none } } description "iApp to create a DTLS virtual server based on your existing TCP VPN virtual server" ignore-verification false requires-bigip-version-max none requires-bigip-version-min none requires-modules { apm } signing-key none tmpl-checksum none tmpl-signature none } Tested this on version: 13.0
