Intelligent Proxy Steering - Office365
Introduction
This solution started back in May 2015 when I was helping a customer bypass their forward proxy servers due to the significant increase in the number of client connections after moving to Office365. Luckily they had a BIG-IP in front of their forward proxy servers load balancing the traffic and F5 had introduced a new “Proxy Mode” feature in the HTTP profile in TMOS 11.5. This allowed the BIG-IP to terminate Explicit Proxy connections, instead of passing them through to the pool members. The original solution was a simple iRule that referenced a data-group to determine if the connection should bypass the forward proxy pool or reverse proxy and load balance the connection as normal.
Original iRule:
when HTTP_PROXY_REQUEST {
# Strip of the port number
set hostname [lindex [split [HTTP::host] ":"] 0]
# If the hostname matches a MS 0ffice365 domain, enable the Forward Proxy on BIG-IP.
# BIG-IP will then perform a DNS lookup and act as a Forward Proxy bypassing the Forward Proxy
# Server Pool (BlueCoat/Squid/IronPort etc..)
if { [class match $hostname ends_with o365_datagroup] } {
# Use a SNAT pool - recommended
snatpool o365_snatpool
HTTP::proxy
} else {
# Load balance/reverse proxy to the Forward Proxy Server Pool (BlueCoat/Squid/IronPort etc..)
HTTP::proxy disable
pool proxy_pool
}
}
As more organisations move to Office365, they have been facing similar problems with firewalls and other security devices unable to handle the volume of outbound connections as they move the SaaS world. The easiest solution may have been just to create proxy PAC file and send the traffic direct, but this would have involved allowing clients to directly route via the firewall to those IP address ranges. How secure is that?
I decided to revisit my original solution and look at a way to dynamically update the Office365 URL list. Before I started, I did a quick search on DevCentral and found that DevCentral MVP Niels van Sluis had already written an iRule to download the Microsoft Office365 IP and URL database. Perfect starting point. I’ve since made some modifications to his iRulesLX and a new TCL iRule to support the forward proxy use case.
How the solution works
- The iRuleLX is configured to pull the O365IPAddresses.xml every hour. Reformat into JSON and store in a LokiJS DB.
- The BIG-IP is configured as an Explicit Proxy in the Clients network or browser settings.
- The Virtual Server has a HTTP profile attached with the Proxy Mode set to Explicit along we a few other settings. I will go in the detail later.
- An iRule is attached that executes on the HTTP_PROXY_REQUEST event to check if the FQDN should bypass the Explicit Proxy Pool.
- If the result is not in the Cache, then a lookup is performed in the iRuleLX LokiJS DB for a result. The result is retuned to the iRule to make a decision to bypass or not.
- The bypass result is Cached in a table with a specified timeout.
- A different SNAT pool can be enabled or disabled when bypassing the Explicit Proxy Pool
Configuration
My BIG-IP is running TMOS 13.1 and the iRules Language eXtension has been licensed and provisioned. Make sure your BIG-IP has internet access to download the required Node.JS packages.
This guide also assumes you have a basic level of understanding and troubleshooting at a Local Traffic Manager (LTM) level and your BIG-IP Self IP, VLANs, Routes, etc.. are all configured and working as expected.
Before we get started
The iRule/iRuleLX for this solution can be found on DevCentral Code Share.
- Download and install my Explicit Proxy iApp
- Copy the Intelligent Proxy Steering - Office365 iRule
- Copy the Microsoft Office 365 IP Intelligence - V0.2 iRuleLX
Step 1 – Create the Explicit Proxy
1.1 Run the iApp
iApps >> Application Services >> Applications >> “Create”
Supply the following:
- Name: o365proxy
- Template: f5.explicit_proxy
Explicit Proxy Configuration
- IP Address: 10.1.20.100
- FQDN of this Proxy: o365proxy.f5.demo
- VLAN Configuration - Selected: bigip_int_vlan
- SNAT Mode: Automap
DNS Configuration
- External DNS Resolvers: 1.1.1.1
- Do you need to resolve any Internal DNS zones: Yes or No
Select “Finished" to save.
1.2 Test the forward proxy
$ curl -I https://www.f5.com --proxy http://10.1.20.100:3128
HTTP/1.1 200 Connected
HTTP/1.0 301 Moved Permanently
location: https://f5.com
Server: BigIP
Connection: Keep-Alive
Content-Length: 0
Yep, it works!
1.3 Disable Strict Updates
iApps >> Application Services >> Applications >> o365proxy
- Select the Properties tab, change the Application Service to Advanced.
- Uncheck Strict Updates
Select “Update" to save.
1.4 Add an Explicit Proxy server pool
In my test environment I have a Squid Proxy installed on a Linux host listening on port 3128.
Local Traffic >> Pools >> Pool List >> “Create”
Supply the following:
- Name: squid_proxy_3128_pool
- Node Name: squid_node
- Address: 10.1.30.105
- Service Port: 3128
Select “Add" and “Finished” to Save.
Step 2 – iRule and iRuleLX Configuration
2.1 Create a new iRulesLX workspace
Local Traffic >> iRules >> LX Workspaces >> “Create”
Supply the following:
- Name: office365_ipi_workspace
Select “Finished" to save.
You will now have any empty workspace, ready to cut/paste the TCL iRule and Node.JS code.
2.2 Add the iRule
Select “Add iRule” and supply the following:
- Name: office365_proxy_bypass_irule
- Select OK
Cut / Paste the following Intelligent Proxy Steering - Office365 iRule into the workspace editor on the right hand side. Select “Save File” when done.
2.3 Add an Extension
Select “Add extension” and supply the following:
- Name: office365_ipi_extension
- Select OK
Cut / Paste the following Microsoft Office 365 IP Intelligence - V0.2 iRuleLX and replace the default index.js. Select “Save File” when done.
2.4 Install the NPM packages
- SSH to the BIG-IP as root
- cd /var/ilx/workspaces/Common/office365_ipi_workspace/extensions/office365_ipi_extension/
- npm install xml2js https repeat lokijs ip-range-check --save
2.5 Create a new iRulesLX plugin
Local Traffic >> iRules >> LX Plugin >> “Create”
Supply the following:
- Name: office365_ipi_plugin
- From Workspace: office365_ipi_workspace
Select “Finished" to save.
2.6 Verify the Office365 XML downloaded
SSH to the BIG-IP and tail -f /var/log/ltm
The Office365 XML has been downloaded, parsed and stored in the LokiJS:
big-ip1 info sdmd[5782]: 018e0017:6: pid[9603] plugin[/Common/office365_ipi_plugin.office365_ipi_extension] Info: update finished; 20 product records in database.
2.7 Add the iRule and the Explicit Proxy pool to the Explicit Proxy virtual server
Local Traffic >> Virtual Servers >> Virtual Server List >> o365proxy_3128_vs >> Resources
Edit the following:
- Default Pool: squid_proxy_3128_pool
Select “Update" to save.
Select “Manage…” and move office365_proxy_bypass_irule to the Enabled section.
Select “Finished" to save.
Step 3 – Test the solution
SSH to the BIG-IP and tail -f /var/log/ltm
3.1 Test a non-Office365 URL first
$ curl -I https://www.f5.com --proxy http://10.1.20.100:3128
HTTP/1.1 200 Connected
HTTP/1.0 301 Moved Permanently
location: https://f5.com
Server: BigIP
Connection: Keep-Alive
Content-Length: 0
Output from /var/log/ltm:
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58190 --> 10.1.20.100:3128
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : ## HTTP Proxy Request ##
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : CONNECT www.f5.com:443 HTTP/1.1
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Host: www.f5.com:443
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : User-Agent: curl/7.54.0
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Proxy-Connection: Keep-Alive
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com not in cache - perform DB lookup
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com - bypass: 0
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58190 (10.1.30.245:24363) --> 10.1.30.105:3128
3.2 Test the same non-Office365 URL again to confirm the cache works
Output from /var/log/ltm:
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58487 --> 10.1.20.100:3128
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : ## HTTP Proxy Request ##
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : CONNECT www.f5.com:443 HTTP/1.1
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Host: www.f5.com:443
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : User-Agent: curl/7.54.0
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Proxy-Connection: Keep-Alive
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com found in cache
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com - bypass: 0
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58487 (10.1.30.245:25112) --> 10.1.30.105:3128
3.3 Test a Office365 URL and check it bypasses the Explicit Proxy pool
$ curl -I https://www.outlook.com --proxy http://10.1.20.100:3128
HTTP/1.1 200 Connected
HTTP/1.1 301 Moved Permanently
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 0
Location: https://outlook.live.com/
Server: Microsoft-IIS/10.0
Connection: close
Output from /var/log/ltm:
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58692 --> 10.1.20.100:3128
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : ## HTTP Proxy Request ##
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : CONNECT www.outlook.com:443 HTTP/1.1
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Host: www.outlook.com:443
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : User-Agent: curl/7.54.0
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Proxy-Connection: Keep-Alive
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.outlook.com not in cache - perform DB lookup
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.outlook.com - bypass: 1
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58692 (10.1.10.245:21666) --> 40.100.144.226:443
It works!
Conclusion
By combining Microsoft Office365 IP and URL intelligence with LTM, produces a simple and effective method to steer around overloaded Forward Proxy servers without the hassle of messy proxy PAC files.
Published Apr 20, 2018
Version 1.0
Smithy
Cirrostratus
CirrostratusSmithyCirrostratus
Cirrostratus
SmithyI just use "crontab -e" to add the cron and "crontab -l" shows the entry for me. I did this as root on my test platform.
AdamCCNimbostratus
Hi,
Thanks for sharing the solution. The script does a great job setting up he url category, this is a huge time saver. Only item I am stuck on is the setup of the cron job. I implemented your job in the code share but when I ran either crontab -l or ls -l /etc/cron.* I do not see the job listed. How can I confirm the job is set?
- Smithy
Hi ,
Here's the link to the Python script. I'm going to update the post and re-write the solution to reference this. https://github.com/brett-at-f5/f5-office365-ip-url-automation. Use the iRule in the GitHub repo instead.
Enjoy!
Cheers,
Brett
- AdamCC
Hi
Just wanted to follow up on the alternate solution. I didn't want to bother you during the holidays.
Thanks
Adam
- AdamCC
Brett,
That would be great.
- Smithy
Hi ,
There is an alternate solution that uses a Python script to download the O365 URLs, IPv4 and IPv6 and creates a custom URL category for APM/SWG. Let me check with the author to see if I can release to DC or GitHub. It's a better solution, as it creates the URL category and/or Datagroup.
- Smithy
Hi
I haven't changed the JS code at all and mine works fine. Also upgraded to 14.1.2.1 and works as expected.
Cheers,
Brett
- AdamCC
Hi Nico-Brett
I to have SWG as forward proxy and I am wondering how your implementation is going. Based on your post seems like you were pointed into using SWG Office category. Based on your following post it looks like you are attempting to implement the download of the Office365 IP and URL database. Were you not able to use the Office category.
Do you have a solution working for SWG? Can you provide more info, maybe a SWG specific post. I am hoping you can spare me some trial and error.
Guy_iHealthI changed const to var, replaced '=>' with function and let by var, to go back to the older syntax.
Now the script passes but it hit's again a 'let' in the imported modules ... the node.js of F5 is not able to use the new syntax I guess.
I'm using a licensed version of a virtual F5 with version 13.1.3.1 which is more or less the same as your used version.
So I don't see what's the difference.
- Guy_iHealth
Yes I'm using version 0.4 from the same location you indicated above.