Technical Articles
F5 SMEs share good practice.
cancel
Showing results for 
Search instead for 
Did you mean: 
Custom Alert Banner
Smithy
Cirrostratus
Cirrostratus

Introduction

This solution started back in May 2015 when I was helping a customer bypass their forward proxy servers due to the significant increase in the number of client connections after moving to Office365. Luckily they had a BIG-IP in front of their forward proxy servers load balancing the traffic and F5 had introduced a new “Proxy Mode” feature in the HTTP profile in TMOS 11.5. This allowed the BIG-IP to terminate Explicit Proxy connections, instead of passing them through to the pool members. The original solution was a simple iRule that referenced a data-group to determine if the connection should bypass the forward proxy pool or reverse proxy and load balance the connection as normal.

Original iRule:

when HTTP_PROXY_REQUEST {
    # Strip of the port number
    set hostname [lindex [split [HTTP::host] ":"] 0]
    # If the hostname matches a MS 0ffice365 domain, enable the Forward Proxy on BIG-IP.
    # BIG-IP will then perform a DNS lookup and act as a Forward Proxy bypassing the Forward Proxy
    # Server Pool (BlueCoat/Squid/IronPort etc..)
    if { [class match $hostname ends_with o365_datagroup] } {
        # Use a SNAT pool - recommended
        snatpool o365_snatpool
        HTTP::proxy
    } else {
        # Load balance/reverse proxy to the Forward Proxy Server Pool (BlueCoat/Squid/IronPort etc..)
        HTTP::proxy disable
        pool proxy_pool
    }
}

As more organisations move to Office365, they have been facing similar problems with firewalls and other security devices unable to handle the volume of outbound connections as they move the SaaS world. The easiest solution may have been just to create proxy PAC file and send the traffic direct, but this would have involved allowing clients to directly route via the firewall to those IP address ranges. How secure is that?

I decided to revisit my original solution and look at a way to dynamically update the Office365 URL list. Before I started, I did a quick search on DevCentral and found that DevCentral MVP Niels van Sluis had already written an iRule to download the Microsoft Office365 IP and URL database. Perfect starting point. I’ve since made some modifications to his iRulesLX and a new TCL iRule to support the forward proxy use case.

How the solution works

  • The iRuleLX is configured to pull the O365IPAddresses.xml every hour. Reformat into JSON and store in a LokiJS DB.
  • The BIG-IP is configured as an Explicit Proxy in the Clients network or browser settings.
  • The Virtual Server has a HTTP profile attached with the Proxy Mode set to Explicit along we a few other settings. I will go in the detail later.
  • An iRule is attached that executes on the HTTP_PROXY_REQUEST event to check if the FQDN should bypass the Explicit Proxy Pool.
  • If the result is not in the Cache, then a lookup is performed in the iRuleLX LokiJS DB for a result. The result is retuned to the iRule to make a decision to bypass or not.
  • The bypass result is Cached in a table with a specified timeout.
  • A different SNAT pool can be enabled or disabled when bypassing the Explicit Proxy Pool

0151T000003d7BWQAY.png

 

Configuration

My BIG-IP is running TMOS 13.1 and the iRules Language eXtension has been licensed and provisioned. Make sure your BIG-IP has internet access to download the required Node.JS packages.

This guide also assumes you have a basic level of understanding and troubleshooting at a Local Traffic Manager (LTM) level and your BIG-IP Self IP, VLANs, Routes, etc.. are all configured and working as expected.

Before we get started

The iRule/iRuleLX for this solution can be found on DevCentral Code Share.

Step 1 – Create the Explicit Proxy

1.1 Run the iApp

iApps >> Application Services >> Applications >> “Create”

Supply the following:

  • Name: o365proxy
  • Template: f5.explicit_proxy

Explicit Proxy Configuration

  • IP Address: 10.1.20.100
  • FQDN of this Proxy: o365proxy.f5.demo
  • VLAN Configuration - Selected: bigip_int_vlan
  • SNAT Mode: Automap

DNS Configuration

  • External DNS Resolvers: 1.1.1.1
  • Do you need to resolve any Internal DNS zones: Yes or No

Select “Finished" to save.

0151T000003d7BXQAY.png

1.2 Test the forward proxy

$ curl -I https://www.f5.com --proxy http://10.1.20.100:3128
HTTP/1.1 200 Connected

HTTP/1.0 301 Moved Permanently
location:
https://f5.com
Server: BigIP
Connection: Keep-Alive
Content-Length: 0

Yep, it works!

1.3 Disable Strict Updates

iApps >> Application Services >> Applications >> o365proxy

  • Select the Properties tab, change the Application Service to Advanced.
  • Uncheck Strict Updates

Select “Update" to save.

0151T000003d7BYQAY.png

1.4 Add an Explicit Proxy server pool

In my test environment I have a Squid Proxy installed on a Linux host listening on port 3128.

Local Traffic >> Pools >> Pool List >> “Create”

Supply the following:

  • Name: squid_proxy_3128_pool
  • Node Name: squid_node
  • Address: 10.1.30.105
  • Service Port: 3128

Select “Add" and “Finished” to Save.

0151T000003d7BZQAY.png

Step 2 – iRule and iRuleLX Configuration

2.1 Create a new iRulesLX workspace

Local Traffic >> iRules >> LX Workspaces >> “Create”

Supply the following:

  • Name: office365_ipi_workspace

Select “Finished" to save.

You will now have any empty workspace, ready to cut/paste the TCL iRule and Node.JS code.

2.2 Add the iRule

Select “Add iRule” and supply the following:

  • Name: office365_proxy_bypass_irule
  • Select OK

Cut / Paste the following Intelligent Proxy Steering - Office365 iRule into the workspace editor on the right hand side. Select “Save File” when done.

0151T000003d7BaQAI.png

2.3 Add an Extension

Select “Add extension” and supply the following:

  • Name: office365_ipi_extension
  • Select OK

Cut / Paste the following Microsoft Office 365 IP Intelligence - V0.2 iRuleLX and replace the default index.js. Select “Save File” when done.

0151T000003d7BbQAI.png

2.4 Install the NPM packages

  • SSH to the BIG-IP as root
  • cd /var/ilx/workspaces/Common/office365_ipi_workspace/extensions/office365_ipi_extension/
  • npm install xml2js https repeat lokijs ip-range-check --save

2.5 Create a new iRulesLX plugin

Local Traffic >> iRules >> LX Plugin >> “Create”

Supply the following:

  • Name: office365_ipi_plugin
  • From Workspace: office365_ipi_workspace

Select “Finished" to save.

0151T000003d7BcQAI.png

2.6 Verify the  Office365 XML downloaded

SSH to the BIG-IP and tail -f /var/log/ltm

The Office365 XML has been downloaded, parsed and stored in the LokiJS:
big-ip1 info sdmd[5782]: 018e0017:6: pid[9603]  plugin[/Common/office365_ipi_plugin.office365_ipi_extension] Info: update finished; 20 product records in database.

2.7 Add the iRule and the Explicit Proxy pool to the Explicit Proxy virtual server

Local Traffic >> Virtual Servers >> Virtual Server List >> o365proxy_3128_vs >> Resources

Edit the following:

  • Default Pool: squid_proxy_3128_pool

Select “Update" to save.

Select “Manage…” and move office365_proxy_bypass_irule to the Enabled section.

Select “Finished" to save.

0151T000003d7BdQAI.png

Step 3 – Test the solution

SSH to the BIG-IP and tail -f /var/log/ltm

3.1 Test a non-Office365 URL first

$ curl -I https://www.f5.com --proxy http://10.1.20.100:3128
HTTP/1.1 200 Connected

HTTP/1.0 301 Moved Permanently
location:
https://f5.com
Server: BigIP
Connection: Keep-Alive
Content-Length: 0

Output from /var/log/ltm:

big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58190 --> 10.1.20.100:3128
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : ## HTTP Proxy Request ##
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : CONNECT www.f5.com:443 HTTP/1.1
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Host: www.f5.com:443
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : User-Agent: curl/7.54.0
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Proxy-Connection: Keep-Alive
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com not in cache - perform DB lookup
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com - bypass: 0
big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58190 (10.1.30.245:24363) --> 10.1.30.105:3128

3.2 Test the same non-Office365 URL again to confirm the cache works

Output from /var/log/ltm:

big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58487 --> 10.1.20.100:3128
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : ## HTTP Proxy Request ##
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : CONNECT www.f5.com:443 HTTP/1.1
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Host: www.f5.com:443
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : User-Agent: curl/7.54.0
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Proxy-Connection: Keep-Alive
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com found in cache
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com - bypass: 0
big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58487 (10.1.30.245:25112) --> 10.1.30.105:3128

3.3 Test a Office365 URL and check it bypasses the Explicit Proxy pool

$ curl -I https://www.outlook.com  --proxy http://10.1.20.100:3128
HTTP/1.1 200 Connected

HTTP/1.1 301 Moved Permanently
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 0
Location:
https://outlook.live.com/
Server: Microsoft-IIS/10.0
Connection: close

Output from /var/log/ltm:

big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58692 --> 10.1.20.100:3128
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : ## HTTP Proxy Request ##
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : CONNECT www.outlook.com:443 HTTP/1.1
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Host: www.outlook.com:443
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : User-Agent: curl/7.54.0
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Proxy-Connection: Keep-Alive
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.outlook.com not in cache - perform DB lookup
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.outlook.com - bypass: 1
big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58692 (10.1.10.245:21666) --> 40.100.144.226:443

It works!

Conclusion

By combining Microsoft Office365 IP and URL intelligence with LTM, produces a simple and effective method to steer around overloaded Forward Proxy servers without the hassle of messy proxy PAC files.

Comments
Jayvee_359603
Altostratus
Altostratus

0691T000006AqpyQAC.png

 

Hi Brett,

 

Thanks for this article. I followed your guide and already integrated the iRule to the VS, but i got this error when I used it as my proxy. Any idea why this came up? I checked the LX plugin but the extension is there.

 

Thanks!

 

Smithy
Cirrostratus
Cirrostratus

Hi Jayvee,

It's hard to tell what has been miss configured based on the log file above. Check the spelling of the extension within the workspace and check it has been loaded in /var/log/ltm. You should see an entry similar to this:

Aug  2 21:02:17 big-ip1 info sdmd[16742]: 018e0017:6: pid[12328]  plugin[/Common/office365_ipi_plugin.office365_ipi_extension] Info: update finished; 20 product records in database.
Aug  2 21:02:17 big-ip1 info sdmd[16742]: 018e0017:6: pid[12329]  plugin[/Common/office365_ipi_plugin.office365_ipi_extension] Info: update finished; 20 product records in database.
Aug  2 21:02:17 big-ip1 info sdmd[16742]: 018e0017:6: pid[12330]  plugin[/Common/office365_ipi_plugin.office365_ipi_extension] Info: update finished; 20 product records in database.
Aug  2 21:02:17 big-ip1 info sdmd[16742]: 018e0017:6: pid[12327]  plugin[/Common/office365_ipi_plugin.office365_ipi_extension] Info: update finished; 20 product records in database.

Cheers, Brett

vu_tien_cuong
Nimbostratus
Nimbostratus

Hi Jayvee, You can try to reload the workspace from Local Traffic ›› iRules : LX Plugins ›› office365_ipi_plugin I did try and it worked.

 

Rgds Cuong

 

Jayvee_359603
Altostratus
Altostratus

Hi Cuong,

 

Will definitely try your workaround. Thank you for the assistance!

 

f5support_at_sf
Nimbostratus
Nimbostratus

Hi Brett,

 

I work with vCMP environement, the test to forward proxy doesn't work from vcmp guest but work in vcmp host. Can you help me please.

 

rgds, Hedi.

 

f5support_at_sf
Nimbostratus
Nimbostratus

Hi Brett,

 

Sep 7 12:40:27 slot1/pr-cha-px-o365-01 err sdmd[6245]: 018e0011:3: Received sigchld for unknown pid 12402 Sep 7 12:40:27 slot1/pr-cha-px-o365-01 err sdmd[6245]: 018e0018:3: pid[12416] plugin[/Common/office365_ipi_plugin.office365_ipi_extension] /var/sdm/plugin_store/plugins/:Common:office365_ipi_plugin_44198_17/extensions/office365_ipi_extension/index.js:150 Sep 7 12:40:27 slot1/pr-cha-px-o365-01 err sdmd[6245]: 018e0018:3: pid[12416] plugin[/Common/office365_ipi_plugin.office365_ipi_extension] for (let url of req.urls) { Sep 7 12:40:27 slot1/pr-cha-px-o365-01 err sdmd[6245]: 018e0018:3: pid[12416] plugin[/Common/office365_ipi_plugin.office365_ipi_extension] ^^^ Sep 7 12:40:27 slot1/pr-cha-px-o365-01 err sdmd[6245]: 018e0018:3: pid[12416] plugin[/Common/office365_ipi_plugin.office365_ipi_extension] SyntaxError: Unexpected strict mode reserved word

 

I have un syntax error when I load plugin from workplace. Have you an idea?

 

Thanks, Hedi.

 

AP
Nimbostratus
Nimbostratus

Hi Jayvee,

 

Did you end up finding a solution to your error? I'm getting the same thing and naming is all correct.

 

Thanks, Andrew

 

Smithy
Cirrostratus
Cirrostratus

If you are getting the following error: "Could not find ILX extension office365_ipi_extension in path office365_ipi_plugin", make sure you have installed all the NPM packages.

Go back and re-run this at root:

$ cd /var/ilx/workspaces/Common/office365_ipi_workspace/extensions/office365_ipi_extension/

$ npm install xml2js https repeat lokijs ip-range-check –save

Make sure you get this:

/var/ilx/workspaces/Common/office365_ipi_workspace/extensions/office365_ipi_extension
    ├── https@1.0.0
    ├─┬ ip-range-check@0.0.2
    │ └── ipaddr.js@1.8.1
    ├── lokijs@1.5.5
    ├── repeat@0.0.6
    └── xml2js@0.4.19

Check /var/log/ltm for any runtime compile errors.

Smithy
Cirrostratus
Cirrostratus

Niels van Sluis, DevCentral MVP has re-written the Microsoft Office 365 IP Intelligence iRuleLX to support the new REST API: https://github.com/nvansluis/f5.office365_endpoints_extension

 

Arnaud_Lemaire
F5 Employee
F5 Employee

with the new ilx from Niels the node module instalmation must be completed with : npm uuid install unique-concat --save

 

Zeeshan_377314
Nimbostratus
Nimbostratus

Trying to run new node.js but seems unable to run the module. Any idea about below error ...

 

[root@localhost:Active:Standalone] office365_ipi_extension npm install xml2js https repeat lokijs ip-range-check --save npm ERR! Linux 3.10.0-514.26.2.el7.ve.x86_64 npm ERR! argv "/usr/bin/node" "/usr/bin/.npm__" "install" "xml2js" "https" "repeat" "lokijs" "ip-range-check" "--save" npm ERR! node v6.9.1 npm ERR! npm v3.10.8 npm ERR! code EAI_AGAIN npm ERR! errno EAI_AGAIN npm ERR! syscall getaddrinfo

 

npm ERR! getaddrinfo EAI_AGAIN registry.npmjs.org:443 npm ERR! npm ERR! If you need help, you may report this error at: npm ERR! https://github.com/npm/npm/issues

 

npm ERR! Please include the following file with any support request: npm ERR! /var/ilx/workspaces/Common/office365_ipi_workspace/extensions/office365_ipi_extension/npm-debug.log

 

Arnaud_Lemaire
F5 Employee
F5 Employee

npm ERR! getaddrinfo EAI_AGAIN registry.npmjs.org:443 seems to be related to dns resolution error while trying to fetch packages. are you on a laptop vm ? if yes try to change your network.

 

Zeeshan_377314
Nimbostratus
Nimbostratus

Thanks Arnaud for your response, i am not using laptop VM. Its currently in ESXi and reachable to internet.

 

[root@localhost:Active:Standalone] config ping registry.npmjs.org PING registry.npmjs.org (104.16.19.35) 56(84) bytes of data. 64 bytes from 104.16.19.35: icmp_seq=1 ttl=58 time=12.0 ms

 

[root@localhost:Active:Standalone] config curl -I https://registry.npmjs.org HTTP/1.1 200 OK Date: Thu, 22 Nov 2018 00:45:39 GMT Content-Type: text/plain Connection: keep-alive Set-Cookie: __cfduid=d02c4a77d1fa0afc39e41055cdd5834cc1542847539; expires=Fri, 22-Nov-19 00:45:39 GMT; path=/; domain=.registry.npmjs.org; HttpOnly Cache-Control: must-revalidate CF-RAY: 47d75fdf59e9a9a8-SIN Expect-CT: max-age=604800, report-uri="; Server: cloudflare

 

Zeeshan_377314
Nimbostratus
Nimbostratus

Managed to successfully loaded but still receiving a similar error. ILX Plugin also stopped.

 

[root@localhost:ModuleNotLicensed::Active:Standalone] office365_ipi_extension npm install xml2js https repeat lokijs ip-range-check --save npm WARN deprecated repeat@0.0.6: Not maintained anymore. Package name available! npm WARN deprecated dfrrd@0.2.0: Not maintained anymore. Promises has been standardized by ECMAScript. Use native promise / polyfill instead. /var/ilx/workspaces/Common/office365_ipi_workspace/extensions/office365_ipi_extension ├── https@1.0.0 ├─┬ ip-range-check@0.0.2 │ └── ipaddr.js@1.8.1 ├── lokijs@1.5.5 ├─┬ repeat@0.0.6 │ └── dfrrd@0.2.0 └─┬ xml2js@0.4.19 ├── sax@1.2.4 └── xmlbuilder@9.0.7

 

TCL error: /Common/office365_ipi_plugin/office365_proxy_bypass_irule - Could not find ILX extension office365_ipi_extension in path office365_ipi_plugin (line 25) invoked from within "ILX::init office365_ipi_plugin office365_ipi_extension"

 

AP
Nimbostratus
Nimbostratus

Hi Zeeshan,

I note you've included xmlbuilder. You'll want to use the newer version of the ILX iRule (https://github.com/nvansluis/f5.office365_endpoints_extension). Microsoft has deprecated the XML service and replaced with a REST service.

Try rebuilding your workspace with the new Node iRule and import the required packages (they've changed) and see how you go following that.

Also, in the TCL iRule, I've added a catch to gracefully handle ILX::init failures for when an Extension termination occurs. Like so:

   if { [catch { set rpc_handle [ILX::init office365_ipi_DEV_plugin office365_ipi_extension] } ] } {
        log local0.error  "Host - $host, ILX::init failure: The ILX Extension may have been terminated."
        return
    }
Smithy
Cirrostratus
Cirrostratus

Thanks Andrew. I've update the iRule on CodeShare

 

Sabeer_Ali_2389
Altostratus
Altostratus

We tried this solution , but all traffic are going directly, nothing is is passing via proxy (pool)

 

AP
Nimbostratus
Nimbostratus

Hi Sabeer,

 

What version are you running? Are you seeing any errors in the LTM logs?

 

Did you try adding some logging to see which conditions and actions are being executed for your test requests?

 

The above solution does work, so it's possible you have missed something along the way.

 

gooby_135305
Historic F5 Account

Note. The repeat package needs to be at version 0.0.6, exactly as per the image. If you simply use

 

npm install repeat --save

 

...you get a later version that results in a barf. Instead use

 

npm install repeat@0.0.6 -- save

 

Nico_Amatorio
Nimbostratus
Nimbostratus

Hello Guys,

 

Upon executing tail -f /var/log/ltm we got the error below.

0691T000006B3jPQAS.png

 

Also, do the setup requires an HA for configsync?

 

Thanks in advance guys!

 

 

Jagadeeswaran
Nimbostratus
Nimbostratus

Hi Brett,

 

Receiving below error when reloading the workspace. (npm installed, https repeat@0.0.6).

 

Is it something specific to DNS or using different partition/RD.

 

-- info sdmd[14315]: 018e0017:6: pid[31621] plugin[/Internet_Partition/office365_ipi_plugin.office365_ipi_extension] Got error: getaddrinfo EAI_AGAIN endpoints.office.com:443

 

Host - smtp.office365.com, ILX failure: ILX timeout.   invoked from within "ILX::call $rpc_handle checkProductURL $o365_product $host"

 

Thanks in advance.

Nico_Amatorio
Nimbostratus
Nimbostratus

Hello @Okta Service Team​  ,

 

If I have SWG license, do I still need to use the iRule LX- NodeJS for the O365 URL Lookup?

What are the changes in the procedures for the Proxy bypass?

 

Thank you!

Arnaud_Lemaire
F5 Employee
F5 Employee

Hello Nico, no you don't have to implement that. SWG or url filtering license are good alternatives. Both of them should have a category for office, thus you can match it to steer it.

Nico_Amatorio
Nimbostratus
Nimbostratus

Hi  

 

Then how can i implement the proxy bypass using SWG/URL Filtering if my traffic is going to O365?

Arnaud_Lemaire
F5 Employee
F5 Employee
You can create can create an irule matching office category and deactivating apm for this traffic keeping proy only dns resolution. But normally you should do it before your proxy architecture, to move away your traffic from proxy architecture. If your bigip is your proxy, then you will still have the load, but you can disable features based on category.
Nico_Amatorio
Nimbostratus
Nimbostratus

 Yes, Our proxy is the BIGIP. How can we call the Office category of SWG? Is there any existing datagroup for O365 that we can use in the iRule?0691T000006B7fvQAC.png

Nico_Amatorio
Nimbostratus
Nimbostratus

 

Hello  ,

 

Is it necessary to have the same value as these below.?

 

  1. ke sure you get this:
  2.  
  3. /var/ilx/workspaces/Common/office365_ipi_workspace/extensions/office365_ipi_extension
  4. ├── https@1.0.0
  5. ├─┬ ip-range-check@0.0.2
  6. │ └── ipaddr.js@1.8.1
  7. ├── lokijs@1.5.5
  8. ├── repeat@0.0.6
  9. └── xml2js@0.4.19

 

We are getting these values below.

0691T000005lpJhQAI.jpg

AP
Nimbostratus
Nimbostratus

Hi Nico,

 

Take a look at the newer version (0.4) of the iRule: https://github.com/nvansluis/f5.office365_endpoints_extension

 

The iRule covered in this thread is for the old XML based API and the new is a REST API. I believe the XML API was going to be decommissioned earlier this year. The xml2js package (for example) is no longer relevant.

 

In terms of package versions, there will have been updates since the article was written. Should be fine but of course there's no guarantees something hasn't broken. You can always try to get the older packages if necessary.

Guy_iHealth
Nimbostratus
Nimbostratus

Hello Brett,

 

I've tried to setup your solution as a test on a virtual F5.

I started with version 12.1.3 but because I got a syntax error unexpected token => when I try to load the plugin I upgraded to version 13.1.3.1 because I saw you used version 13.

But I still have the same error.

In the workspace no problem is mentioned, I can save the index.js but when the plugin is reloaded I see the following in the ltm log :

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst] /var/sdm/plugin_store/plugins/:Common:lxpi_o365_tst_38969_15/extensions/lxex_o365_tst/index.js:75

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst]   products.find().forEach((v, i) => {

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst]                  ^^

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst] SyntaxError: Unexpected token =>

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst]   at exports.runInThisContext (vm.js:73:16)

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst]   at Module._compile (module.js:443:25)

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst]   at Object.Module._extensions..js (module.js:478:10)

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst]   at Module.load (module.js:355:32)

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst]   at Function.Module._load (module.js:310:12)

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst]   at Function.Module.runMain (module.js:501:10)

Dec 11 05:46:20 vlbintra err sdmd[23542]: 018e0018:3: pid[14104] plugin[/Common/lxpi_o365_tst.lxex_o365_tst]   at startup (node.js:129:16)

I'm not familiar with Javascript but this seams a valid syntax to me :

function logStatistics() {

   

  // statistics about entries in loki database.

  var v, i;

  console.log('info: ' + products.count() + ' serviceAreas found.');

  products.find().forEach((v, i) => {

    console.log ('info: serviceArea ' + v.name + ' holds ' + v.ipAddresses.length + ' IP addresses and ' + v.urls.length + ' URLs');

  });

}

Am I overlooking something ?

 

Thanks in advance.

Best regards,

Guy

Guy_iHealth
Nimbostratus
Nimbostratus

I forgot to mention I commented 'use strict' because the code stopped executing when arriving on line const uuidv4 = require('uuid/v4');

Smithy
Cirrostratus
Cirrostratus

Are you using the new version? Check out: https://github.com/nvansluis/f5.office365_endpoints_extension

Guy_iHealth
Nimbostratus
Nimbostratus

Yes I'm using version 0.4 from the same location you indicated above.

Guy_iHealth
Nimbostratus
Nimbostratus

I changed const to var, replaced '=>' with function and let by var, to go back to the older syntax.

Now the script passes but it hit's again a 'let' in the imported modules ... the node.js of F5 is not able to use the new syntax I guess.

I'm using a licensed version of a virtual F5 with version 13.1.3.1 which is more or less the same as your used version.

So I don't see what's the difference.

AdamCC
Nimbostratus
Nimbostratus

Hi Nico-Brett

 

I to have SWG as forward proxy and I am wondering how your implementation is going. Based on your post seems like you were pointed into using SWG Office category. Based on your following post it looks like you are attempting to implement the download of the Office365 IP and URL database. Were you not able to use the Office category.

 

Do you have a solution working for SWG? Can you provide more info, maybe a SWG specific post. I am hoping you can spare me some trial and error.

Smithy
Cirrostratus
Cirrostratus

Hi  

 

I haven't changed the JS code at all and mine works fine. Also upgraded to 14.1.2.1 and works as expected.

 

Cheers,

 

Brett

Smithy
Cirrostratus
Cirrostratus

Hi  ,

 

There is an alternate solution that uses a Python script to download the O365 URLs, IPv4 and IPv6 and creates a custom URL category for APM/SWG. Let me check with the author to see if I can release to DC or GitHub. It's a better solution, as it creates the URL category and/or Datagroup.

AdamCC
Nimbostratus
Nimbostratus

Brett,

 

That would be great.

AdamCC
Nimbostratus
Nimbostratus

Hi  

 

Just wanted to follow up on the alternate solution. I didn't want to bother you during the holidays.

 

Thanks

Adam

Smithy
Cirrostratus
Cirrostratus

Hi  ,

 

Here's the link to the Python script. I'm going to update the post and re-write the solution to reference this. https://github.com/brett-at-f5/f5-office365-ip-url-automation. Use the iRule in the GitHub repo instead.

 

Enjoy!

 

Cheers,

 

Brett

AdamCC
Nimbostratus
Nimbostratus

Hi,

 

Thanks for sharing the solution. The script does a great job setting up he url category, this is a huge time saver. Only item I am stuck on is the setup of the cron job. I implemented your job in the code share but when I ran either crontab -l or ls -l /etc/cron.* I do not see the job listed. How can I confirm the job is set?

Smithy
Cirrostratus
Cirrostratus

I just use "crontab -e" to add the cron and "crontab -l" shows the entry for me. I did this as root on my test platform.

Version history
Last update:
‎20-Apr-2018 03:08
Updated by:
Contributors