Enabling alarm and block of IP Intelligence through CLI and curl
Is there an API for IP Intelligence enabling and disabling as well as for enabling alerting and/or blocking of items within the IP Intelligence feature in ASM. I am wanting to run a curl command like I do for IP Whitelisting and stuff to make a mass update of all ASM policies to enable IP Intelligence on all of them in one go. Please advise.
Intelligent Proxy Steering - Office365
Introduction This solution started back in May 2015 when I was helping a customer bypass their forward proxy servers due to the significant increase in the number of client connections after moving to Office365. Luckily they had a BIG-IP in front of their forward proxy servers load balancing the traffic and F5 had introduced a new “Proxy Mode” feature in the HTTP profile in TMOS 11.5. This allowed the BIG-IP to terminate Explicit Proxy connections, instead of passing them through to the pool members. The original solution was a simple iRule that referenced a data-group to determine if the connection should bypass the forward proxy pool or reverse proxy and load balance the connection as normal. Original iRule: when HTTP_PROXY_REQUEST { # Strip of the port number set hostname [lindex [split [HTTP::host] ":"] 0] # If the hostname matches a MS 0ffice365 domain, enable the Forward Proxy on BIG-IP. # BIG-IP will then perform a DNS lookup and act as a Forward Proxy bypassing the Forward Proxy # Server Pool (BlueCoat/Squid/IronPort etc..) if { [class match $hostname ends_with o365_datagroup] } { # Use a SNAT pool - recommended snatpool o365_snatpool HTTP::proxy } else { # Load balance/reverse proxy to the Forward Proxy Server Pool (BlueCoat/Squid/IronPort etc..) HTTP::proxy disable pool proxy_pool } } As more organisations move to Office365, they have been facing similar problems with firewalls and other security devices unable to handle the volume of outbound connections as they move the SaaS world. The easiest solution may have been just to create proxy PAC file and send the traffic direct, but this would have involved allowing clients to directly route via the firewall to those IP address ranges. How secure is that? I decided to revisit my original solution and look at a way to dynamically update the Office365 URL list. Before I started, I did a quick search on DevCentral and found that DevCentral MVP Niels van Sluis had already written an iRule to download the Microsoft Office365 IP and URL database. Perfect starting point. I’ve since made some modifications to his iRulesLX and a new TCL iRule to support the forward proxy use case. How the solution works The iRuleLX is configured to pull the O365IPAddresses.xml every hour. Reformat into JSON and store in a LokiJS DB. The BIG-IP is configured as an Explicit Proxy in the Clients network or browser settings. The Virtual Server has a HTTP profile attached with the Proxy Mode set to Explicit along we a few other settings. I will go in the detail later. An iRule is attached that executes on the HTTP_PROXY_REQUEST event to check if the FQDN should bypass the Explicit Proxy Pool. If the result is not in the Cache, then a lookup is performed in the iRuleLX LokiJS DB for a result. The result is retuned to the iRule to make a decision to bypass or not. The bypass result is Cached in a table with a specified timeout. A different SNAT pool can be enabled or disabled when bypassing the Explicit Proxy Pool Configuration My BIG-IP is running TMOS 13.1 and the iRules Language eXtension has been licensed and provisioned. Make sure your BIG-IP has internet access to download the required Node.JS packages. This guide also assumes you have a basic level of understanding and troubleshooting at a Local Traffic Manager (LTM) level and your BIG-IP Self IP, VLANs, Routes, etc.. are all configured and working as expected. Before we get started The iRule/iRuleLX for this solution can be found on DevCentral Code Share. Download and install my Explicit Proxy iApp Copy the Intelligent Proxy Steering - Office365 iRule Copy the Microsoft Office 365 IP Intelligence - V0.2 iRuleLX Step 1 – Create the Explicit Proxy 1.1 Run the iApp iApps >> Application Services >> Applications >> “Create” Supply the following: Name: o365proxy Template: f5.explicit_proxy Explicit Proxy Configuration IP Address: 10.1.20.100 FQDN of this Proxy: o365proxy.f5.demo VLAN Configuration - Selected: bigip_int_vlan SNAT Mode: Automap DNS Configuration External DNS Resolvers: 1.1.1.1 Do you need to resolve any Internal DNS zones: Yes or No Select “Finished" to save. 1.2 Test the forward proxy $ curl -I https://www.f5.com --proxy http://10.1.20.100:3128 HTTP/1.1 200 Connected HTTP/1.0 301 Moved Permanently location: https://f5.com Server: BigIP Connection: Keep-Alive Content-Length: 0 Yep, it works! 1.3 Disable Strict Updates iApps >> Application Services >> Applications >> o365proxy Select the Properties tab, change the Application Service to Advanced. Uncheck Strict Updates Select “Update" to save. 1.4 Add an Explicit Proxy server pool In my test environment I have a Squid Proxy installed on a Linux host listening on port 3128. Local Traffic >> Pools >> Pool List >> “Create” Supply the following: Name: squid_proxy_3128_pool Node Name: squid_node Address: 10.1.30.105 Service Port: 3128 Select “Add" and “Finished” to Save. Step 2 – iRule and iRuleLX Configuration 2.1 Create a new iRulesLX workspace Local Traffic >> iRules >> LX Workspaces >> “Create” Supply the following: Name: office365_ipi_workspace Select “Finished" to save. You will now have any empty workspace, ready to cut/paste the TCL iRule and Node.JS code. 2.2 Add the iRule Select “Add iRule” and supply the following: Name: office365_proxy_bypass_irule Select OK Cut / Paste the following Intelligent Proxy Steering - Office365 iRule into the workspace editor on the right hand side. Select “Save File” when done. 2.3 Add an Extension Select “Add extension” and supply the following: Name: office365_ipi_extension Select OK Cut / Paste the following Microsoft Office 365 IP Intelligence - V0.2 iRuleLX and replace the default index.js. Select “Save File” when done. 2.4 Install the NPM packages SSH to the BIG-IP as root cd /var/ilx/workspaces/Common/office365_ipi_workspace/extensions/office365_ipi_extension/ npm install xml2js https repeat lokijs ip-range-check --save 2.5 Create a new iRulesLX plugin Local Traffic >> iRules >> LX Plugin >> “Create” Supply the following: Name: office365_ipi_plugin From Workspace: office365_ipi_workspace Select “Finished" to save. 2.6 Verify the Office365 XML downloaded SSH to the BIG-IP and tail -f /var/log/ltm The Office365 XML has been downloaded, parsed and stored in the LokiJS: big-ip1 info sdmd[5782]: 018e0017:6: pid[9603] plugin[/Common/office365_ipi_plugin.office365_ipi_extension] Info: update finished; 20 product records in database. 2.7 Add the iRule and the Explicit Proxy pool to the Explicit Proxy virtual server Local Traffic >> Virtual Servers >> Virtual Server List >> o365proxy_3128_vs >> Resources Edit the following: Default Pool: squid_proxy_3128_pool Select “Update" to save. Select “Manage…” and move office365_proxy_bypass_irule to the Enabled section. Select “Finished" to save. Step 3 – Test the solution SSH to the BIG-IP and tail -f /var/log/ltm 3.1 Test a non-Office365 URL first $ curl -I https://www.f5.com --proxy http://10.1.20.100:3128 HTTP/1.1 200 Connected HTTP/1.0 301 Moved Permanently location: https://f5.com Server: BigIP Connection: Keep-Alive Content-Length: 0 Output from /var/log/ltm: big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58190 --> 10.1.20.100:3128 big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : ## HTTP Proxy Request ## big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : CONNECT www.f5.com:443 HTTP/1.1 big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Host: www.f5.com:443 big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : User-Agent: curl/7.54.0 big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Proxy-Connection: Keep-Alive big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com not in cache - perform DB lookup big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com - bypass: 0 big-ip1 info tmm2[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58190 (10.1.30.245:24363) --> 10.1.30.105:3128 3.2 Test the same non-Office365 URL again to confirm the cache works Output from /var/log/ltm: big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58487 --> 10.1.20.100:3128 big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : ## HTTP Proxy Request ## big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : CONNECT www.f5.com:443 HTTP/1.1 big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Host: www.f5.com:443 big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : User-Agent: curl/7.54.0 big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Proxy-Connection: Keep-Alive big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com found in cache big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.f5.com - bypass: 0 big-ip1 info tmm1[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58487 (10.1.30.245:25112) --> 10.1.30.105:3128 3.3 Test a Office365 URL and check it bypasses the Explicit Proxy pool $ curl -I https://www.outlook.com --proxy http://10.1.20.100:3128 HTTP/1.1 200 Connected HTTP/1.1 301 Moved Permanently Cache-Control: no-cache Pragma: no-cache Content-Length: 0 Location: https://outlook.live.com/ Server: Microsoft-IIS/10.0 Connection: close Output from /var/log/ltm: big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58692 --> 10.1.20.100:3128 big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : ## HTTP Proxy Request ## big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : CONNECT www.outlook.com:443 HTTP/1.1 big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Host: www.outlook.com:443 big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : User-Agent: curl/7.54.0 big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : Proxy-Connection: Keep-Alive big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.outlook.com not in cache - perform DB lookup big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : www.outlook.com - bypass: 1 big-ip1 info tmm3[12384]: Rule /Common/office365_ipi_plugin/office365_proxy_bypass_irule : 10.10.99.31:58692 (10.1.10.245:21666) --> 40.100.144.226:443 It works! Conclusion By combining Microsoft Office365 IP and URL intelligence with LTM, produces a simple and effective method to steer around overloaded Forward Proxy servers without the hassle of messy proxy PAC files.Get Smart with IP Intelligence
There are always threats out there on the big bad internet. The majority of breaches happen at the application layer and many OWASP Top 10s like SQL injection are still malicious favorites to gain entry. Add to that the availability of DDoS tools, anonymous proxies and the rise of hacktivism means networks and systems are bigger targets than ever. Threat detection today relies on a couple elements: Identifying suspicious activity among the billions of data points and refining a large set of suspicious incidents down to those that matter. Today’s cyber-criminals use various techniques to hide their identities and activity. Keeping them out of your systems requires constant vigilance. Every packet that transverses the internet has a source IP address so disabling inbound communications from known malicious IPs can be highly effective. You may not know but F5 offers IP Intelligence Services which provides the functionality to block known malicious IP addresses. It is a layer of IP threat protection and an additional way to allow BIG-IP customers to defend against malicious activity and infrastructure attacks. The IP Intelligence service is offered on several BIG-IP platforms. With IP Intelligence, BIG-IP AFM can be configured to block or allow traffic entering the system based on the reputation of the source IP address. BIG-IP AFM determines reputation using two methods. One is a continuous feed of known or suspected malicious IP addresses provided by a third-party service Webroot BrightCloud. You can also create custom feed lists that specifies IP addresses that have been blacklisted or whitelisted by the organization. The BrightCloud feed is updated every 5 minutes by default and custom feed lists are unique to the AFM and are polled at intervals of your choosing. These two methods are jointly referred to as IP Intelligence and can be used independently or in tandem to filer traffic on the BIG-IP systems. The BrightCloud option is licensed separately through F5 and requires internet connectivity and DNS resolution from your BIG-IP system. Custom feed lists do not need connectivity since it is local to the BIG-IP. IP Intelligence can be applied via AFM firewall policy to the Route Domain or Virtual Server. Once enabled, it will affect all traffic that arrives on your BIG-IP system no matter the access point. The IP Intelligence data is organized into categories that help you differentiate between types of listed IP addresses. There are 11 pre-defined categories including botnets, scanners, infected sources, illegal websites and more. These correspond to the categories in the BrightCloud feed. You can also create up to 51 custom categories to meet your own specific needs. Networks, infrastructures, systems and applications are all under attack these days. While you can do your best at securing your data, sometimes a little call blocking can go a long way in ensuring these known rascals cannot get through. Peace of mind is always a secure feeling. ps
IP Intelligence
Hello, We've deployed IP Intelligence in our organization and some questions arise: Due to the nature of the dynamic IPs, the update of the database should also include the removal of those IPs no longer considered as bad reputation, right? The update details shows the "number of IP Addresses received in the last update" but it does not mention nothing about IPs removed. Does IP Intelligence take place before any other protection? I mean, if a suspicious IP arrives, it is blocked by the IPI and not analysed by the DoS or web scrapping policies, correct? Thanks.IP Intelligence Categories
Hi, Can anyone recommend a resource where I can read more about the 10 specific categories in IP Intelligence. I have a brief one line description of each category however I'd like more information about exactly what would be blocked, how it is identified, how and when the rules are updated for each category, the standard expectation on false positives, why each category should or should not be enabled. Eg, the Cloud-based security category seems quite broad to block all this traffic.IP Intelligence - The Scanners category
Hi, One of the categories that can be enabled to monitor or block with IP Intelligence via the ASM is "Scanners". The definition for this reads "The Scanners category includes all reconnaissance, such as probes, host scan, domain scan, and password brute force." My question is, if this category is enabled and set to block, will it block search engine bots? Will any category in IP Intelligence block search engine traffic? As you can imagine, this is not a desired outcome. Thank youWeb Attacks in IP Intelligence Vs ASM Policy
Hi, We have a variety of applications that each have ASM policies to protect against web attacks etc. We also have IP Intelligence enabled in monitoring mode at the moment which we will switch to blocking mode for some categories shortly. One of the available categories in the IPI setup is "Web Attacks". I am curious as to whether there is any benefit or risk enabling this if I already have a tailored, configured ASM policy. Which takes precedence, the ASM Policy Rules or the IPI Rules if they are each running. I expect if both the ASM Policy and IPI web attack prevention are each enabled, then the traffic would be subject to both sets of rules? Thank you.
