New SSL Orchestrator Support for Netscout Appliances - How to Achieve Rich Traffic Analysis

This DevCentral article provides details on how the newly released F5 BIG-IP version 17 and the Netscout Infinistream capture and analysis appliance family have been tied together to provide a simple visibility solution into TLS encrypted traffic.  The value of access to otherwise opaque traffic flows is significant, simple examples include rich debugging where layer 7 application response codes are now clearly visible or overall security is enhanced by long term logging of traffic content to or from sensitive web sites.

The breadth of TLS adoption has only accelerated in 2022, after becoming the de facto standard for virtually all Internet web browsing. TLS is now rapidly displacing legacy and well understood DNS flows.  Today, rather than clear text UDP-based traffic, DNS is rapidly becoming encrypted via TLS through the sudden rise of DNS over HTTPS (DoH).  Short of a F5 and NetScout style of visibility approach the packet flow in encrypted format becomes largely impenetrable by DNS analysts or other tools that require visibility.

In short, rapid instrumentation with the aim of easy access to key encrypted TLS traffic is critical.  The new version of F5 SSL Orchestrator v10, which is embedded in the BIG-IP v17 stream, and Netscout Infinistream technology makes the setup and execution of TLS analytics painless.

F5 SSL Orchestrator Overview

The F5 BIG-IP has many years of wide adoption in the industry, one popular deployment style is in a reverse proxy topology where, among other features, server load balancing takes place.  This is frequently referred to as application delivery control (ADC) use case.  In such deployments, client side TLS is normally intercepted and decrypted so that decisions on forwarding can be made by deep inspection of the things like HTTP headers or cookies.  With traffic decrypted at the BIG-IP this raises the possibility to forward that traffic through customer high-value analytics services; tools such as traffic analyzers or active devices like IDS/IPS risky traffics inspectors.

F5 SSL Orchestrator provides a simple, single pass GUI to configure a BIG-IP to selectively send decrypted traffic to such ancillary services, with different services easily configurable.  For instance, perhaps traffic categorized as financial is not provided in decrypted format, traffic from key partners is only sent through the IDS/IPS service, and all other traffic is sent through a chain of services that includes packet capture, web traffic analytics, and IDS/IPS.  In summary, SSL Orchestrator provides a simple setup experience and the fidelity to treat all traffic exactly as the operator wishes.

An important note about BIG-IP from F5, the deployment models are diverse and flexible.  Although a reverse proxy format described earlier is common, the BIG-IP can also be deployed as a forward proxy in transparent mode or possibly explicit modes where client hosts are aware of the proxy’s presence.  The forward proxy transparent mode allows for a rapid TLS access strategy, such as placing BIG-IP at the egress point of an enterprise location and leveraging SSL Orchestrator to provide decrypted TLS traffic to existing service chain appliances.  This is timely as many premiere appliances have become undervalued due to the vast and fairly recent adoption of TLS, something many tools were not originally designed to decrypt, and almost certainly not at today’s network speeds.  F5 can address this pain point with a one stop approach to TLS interception.

SSL Orchestrator Incorporates Netscout Into GUI-Based Setup

With the latest release of SSL Orchestrator, bundled into the BIG-IP v17 release of 2022, the NetScout Infinistream family of appliances, including the vStream and InfinistreamNG, can be incorporated into a SSL 0rchestrator service chain in just minutes.  The complete workflow of an SSL Orchestrator setup, referred to as the “Guided Configuration” is captured in the following screenshot, which shows the summary of the setup phases from left to right which are passed through to reach a configured state.

 

The portions of the SSL Orchestrator setup that are specific to NetScout include the “Service” and “Service Chain” configuration panes of the overall guided configuration process.  Some services offer pass through features, such as an IPS where traffic is received by the service and returned to BIG-IP.  Other services operate in a “tap” mode where they only need to receive traffic, which may or may not be decrypted as per SSL Orchestrator interception rules.

The following image demonstrates a catalog of tap mode services available within the SSL Orchestrator workflow, new with this release is the support shown for the Netscout option.

 

 

By selecting the Netscout tap button, a single-screen setup is quickly carried out.  The key items that may be configured are:

• a meaningful name for your vStream or InfinistreamNG
• a destination MAC address that may be useful in environments where multiple F5 appliances feed the Netscout solution, and a simple tracking method is desirable
•  the VLAN/BIG-IP interface for the traffic to be passed to in order to reach the Netscout gear
• Optionally, TCP port remapping can be carried out, if tools prefer to see decrypted TLS traffic on, say, port 80 or port 8080 as opposed to the default port which most frequently is 443

 

Service Chains and SSL Orchestrator

The second aspect of setting up the BIG-IP and Netscout relationship is to include the newly added Netscout service to a service chain.  Service chains allow specific traffic types to be directed to through any combination of services that have been added.  In our example, we have a simple service chain consisting of exclusively the Netscout solution and as such we need only add that service to the right-hand column.  As shown, an optional Broadcom (Symantec) DLP solution could be added to the chain, or the DLP solution might be added to another service chain for a very specific category of traffic only.

 

 

With the Netscout security solution now integrated with BIG-IP as a service chain traffic is now ready for handling.  As mentioned, there are multiple topologies supported by BIG-IP, one interesting use case would be for existing F5 users, utilizing reverse proxy mode and supporting web servers under their purview, to direct decrypted traffic flows to critical virtual servers to Netscout for analysis.

Many off-the-self data center application servers now enforce TLS encryption even on the server side of a load balancer, meaning a simple physical tap that previously fed tools clear HTTP within the datacenter is no longer sufficient.  The SSL Orchestrator feature allows per virtual server decrypted traffic to be fed to Netscout for continued analysis even with TLS on both sides of the load balancer.

This article focuses on another use case, the transparent layer3 forward proxy mode, where client traffic is flowing outward to Internet sites where the application resources and SSL key are obviously not under the control of the enterprise IT team.  Categorization of traffic allows those network flows warranting inspection to be directed in decrypted form to the tool stack, including Netscout.

This “outbound” topology normally involves the following steps:

• The internal traffic takes a routed path into and out of the BIG-IP. The client is unaware of the SSL Orchestrator device while en route to the Internet server
• The traffic is decrypted and is sent to security devices, such as the Netscout vStream or InfinistreamNG
• Optionally, traffic that is processed by an active device like an intrusion prevention system is routed back to BIG-IP where it is re-encrypted and is sent out to the applications and sites hosted on the Internet

The final bullet point above, the ability to adjust or filter traffic in some form all in real-time, without simply dropping the connection is a distinguishing and highly valued feature of BIG-IP and the SSL Orchestrator.

 

 

Utilize a Trusted Certificate Authority (CA) to Re-Sign TLS Certificates

After deploying the configured SSL Orchestrator one need simply allow traffic to pass through the forward proxy and analysis will take place on the Netscout.  It is important that the local clients presenting traffic to the transparent proxy accept the certificates, which entails using a trusted certificate authority (CA) on the BIG-IP.  Many approaches to distributing CA certificates to end users exist, one approach in a corporate environment is to utilize Active Directory – Group Policy Objects (GPOs) to quickly achieve this.

For the purposes of this article’s demonstration a certificate, or certificate chain, was loaded into the “SSL Configuration” phase of the guided configuration.  The top certificate (bigip_f5demo) is utilized by the BIG-IP and the bottom certificate corresponds to the certificate authority (CA) created.  Outbound, the BIG-IP by default trusts the standard CA’s that come out of the box in popular browsers such as Chrome or Firefox.

 

 

The Solution in Action

In the following example we see a sample web page visit to an Internet site, a quick verification that the traffic is being intercepted is to simply check the certificate within the browser.  In this example we see a certificate issued by the BIG-IP demo administrator, confirming interception is indeed taking place.

 

Without the SSL Orchestrator the analysis would be limited.  Although one can read from the encrypted packet stream the destination address, due to HTTP virtual hosting hundreds of different servers might share addresses and increasingly such addresses are tied back to content delivery networks (CDNs).  DNS requests and responses immediately prior to TLS sessions have historically been used to guess the domain name contacted, however with the rise of TLS encrypted DNS over HTTPS (DoH) even DNS may be opaque to packet analyzers.

Finally, a well-established technique it to look at the clear text server name indicator (SNI) in TLS traces to try to determine target domain names.  A major issue with this is the pending TLS version 3 adoption of encrypted TLS Hello messages, the ability to see SNI values may soon become extremely difficult.

With SSL Orchestrator performing TLS interception and providing Netscout with a decrypted stream, the following example demonstrates the traffic in the previous screenshot clearly being decoded within the Packet Analysis tools offered by Netscout.

 

The yellow highlighted entries in the top pane, packet summary, show full decoding of the traffic as an HTTP GET command and successful 200 OK response.  The highlights in the lower pane, packet details, demonstrate individual elements of data the analyst now can access, such as the numeric (200) response code, the date/time the web page was last modified and finally that the server hosting the web site is in fact a nginx server.

The following screenshot reiterates the value presented by the F5 and Netscout combination, as the left side trace shows decoded commands and responses issue to build the webpage in its entirety in the browser.  The right side demonstrates the same page load, minutes later, without the BIG-IP serving decrypted traffic.  The value of the packet trace is greatly diminished as only TLs encrypted cipher streams are delivered to the Infinistream.

 

 

DNS over HTTPS (DoH) Analysis with SSL Orchestrator and Netscout

Beyond web browsing, the SSL Orchestrator can provide a tool like Netscout’s other high value TLS decrypted traffic.  One example is DNS over HTTPS which is valuable as many analysis solutions have relied upon clear channel UDP-based DNS transactions to understand endpoints being communicated with.  With DoH the encryption issue could be a major hurdle.  However, as seen in the following image when traffic has been delivered through SSL Orchestrator, DoH-based requests and responses are now visible to the analyst.

 

 

In this example the tool, and thus the NetOps or SecOps analyst using the tool, have clear visibility into the DNS response.  As shown the IPv4 address of the website www.saskatoon.ca was provided and the DNS time-to-live, as well as other DNS fields, are fully exposed.

Summary

The recent BIG-IP release version 17.0 has been many new features, only one of which was touched upon in this article.  As seen, the latest version of the SSL Orchestrator module, and its simple guided configuration setup, directs selected decrypted TLS traffic categories to the Netscout Infinistream series of appliances, both physical and virtual.  This facilitates rapid adoption of high value monitoring and analysis at critical points within network, such as in front of web servers or at egress points where site traffic is directed to the Internet.  The F5 BIG-IP can be harnessed in multiple topologies, including reverse and forward proxy modes, to provide real-time insights into the variety of application traffic that today increasingly use TLS for transport.