SSL Orchestrator Advanced Use Cases: Reducing Complexity

Introduction

Think back to any time in your career where one IT security solution solved all of your problems right out-of-the-box. Go ahead, I'll wait.

Chances are you're probably struggling to think of one, or at least more than one. Trust me when I say it doesn't matter how many other industry professions are doing what you do, many of your security challenges are absolutely unique to you. "Best of breed" means little if it can't solve all of your problems. That's why an entire ecosystem exists of best-of-breed security solutions...because each is best at a subset of the total problem space. The problem with this inescapable truth is that using multiple solutions to solve for the entire security conundrum also breeds complexity.

The spectacular beauty of an F5 SSL Orchestrator solution is that it can drastically reduce that complexity and at the same time, increase manageability, scalability, and proficiency. With great ease and finesse, a sprawl of heavily burdened security products can be transformed into dynamically addressable and independently scalable malware munching monsters, void of the weight of decryption, each focused purely on their strengths. Best of all, SSL Orchestrator sits on top of the F5 BIG-IP, which itself provides unparalleled flexibility. If the SSL Orchestrator itself doesn't meet your every need, it's almost certain there's some configuration or programmatic way to achieve what you're looking for. In this article we will explore some of that immense flexibility.

I should note here that the following guidance stands equally as a set of best practices for configuring and managing SSL Orchestrator, and for reducing complexity in an existing environment.

Let's go.

SSL Orchestrator Use Case: Reducing Complexity

We have already solved for the security product complexity of integrating multiple security products, so this article specifically tackles another challenge: what happens when the SSL Orchestrator configuration itself gets to be complicated? The goal is to reduce overall complexity, but as with all other things we tend to think about challenges one at a time, creating separate solutions for each problem as it comes up. This becomes troublesome when we build large numbers of nearly duplicate topologies, or create super-complex security policies. The most important consideration here is that an SSL Orchestrator configuration creates and manages ALL of the objects it needs, and very often that can be a lot. For example, a layer 3 outbound topology with one inline service will create no less than 700 dependent configuration objects. An inbound topology will create about 300 objects. Compare that to a typical LTM virtual server, where common objects are often re-used, and that's around 20. The net result is that creating a ton of SSL Orchestrator configurations can put a strain on the control plane, but also sort of defeats the point of reducing complexity. But fear not, there are a number of really interesting ways to reduce all of this without losing anything, and in some cases even increase capacity and flexibility. The solutions I am about to present are broken down by topology direction:

• Reducing Complexity for Reverse Proxy Topologies (Inbound)
• Reducing complexity by using shared security policies
• Reducing complexity with the Existing Applications topology
• Reducing complexity with existing applications and shared policies
• Reducing complexity with gateway mode
• Reducing complexity with existing application, SNI switching and address lists
• Reducing Complexity for Forward Proxy Topologies (Outbound)
• Reducing complexity with layered architectures

For the purpose of illustration, I've also extracted the average number of SSL Orchestrator security policy objects for comparison, where a minimal outbound security policy is about 80 objects, and an inbound policy is around 60 objects. I will use these numbers to make basic comparisons throughout, but of course this will always vary per environment. You can find the total number of control plane objects using this command:

tmsh -q -c 'cd /; list recursive one-line'|wc -l

If you don't have access to the BIG-IP command line, but do have a copy of the bigip.conf, you can do this:

cat bigip.conf |egrep '^[a-z]' |wc -l

And the total number of access (security policy) objects using this command:

cat bigip.conf |egrep '^apm ' |wc -l

Reducing Complexity for Reverse Proxy Topologies

There are a number of optimizations that can be done for reverse proxy topologies, but let us start with the simplest and most profound update. 

Option 1: Reducing complexity by using shared security policies

One single topology will create its own security policy, but very often your security policies are all going to be doing basically the same thing. You can make a dramatic impact on complexity and overall object count by simply reducing the total number of security policies and reusing these across your topologies. 

figure 1: Reducing complexity in a reverse proxy with shared security policies

Let's break it down to show the benefit of this approach. We'll illustrate using twelve separate SSL Orchestrator inbound topologies. As I mentioned earlier, a basic inbound layer 3 topology with one inline service will create about 300 control plane objects. Of that, less than half of these are the security policy, service chain and the inline service, around 120. An inline layer 2 service will create around 60 objects, and as there's a finite number of security services to begin with, they'll generally be shared between the security policies so we won't count these. If we then focus just on the security policy and service chain, that's around 60 unique objects. If each of the above twelve topologies creates its own security policy, reducing that down to just 3 unique security policies (for example), can remove over 500 objects, a nearly 75% reduction, and by the way you have also just made overall policy management simpler. Configuration object reuse has tremendous benefit.

If you are building a new SSL Orchestrator environment, consider this option if you simply cannot reduce the total number of inbound topologies, but can reasonably reduce security policies down to a smaller set of unique policies that can be shared.

If you have an existing environment and need to reduce complexity, setting this up is also very simple:

• Use the above scripts to get a total object count before taking any action.
• Identify a set of security policies that perform the same steps, pick one or create a new one, and then under each respective topology replace the existing security policy with this single shared security policy.
• You can then delete the old unused security policies, then repeat this action for each topology until all duplicate policies are removed.
• Run the object counts tools one more time and compare the difference.

Option 2: Reducing complexity with the Existing Applications topology

Shrinking the number of active security policies is an easy win, but even that isn't where the bulk of objects are. Recall again that a minimal inbound topology is about 300 objects, so minus the 120 from the security policy, service chain and inline service objects, 180 remain in the topology and its other dependent configurations. A layer 3 inbound SSL Orchestrator topology is, basically, a reverse proxy virtual server, SSL profiles, security policies, and various other dependent profiles and configurations. If you have LTM licensed on the BIG-IP, you can further reduce this by switching to an Existing Application. The Existing Application topology simplifies the configuration by only building the security policy, service chain and security services. You then attach that security policy to an existing LTM application virtual server.

figure 2: Reducing complexity in a reverse proxy with existing application topologies

Let's break it down again. Using the same twelve topologies with individual policies, disregarding the security services, you're seeing around 2800 objects. If you were to replace each FULL topology with an LTM application virtual server using an Existing Application security policy, you can see in the image above a pretty significant drop in configuration object count, to about 960 objects. For twelve topologies that's a 77% reduction in configuration objects. Full topology creation is, by design, constrained to the most essential (optimal) elements and thus limits some flexibility of manual configuration. Using a manually defined LTM VIP with an Existing Application topology instead of a full inbound topology will provide the full flexibility of LTM together with security policies and inline services. One caveat, the existing application option requires an LTM license.

If you are building a new SSL Orchestrator environment, consider this option if you can see the benefits of deploying existing application virtual servers, where this technique provides additional capabilities and near unlimited customization.

If you have an existing environment and need to reduce complexity, putting this one together requires a bit more work in an existing environment, but it's not insurmountable. If this is a new environment, then clearly starting with this configuration approach could be beneficial. To test this configuration, we'll use the virtual server's Source Address parameter.

• Use the above scripts to get a total object count before taking any action.
• Create a new Existing Application topology to define your security service(s), service chain and corresponding security policy.
• Disable strictness on an existing topology. Assuming traffic is flowing to this topology already, we'll need to make a quick switch at the command line so we need strictness to be disabled.
• Create an LTM application virtual server that matches the topology listener (ie. destination address, port, pool, etc.), but for testing purposes enter just YOUR client IP/mask in the Source Address field. BIG-IP virtual servers follow a "most specific" order-of-precedence, so by adding a specific source address, only your traffic will flow to the new VIP and not anything else. Also attach your new Existing Application security policy to this virtual server. Under the Access Policy section, Access Profile setting, select "ssloDefault_accessProfile". And then under Per-Request Profile, select your new security policy.

figure 3: SSL Orchestrator Existing Application topology selection

• You should now have an LTM virtual server that matches the topology virtual server, except that it only listens for traffic to your address. You can alternatively apply a source Address List here to allow multiple testers to access the new VIP. Make sure that you are able to access the application and that decrypted traffic is passing to the security service(s).
• When you are ready to make the switch for all traffic, you'll need to change the Source Address field in the topology listener virtual to something specific, and then change the Source Address field in your new LTM virtual to 0.0.0.0%0/0, again manipulating the order-of-precedence. You can do this manually, or via simple TMSH transaction in a Bash script to switch both services simultaneously:

#!/bin/bash

tmsh << EOF
create cli transaction
modify ltm virtual sslo_app1.app/sslo_app1 source 10.10.10.10/32
modify ltm virtual app1-ea-vip source 0.0.0.0%0/0
submit cli transaction
EOF

The first modify command points at the existing topology listener virtual and changes the source to something unique. The second modify command changes your new LTM application virtual source to allow traffic from all (0.0.0.0%0/0). All of this is done inside a transaction, so the change is immediate for both.

• Perform the steps above for each inbound layer 3 topology that you need to replace with an LTM virtual and Existing Application policy.
• When you're satisfied that everything is now flowing through your LTM application VIPs, go back and delete the unused topologies, security policies, service chains and SSL configurations.
• Run the object count tools one more time and compare the difference.

Option 3: Reducing complexity with existing applications and shared policies

Okay, so far we've separately reduced complexity by either sharing security policies, or using Existing Application topologies with LTM VIPs. What if we actually combined these two methods? 

figure 4: Reducing complexity in a reverse proxy with existing application topologies and shared policies

Again, if we remove the security services from the equation, you should see an absolutely enormous drop in object count. Twelve separate topologies and security policies at around 2800 objects, consolidated to twelve LTM VIPs and three unique security policies is an 86% reduction in configuration objects! Plus, again, you've expanded your flexibility by switching to LTM application VIPs and made SSL Orchestrator policy management much simpler.

Migration to this configuration option is more or less the same as the last, except instead of twelve individual Existing Application topologies, you'll only need to create a smaller set of unique policies and share these accordingly.

If you are building a new SSL Orchestrator environment, consider this option if you can see the benefits of deploying existing application virtual servers, and you can reasonably reduce security policies down to a smaller set of unique policies that can be shared.

Option 4: Reducing complexity with gateway mode

At this point, you've now potentially removed 86% of the total configuration objects and made your environment clean, lean, simpler and more flexible. What more could you ask for? Well I'm glad you asked. The following isn't for everyone, but if it makes sense in your environment, it can create a HUGE simplification of resources and management. An SSL Orchestrator inbound "gateway mode" topology is intended as a routed path. Conversely what we've been talking about so far are "application mode" topologies, where the destination IP:port are configured within the topology, or LTM application virtual server. External clients target this address directly. But in gateway mode, the listener is a network address such as 0.0.0.0/0, so the destination address will actually be behind the BIG-IP, possibly a separate BIG-IP, another load balancer, router, or the application server itself. Traffic is routed through the BIG-IP to get there. You may see this as an advantage if the SSL Orchestrator sits closer to the edge of the network, and/or not owned and managed by the same teams. In any case, by virtue of the single 0.0.0.0/0 listener, you really only need ONE topology and ONE security policy, though your security policy may tend to be a bit more complex.

figure 5: Reducing complexity in a reverse proxy with gateway mode

If you are building a new SSL Orchestrator environment, consider this option if you can insert the BIG-IP as a routed hop in the application path.

If you have an existing environment and need to reduce complexity, we can actually use that most specific order-of-precedence thing again to our advantage. As long as you have a bunch of application mode topologies configured with specific destination addresses, only new traffic flows that don't match one of these will flow to your wildcard gateway mode topology. Create a gateway mode SSL Orchestrator inbound topology by ensuring that the destination address is 0.0.0.0/0 (or other appropriate network address) and no pool is assigned. This will also automatically disable address and port translation so this effectively becomes a forwarding virtual server. You can test this topology by sending a request to a destination address that isn't defined by one of the application mode topologies, but does exist beyond the BIG-IP. When you're ready to move traffic over to the gateway topology, you can either start deleting the application mode topologies (or LTM virtual servers), or simply disable them. 

Now there's one other thing you have to address in the gateway mode configuration, and that's how to handle server certificates. In inbound application mode, each topology has its own SSL configuration and applied server certificate and key. But what do you do if all HTTPS traffic flows through a single virtual server? There are two options here. In both cases you'll need to create multiple SSL profiles, which you can either do in the SSL Orchestrator UI or manually in the BIG-IP.

Certificate selection option 1: Attach multiple SSL profiles to the topology listener

You will need to disable strictness on the topology to use this approach, as you'll need to edit the virtual server directly. You're going to add all of the client SSL profiles to the virtual server. The benefit of this method is that it is fast. The downsides are that you have to disable topology strictness (and leave it disabled), and also keep track of which client SSL profile has the "Default for SNI" setting.

• Create each client SSL profile and specify a unique server certificate and key. You'll also need to edit the "Server Name" field and enter the unique SNI hostname. The BIG-IP will select the correct client SSL profile automatically by virtue of the client's TLS ClientHello "servername" extension (SNI). In this method you need to select one of the client SSL profiles as "Default for SNI". This is the profile the BIG-IP will select in the very rare case that the client does not present an SNI. It's also worth noting here that as of BIG-IP 14.1, SSL profile selection can also be done based on the subject and subject alternative name (SAN) values in the applied certificate, versus a single static "Server Name" entry in the client SSL profile. You should still enter a Server Name value in the profile, but profile selection will only use this if the SNI does not match the subject or SAN in the applied certificate.
• Attach all of the client SSL profiles to the topology listener virtual server.

Further detail: K13452: Configure a virtual server to serve multiple HTTPS sites using the TLS Server Name Indication feature

Certificate selection option 2: Dynamically assign the SSL profiles via client SNI

Using a slightly different approach, we'll employ an iRule on the topology interception rule that dynamically selects the client SSL profile based on the client's ClientHello SNI. The benefits here are that this doesn't require non-strict changes to the topology, and you don't have to set a Server Name value in the SSL profiles or worry about the Default for SNI setting. The disadvantage is that iRule-based dynamic profile selection will generate a small amount of additional runtime overhead. You'll need to weigh these two options against how heavily loaded your system will be.

• Create each client SSL profile and specify a unique server certificate and key.
• Create a string datagroup that maps the SNI server name value to the name of the client SSL profile. Example:

    www1.foo.com := /Common/www1.foo.com-clientssl
    www2.foo.com := /Common/www2.foo.com-clientssl
    www3.foo.com := /Common/www3.foo.com-clientssl

• Navigate over to https://github.com/f5devcentral/sslo-script-tools/tree/main/sni-switching and grab the "library-rule.tcl" iRule and import this to your BIG-IP. Name it "library-rule".
• On that same page, also grab and import the "in-t-rule-datagroup.tcl" iRule. Modify the data group name on line 23 with the name of your data group.
• Finally, navigate to the Interception Rules tab in the SSL Orchestrator UI and edit the corresponding interception rule. At the bottom of the page add the new "in-t-rule-datagroup" iRule and re-deploy. 

As TLS traffic enters the gateway mode topology listener, the above iRule will parse the client's ClientHello SNI, lookup the corresponding client SSL profile in the data group, and switch to that profile. Whenever you need to add a new site, simply create a new client SSL profile and add the respective key and value to the data group.

Option 5: Reducing complexity with existing application, SNI switching and address lists

I've saved this one for last to first introduce you to some of the underlying techniques, specifically address lists and SNI switching. You saw mention of address lists in testing the migration to existing application security policies, and we covered SNI switching in the previous gateway mode solution. If you can't employ a gateway mode (routed path) architecture in your environment, you can still reduce the total number of LTM virtual servers in an Existing Application topology option by using address lists. 

figure 6: Reducing complexity in a reverse proxy with existing app, SNI switching, and address-lists

An LTM virtual server supports either a single static source or destination IP, or an address list that you can define in the UI under Shared Objects >> Address Lists. We use address lists to consolidate multiple LTM virtual servers into a single virtual server using an address list of multiple destination IPs. Since this is an LTM virtual server and not a full topology, you can also easily just add the multiple client SSL profiles without dealing with topology strictness. But you do still need to define one client SSL profile as Default for SNI. In lieu of that you can also use the above SNI switching iRules. If you then also reduce and reuse a unique set of security policies, you can drop control plane configuration objects as low as they can go. I've stated this a few times already, but by using LTM virtual servers instead of full inbound topologies you not only cut down on control plane congestion but also regain some flexibility. And moving to shared security policies means configuration management becomes a breeze.

Reducing Complexity for Forward Proxy Topologies

Where inbound reverse proxy complexity usually manifests as lots of topologies and lots of security policies, there's usually only one or a small few forward proxy topologies. Most of the complexity in a forward proxy topology is actually in the security policy itself, or in situations where specific environmental requirements necessitate non-strict customization. To solve for this sort of complexity you can employ what is referred to as an "internal layered architecture." This idea is covered in great detail here: https://devcentral.f5.com/s/articles/SSL-Orchestrator-Advanced-Use-Cases-Reducing-Complexity-with-Internal-Layered-Architecture. Essentially this method uses a "steering" virtual server positioned in front of a set of SSL Orchestrator topologies (on the same BIG-IP) and directs traffic to one of the topologies via local steering policy decision. 

figure 7: Reducing complexity in a forward proxy with an internal layered architecture

This architecture has some pretty interesting advantages in its own right:

• The steering policy is simple, flexible and highly automatable.
• In the absence of policy decisions, the SSL Orchestrator topologies are reduced to essentially static "functions", a unique combination of actions (allow/block, TLS intercept/bypass, service chain, egress). 
• Topology objects are re-usable so while creating multiple topology functions, you'll likely re-use most of the underlying objects, like SSL configurations and services.
• Topology functions support dynamic egress (egress via different paths).
• Topology functions support dynamic local CA issuer selection (for example, using a different local CA for different "tenants").
• Topology functions support more flexible traffic bypass options.
• Topologies as static functions will likely not require any non-strict customization, so makes management and upgrades simpler.

The above DevCentral article provides much more insight here and offers a set of helper utilities to make steering policy super simple to deploy. To migrate to an internal layered architecture, we can use the most specific order-of-precedence thing again. 

• Create a set of "dummy" VLANs, just an empty VLAN with no assigned interface. SSL Orchestrator topologies require a VLAN, but these are going to be "internal".
• Create your set of SSL Orchestrator topologies. The key idea here is that each will have a very basic security policy. The Pinners rule is actually covered in the helper utility, so you could reduce each security policy to a single "All Traffic" rule and apply the unique combination of actions. Last, assign a dummy VLAN to the topology and deploy it.
• Create your steering VIP as described in the other article but give it a source address filter that matches your local client address or use an address list to let your local team test it. 
• Once you're confident that traffic is flowing through the steering virtual and to the correct internal topologies via policy, you can use the same TMSH transaction technique from the reverse proxy Existing Application use case above. That will move the 0.0.0.0/0 source address filter from your existing forward proxy topology to the steering VIP. Once you've done that, you can delete the unused client-facing topology and any other unused objects.

Summary

If you are just getting started with SSL Orchestrator, you have a lot of options to choose from. The objective here is to encourage simplicity. For SSL Orchestrator specifically that can mean reducing inbound topologies down to a single gateway mode, or utilizing common BIG-IP capabilities like address lists to reduce the number of existing application virtual servers. Or at the very least, consolidate to a common shared set of unique security policies. In fact, many of the above techniques for reducing and simplifying architectures are useful well beyond just SSL Orchestrator. Any time you have an environment with an enormous number of virtual servers, you can almost always consolidate a lot of these with practices like address lists and SNI switching. That helps by cleaning up the control plane and can simplify configuration, where "common" settings can now be managed in one place.

This article was long, and for that I sincerely thank you for bearing with me to the end. Hopefully you can use some of this information as a best practice in managing what is easily one of the most flexible application delivery control devices on the planet. And in doing so, also vastly reduce the complexity in your security stack with SSL Orchestrator.