HTTP Explicit Proxy Explained in Plain English

Introduction

If you've ever used the old Linux Squid proxy or F5's Secure Gateway solution, you might be familiar with the existence of HTTP Explicit Proxy. If all you're looking for is to configure it ASAP, there's an iApp available to set things up in no time. This technology is also used by F5's Secure Web Gateway Services (SWG). This article will walk you through the mechanics behind the scenes, i.e. how to manually configure it and understand how it works through a lab test.

1. Lab Scenario

Expected result is that client (.135) sends HTTP GET request to server2.rodrigo.example using 10.199.3.100/32 as http proxy and BIG-IP queries our DNS server (172.16.199.31) for server2.rodrigo.example's IP address, retrieves web page and passes on to client.

2. How Explicit Proxy works

In HTTP Explicit proxy, we configure our client (application) to point to BIG-IP's virtual server which will act as an HTTP proxy for external websites. Once client issues a request, BIG-IP will then open a separate connection with external server the client is trying to connect to and issue the requests on behalf of client passing requests/responses back and forth between client and external server.

In order to test HTTP Explicit proxy functionality, I configured my client's browser pointing to BIG-IP's HTTP Explicit proxy virtual server. BIG-IP is supposed to issue the requests on behalf of client and pass them back to client. The client is explicitly configured to use VIP's IP address as a proxy. Here's my config on Firefox (Preferences → Advanced → Network):

When using any form of unencrypted traffic Firefox forwards request straight to BIG-IP without issuing an HTTP CONNECT request:

We can see that BIG-IP just forwards request mostly as it is. The reason for GET / instead of FQDN on server-side is because BIG-IP performed DNS resolution before that and I filtered it.

If request is HTTPS instead of HTTP, things are slightly different. First, Firefox (my browser) tries to establish an HTTP tunnel (using CONNECT HTTP method) to https://server2.rodrigo.example, BIG-IP then immediately establishes TCP connection with remote-destination (after DNS lookup of course!). Lastly, once TCP connection is established with remote-destination, BIG-IP responds (to client) with 200 Connected signalling tunnel is successfully established and BIG-IP is ready to forward any requests through recently established tunnel:

Now that tunnel is properly established, BIG-IP just forwards whatever Firefox sends. In this case, the SSL handshake Client Hello was the first packet which was captured:

It is interesting to note that there is still an HTTP header in the messages between client and BIG-IP with proxy information sent by firefox (frame 47) that encapsulates SSL message:

Now that we understand how HTTP Explicit proxy works under the hood, we're ready to learn how to set up BIG-IP as Explicit proxy.

3. Setting Up Explicit proxy on BIG-IP

3.1 Create Virtual Server with no pool (you can leave http profile for later):

3.2 Create DNS resolver:

In the GUI, it would be on Network → DNS Resolvers → Create. Then, we'd click on Forward Zones and add a dot '.' if we want all queries to be sent to this name server. For the purposes of this specific test, I could also have named this zone as rodrigo.example. instead.

Note: In my lab test I worked out that forward zone's name is really important! When I named it 'testing' or 'google' BIG-IP did not forward DNS query to 172.16.199.31. However, when I named it either . or rodrigo.example. then DNS resolution worked.

3.3 Create tunnel interface

BIG-IP already has a default http-tunnel interface. As long as encapsulation type is tcp-forward, we can either stick to default or create a new one. I decided to create a new one for the purpose of this lab test:

3.4. Create http profile using http-explicit as parent and assign it to VIP created previously:

Note: When configuring it from GUI just set proxy type to explicit and http-explicit will be set to parent profile.

The equivalent in the GUI:

3.5. Testing Connection

3.6 Enabling encrypted requests

Initially I did not understand why HTTP traffic worked when default-connect-handling was set to deny but HTTPS did not. Below is the summary of my tests:

Looking at the packet capture, I noticed that BIG-IP was explicitly denying Firefox's request to establish a tunnel as shown below:

Note: For this test I created another Virtual Server called using a different address 10.199.3.101 instead of the one I originally tested just to test default-connect-handling.

After reading K40243113: Overview of the HTTP profile, I kind of understood the point:

"indicates that outbound requests are delivered only if another virtual server is listening on the tunnel for the requested outbound connection. With this setting, virtual servers are required, and the system processes the outbound traffic before it leaves the device."

I then created another VIP with same destination IP address and port as back-end server client was trying to connect to just to confirm if this time traffic would go through:

And indeed it worked:

I also tested with a wildcard virtual server listening on *:443 instead of 172.16.199.32/32 and it worked fine as well.

What we have observed so far is that when default-connect-handling is set to deny, in order for encrypted client traffic to be accepted by BIG-IP and proxy tunnel established, the destination address that matches the one on client request has to have a listener on BIG-IP for return traffic. Therefore, we can conclude that default-connect-handling setting governs what traffic is accepted by the tunnel interface, but unencrypted requests are not affected.

The below picture sums up the explanation:

default-connect-handling just adds the extra step to validate whether there is a listener on BIG-IP for destination address:port or not. If not, we deny request. If yes, request goes through.