Change server SSL profile based on HTTP content

[_JOHN_]

I have a single Virtual Server which provides internal to external access for a specific set of FQDNs (and nothing else).

It is a wildcard VS which accepts traffic destined to the actual external IPs. The VS does not perform destination IP/port translation (but source IP is SNATed) - i.e. traffic arrives at VS by:

• Client does DNS lookup and gets external 'real' IP
• Client establishes connection to this IP - but traffic actually directed to, and picked up by, F5 wildcard VS

VS has a pool with only 1 member, which is an upstream device that transports traffic to the internet (F5 basically uses pool member to get layer 2 MAC to forward the traffic to, but the layer 3 IP addressing of the traffic remains as that originally received, i.e. the actual external server IPs).

Each FQDN has its own specific Server SSL profile, which is assigned by iRule (based on client presented SNI).

This has all been working OK for a long time.

There is now a requirement that a couple of the FQDNs will need different server SSL profiles based on content within the HTTP request. It is possible that a single incoming TCP connection to the VS could contain HTTP requests which are to be handled differently, e.g.

HTTP Header X = A Server connection should use SSL profile A

HTTP Header X = B Server connection should use SSL profile B

The problem here is that you can't just set up a server connection based on the initial HTTP Header X received on a new clientside connection to the VS. It is posible that different HTTP requests could then come across that single client TCP stream each requiring a different Server SSL profile (choice of two, A or B).

My question is can anyone suggest how this can be accomplished without having to change anything on the clientside?

I have set something up which on first inspection appears to work in a 'lab' (i.e. PC running VMWare and LTM VE :-). I don't want to describe it just yet in case what I have done isn't the best solution (or maybe wouldn't even work properly in a real world setting with lots of load). At this point I am just looking for ideas of how other's might approach this, and don't want to influence anyone's ideas by describing my first attempt 🙂

So any ideas??? Please ask any questions as needed to help clarify the ask 🙂

[Josiah]

The OneConnect profile allows you to load balance client http requests to server flows in a non 1-to-1 manner. While perhaps the traditional idea is multiple client connections reuse a single serverside connection, it also allows a single client connection to use multiple different serverside connections.

https://support.f5.com/csp/article/K7208

[_JOHN_]

Thank you very much for the reply. It would be good to use oneconnect if possible, but I am not sure how to make that happen yet 😊

The lab setup I tried (please feel free to rubbish this setup and suggest any better alternatives) uses 1 pool per FQDN/HTTP combination. For example if I describe using just two pools for simplicity:

-      {www.test.com with Header X = A} maps to Pool A

-      {www.test.com with Header X = B} maps to Pool B

I then used a traffic policy to forward traffic to the appropriate pool based on this FQDN/HTTP combination.

Other than name, each pool is 100% the same – same single member (IP & port). The only reason I used different pools was to give different ‘targets’ to load balance to (very welcome to suggestions which only require 1 pool).

I then used an iRule to pick up LB_SELECTED events and assign the appropriate Server SSL profile based on the name of the pool chosen, with the profile actually being attached and used in the subsequent SERVER_CONNECTED event.

This appears to work OK, with a traffic profile along the lines of:

-      {www.test.com with Header X = A}

first HTTP request received – create new server connection using server SSL profile A

-      {www.test.com with Header X = A}

Any number of subsequent HTTP requests received with same FQDN/HTTP combination –LB_SELECTED doesn’t even trigger, traffic just goes across existing server connection OK

-      {www.test.com with Header X = B}

HTTP request received with diff FQDN/HTTP combination – existing server connection closed (TCP FIN) and new connection established to same server but using server SSL profile B

-      {www.test.com with Header X = B}

Again connection remains constant while subsequent HTTP requests present with the same FQDN/HTTP combination

-     {www.test.com with Header X = A}

Cycle repeats…….

At the TCP level the two server connections (A & B) will be the same, i.e. client addr/port & server add/port will be the same. However the key difference is that each will have used a different server SSL profile, i.e.

Pool A connections use server SSL profile A

Pool B connections use server SSL profile B

Is the oneconnect TCP reuse pool aligned per LTM pool, or is it ‘global’? What is the relationship between TCP and SSL with regard to the idle connections in the reuse pool – i.e. are they purely TCP connections or do they also have active SSL sessions?

I did try enabling oneconnect but results were inconsistent, although usually it just kept all traffic going across the first connection established. I therefore assumed the oneconnect reuse pool was global rather than being a separate reuse pool per LTM pool. I also assume SSL remains established per TCP connection.

If you could help me understand how to amend the setup (happy to scrap what I have and go with something completely different) so as to leverage the power of oneconnect, but while still maintaining the key goal of ensuring the traffic is sent across a pipe which has used the correct server SSL profile it would be very much appreciated 😊

Basically the ideal goal would be traffic switching between two different ‘SSL pipes’ to the same server, similar to that listed above, but with each connection remaining up rather than only one being activated at a time. I know the objective of oneconnect is to keep connections active like that, but I need to understand more about what a oneconnect idle connection actually represents and how the F5 makes a decision on which one to choose – e.g. if it is purely on IP/port info, or if there is some way I can align this to account for the different server SSL profiles used as well.

[Josiah]

Essentially the oneconnect profile is used IN ADDITION to what you have already configured in your irule/LTM profile. Without a oneconnect profile, your only option is to create a brand new serverside connection every time your header changes. With a oneconnect profile, you can have persistent serverside connections to your two destinations "kept open" and then, depending on the header in the client request, it will just send it to an available connection instead of creating a new one.

That said, oneconnect is not going to pipeline multiple clients to a single server connection, it'll only use an existing connection if that connection is IDLE. If there are no IDLE serverside connections, it'll create a new one, similarly, if you don't meet the mask (for example a /32 mask means clients can only reuse their own connections, whereas a /24 you could let clients with source ips in the same subnet share connections with each other) criteria, it'll also create a new connection (even if there is an idle one for a different mask). The interaction of all this various logic probably explains what you observed as "inconsistent".

It is definitely one of the stranger profiles to understand and tune, but it sounds to me like if you keep your current setup and just add a /32 oneconnect, and it has a timeout that is, say, about twice as long as what you would estimate as the average time between user requests with headers for DIFFERENT destinations, you'd probably see some performance gain.

On the other hand, you can just create new connections every time like you are now, if you have the memory to spare. As long as your tcp idle timeout is reasonable and connections get culled regularly, the impact on the box isn't serious (although your users will obviously experience the delay of a tcp + tls handshake every time the headers switch)

[_JOHN_]

Thanks again for the update 🙂

Regarding the mask - I had left it at default because the VS uses a SNAT pool with a single IP. My understanding is that oneconnect examines the post SNAT source IP, so essentially the mask has no impact for my setup? I did try with /32 just to check but saw no difference.

As for my question on what 'level' the oneconnect reuse pool operates at - looking at the help section in the oneconnect profile page on the F5 GUI it looks like it is at the VS level. I think that will likely cause me a problem for the current setup (not tied to that setup though, happy to change it :-). I use different pools so that an iRule can assign a server SSL profile based on a 1-1 mapping between the pool name and the SSL profile to use.

To me the following is what I _suspect_ is happening:

- Initial traffic sent to pool A and server side connection setup correctly using server SSL profile A

- When HTTP header change happens, the traffic policy instead chooses pool B

- Oneconnect kicks in at this point and sees that the required connection is from SNAT IP to website IP port 443. This is on pool B but that was chosen by the VS with the oneconnect profile, and the mask covers the source IP OK, so this connection is eligible to be switched onto a reused TCP connection.

- At the TCP level i.e. IP/port this new connection request in pool B exactly matches the existing connection from pool A. Also at this point the server SSL profile would match, because I can't assign that in the iRule until a serverside event (please correct me if I have that wrong). This is why I asked if oneconnect takes the likes of SSL into account to differentiate between connections or if it is just on the lower level protocol.

- Oneconnect switches the traffic onto the existing connection, and the SERVER_CONNECTED event never triggers, which is where I would have assigned server SSL profile B

So basically all my traffic is just going over that single connection, and any time that HTTP Header X = B the traffic is actually sent on the 'wrong' pipe.

I can see how oneconnect would work if I could somehow confine it to a pool rather than a VS. Essentially I would then just use it to keep a connection up per pool, and I would know that each connection had been established using the correct SSL profile, so any traffic directed to the pool by the traffic policy would definitely be sent across a 'correct' pipe. Is this possible?

[Josiah]

Sorry, there is a lot to clarify and explain and possibly correct in that and I think we are getting beyond the scope of DevCentral. As I mentioned before, the name, usage, and configuration of a oneconnect profile is confusing and a little unintuitive. If you are expecting to see "one" connection per pool, you will definitely think it is behaving incorrectly, because it is not a "tunnel" or a "pipe" like you are saying. It is a tcp connection and it will never be used for multiple requests at the same time, it must be idle or a new connection will be created. In fact, some people put the default oneconnect profile on vips hoping for a "one click" performance improvement only to find they now have more serverside connections than before (especially if their non-oneconnect vip was layer 4 or was doing pipelining etc).

Ultimately, there is only so much that can be done here. If you are able to get something that works for you taking this conversation (and others here) into consideration, great. If not, you should probably work with someone with whom you can share specific config details and pcaps and get personalized consulting that should not be done over public internet.

[_JOHN_]

Thanks again for the reply.

Apologies for the confusion in the terms that I used. Please note that I do not expect multiple clientside TCP connections to be multiplexed onto a single serverside TCP connection. I was saying one serverside connection per pool because I was focusing on the simple example being discussed – a single clientside TCP connection which in my case should result in 2 completely separate TCP connections serverside – one per pool (one ACTIVE, one IDLE, assuming IDLE hadn’t timed out yet). This same connection layout would then scale up as new concurrent clientside connections were activated. 

In fact in my case the traffic profile is sort of round the other way to how oneconnect is normally used. Depending on traffic volume/profile I would generally expect to see more serverside connections than clientsdie in my specific case, whereas the normal deployment would generally result in fewer serverside connections compared to clientsdie.

I used the term “pipe” because the simple term TCP connection seems a little to vague to me. All the oneconnect documentation I have read simply mentions TCP connections, but as I see it these connections are much more than that. They are fully established connections using higher level protocols ready to take Layer 7 HTTP request/response messages. In my case the important piece of info that the term “TCP connection” leaves out is regarding SSL. So I should maybe use (TCP+SSL-A) and (TCP+SSL-B). At the TCP level these two connections will look exactly the same, but if you take the higher level protocol stack into consideration they are actually very different connections (what I meant by “pipe” – I realise though that I said it with no explanation/context so it will have been very confusing, apologies).

It feels to me like oneconnect is working solely at the TCP level and ignoring the higher level components. This would work fine for a normal setup where the Virtual Server had just one server SSL profile – however in my case I use different server SSL profiles which it feels like oneconnect may simply ignore.

However I realise I probably have this wrong, as a more normal scenario would be the F5 acting as reverse proxy in front of a number of internal webservers which host multiple domains and use SNI to confirm which connection aligns with which FQDN. I assume oneconnect must have a way to cope with this situation and if I understood that operation then elements of it would likely be relevant in my case.

Anyway I see that this is getting too complicated to bottom out in the format of this forum. Thank you for taking the time to try to help though, it is greatly appreciated 😊