How Proxy SSL works on BIG-IP

1. Lab Scenario

Lab test results:

Client completes 3-way handshake with BIG-IP and BIG-IP immediately opens and completes 3-way handshake with back-end server
Upon receiving Client Hello on client-side, BIG-IP immediately sends Client Hello on server-side as it is
BIG-IP copies same cipher suite list seen on client-side Client Hello to server-side Client Hello
BIG-IP ignores what is configured on both Client SSL or Server SSL profiles.
As soon as BIG-IP receives Server Hello it confirms two things:
Is RSA the chosen key exchange mechanism in Cipher Suite on Server Hello message sent from back-end server?
Does Certificate sent by back-end matches the one configured on Server SSL profile on BIG-IP?
ONLY if both conditions above match BIG-IP proceeds with handshake.
Otherwise, BIG-IP sends a Handshake Failure (Fatal Alert) to both Server and Client with reset cause of illegal_parameter
BIG-IP copies same Server SSL/Back-end Server certificate to Certificate message sent to Client on client-side
BIG-IP completely ignores certificate you configured on Client SSL. It always uses the same server-side certificate.
Assuming TLS handshake completes successfully BIG-IP is able to decrypt all client-side as well as server-side data which is the whole purpose of Proxy SSL.

2. How Proxy SSL works

When Proxy SSL is enabled, BIG-IP does its best to match client-side to server-side connection in terms of negotiation and traffic to make it as transparent as possible to both client and back-end server and at the same allowing BIG-IP to decrypt traffic.

About data transparency

BIG-IP achieves transparency by copying whatever client and server sends back and forth.

About data decryption

BIG-IP has an extra configuration requirement for Proxy SSL configuration (according to K13385) that you should add the same certificate/key present on the back-end server to Certificate/Key fields on Server SSL proxy of BIG-IP. This way BIG-IP can decrypt both client and server sides of connection.

In practical terms and to achieve this, BIG-IP completely changes the original purpose of Server SSL Certificate and Key fields. Here's my config:

Certificate/Key fields above are no longer used for the purpose described in K14806, i.e. to independently authenticate BIG-IP to back-end server through Client Authentication. Instead, when Proxy SSL is enabled it is used to validate if Certificate sent by back-end server is the same one in this field. If so, BIG-IP also copies this certificate to Certificate message sent to Client on client-side.

Notice that when Proxy SSL is NOT bypassed Certificate configured on Client SSL profile is never used.

As soon as I sent first request to Wildcard forwarding VIP BIG-IP establishes TCP connection on client-side first and then immediately on server-side:

The next step is to forward exactly the same Client Hello we receive from Client on client-side to Server on server-side:

Now server sends Server Hello:

The first thing BIG-IP checks is key exchange mechanism as Proxy SSL has to use RSA (frame 12):

Now BIG-IP checks if Certificate inside Certificate message is the same as the one configured on Server SSL.

In this case I have used Certificate's unique serial number to confirm this.

On BIG-IP:

Now on Server's message (frame 12):

Now BIG-IP sends this same certificate in Server Hello message client-side and we can confirm from Serial number that it is the same:

From this moment handshake should complete successfully and BIG-IP is maintaining 2 separate connections using the same certificate/key pair on client and server side with the ability to decrypt both.

3. Troubleshooting Proxy SSL when BIG-IP is the culprit

Typically, if BIG-IP is the culprit it either because back-end server selected non-RSA key exchange cipher or because cert/key which are not supported.

In another test I used DHE on purpose and BIG-IP resets connection immediately after Server Hello message is received from back-end server which is typical sign of validation error: