Client completes 3-way handshake with BIG-IP and BIG-IP immediately opens and completes 3-way handshake with back-end server
Upon receiving Client Hello on client-side, BIG-IP immediately sends Client Hello on server-side as it is
BIG-IP copies same cipher suite list seen on client-side Client Hello to server-side Client Hello
BIG-IP ignores what is configured on both Client SSL or Server SSL profiles.
As soon as BIG-IP receives Server Hello it confirms two things:
Is RSA the chosen key exchange mechanism in Cipher Suite on Server Hello message sent from back-end server?
Does Certificate sent by back-end matches the one configured on Server SSL profile on BIG-IP?
ONLY if both conditions above match BIG-IP proceeds with handshake.
Otherwise, BIG-IP sends a Handshake Failure (Fatal Alert) to both Server and Client with reset cause of illegal_parameter
BIG-IP copies same Server SSL/Back-end Server certificate to Certificate message sent to Client on client-side
BIG-IP completely ignores certificate you configured on Client SSL. It always uses the same server-side certificate.
Assuming TLS handshake completes successfully BIG-IP is able to decrypt all client-side as well as server-side data which is the whole purpose of Proxy SSL.
2. How Proxy SSL works
When Proxy SSL is enabled, BIG-IP does its best to match client-side to server-side connection in terms of negotiation and traffic to make it as transparent as possible to both client and back-end server and at the same allowing BIG-IP to decrypt traffic.
About data transparency
BIG-IP achieves transparency by copying whatever client and server sends back and forth.
About data decryption
BIG-IP has an extra configuration requirement for Proxy SSL configuration (according to K13385) that you should add the same certificate/key present on the back-end server to Certificate/Key fields on Server SSL proxy of BIG-IP. This way BIG-IP can decrypt both client and server sides of connection.
In practical terms and to achieve this, BIG-IP completely changes the original purpose of Server SSL Certificate and Key fields. Here's my config:
Certificate/Key fields above are no longer used for the purpose described in K14806, i.e. to independently authenticate BIG-IP to back-end server through Client Authentication. Instead, when Proxy SSL is enabled it is used to validate if Certificate sent by back-end server is the same one in this field. If so, BIG-IP also copies this certificate to Certificate message sent to Client on client-side.
Notice that when Proxy SSL is NOT bypassed Certificate configured on Client SSL profile is never used.
As soon as I sent first request to Wildcard forwarding VIP BIG-IP establishes TCP connection on client-side first and then immediately on server-side:
The next step is to forward exactly the same Client Hello we receive from Client on client-side to Server on server-side:
Now server sends Server Hello:
The first thing BIG-IP checks is key exchange mechanism as Proxy SSL has to use RSA (frame 12😞
Now BIG-IP checks if Certificate inside Certificate message is the same as the one configured on Server SSL.
In this case I have used Certificate's unique serial number to confirm this.
Now on Server's message (frame 12😞
Now BIG-IP sends this same certificate in Server Hello message client-side and we can confirm from Serial number that it is the same:
From this moment handshake should complete successfully and BIG-IP is maintaining 2 separate connections using the same certificate/key pair on client and server side with the ability to decrypt both.
3. Troubleshooting Proxy SSL when BIG-IP is the culprit
Typically, if BIG-IP is the culprit it either because back-end server selected non-RSA key exchange cipher or because cert/key which are not supported.
In another test I used DHE on purpose and BIG-IP resets connection immediately after Server Hello message is received from back-end server which is typical sign of validation error:
I confirmed back-end server had selected ECDHE key exchange cipher which is not supported by Proxy SSL (frame 12😞
In case you don't know yet here's how you work out key exchange cipher:
Disabling non-RSA cipher on back-end server did the trick to fix the above error as Proxy SSL only supports RSA key exchange.
4. Setting Up Proxy SSL on BIG-IP
I used very minimal configuration for this lab and the only thing I did was to create a wildcard forwarding virtual server using Standard VIP:
I enabled proxy-ssl on both Client and Server SSL profiles and added back-end server certificate (ltm3.crt) and key (ltm3.key) to Server SSL profile:
I also disabled (EC)DHE and explicitly configured RSA as key exchange mechanism in my back-end server.
I also confirmed back-end server was also using ltm3.crt/ltm3.key as it must match the one configured on Server SSL profile: