HTTPS traffic taking 22 seconds on first response and 4 seconds on second.

I have 2 VS. One for http traffic 10.50.171.8:80 and one for https Traffic 10.50.171.8:443. SSL traffic is offloaded on F5. I have a self sign certificate. From F5 to servers traffic is http.

 

Now the problem I am facing is that when upstream machines dials http://10.50.171.8:80 it takes few ms to reach the server in the pool. but when machine dials https://10.50.171.8:443 it takes 22 seconds in response.

 

I removed the certificate and applied default certificate of F5 still the connections are established but still taking 22 seconds to same pool to which http is taking ms. I saw the statistics and found something like this. Can someone tell me what is the problem.

 

2nd request takes 4 seconds.