https not working with http profile

Any chance that you have some custom settings in your server SSL profile? Take a look at your LTM log while testing:

tail -f /var/log/ltm

Do you see any references to RFC 5746 and secure renegotiation? Any chance that the SSL configuration on the IIS8 server is different than the IIS6?

Worst case you can do the following:

1. Run a TCPDUMP on the internal (web server facing) interface. If you see traffic going to the web server (but failing), then you're pretty certain that the client SSL settings are good.

2. Run an SSL dump on the internal (web server facing) interface. More information on how to set that up is below. You're looking for specific SSL-related anomalies. If there's a problem with the SSL negotiation, you should see that in the capture.

https://devcentral.f5.com/articles/troubleshooting-tls-problems-with-ssldump.UmV4-xCK7wQ

Thanks. I will try to sun SSL dump. I do see traffic going to web server. When run tcp dump, I see F5 reset connection. Also another interesting thing is that, if I config without SSL server profile and let F5 offload SSL it work even with http profile on VIP. Here the error message I got.

http_process_state_prepend - Invalid action EV_INGRESS_DATA during ST_HTTP_PREPEND_HEADERS (Server side: vip=VIP_TEST_SSL profile=http pool=Pool_Test_SSL)

if I config without SSL server profile and let F5 offload SSL it work even with http profile on VIP.

That is interesting. So just out of curiosity, are you sending to a port 80 pool of servers, or any chance that IIS8 is binding to port 443 but not doing SSL?

1. with http profile- ssl client and server is applied and backend server doing ssl ---- it didn't work a. without http profile --- it work

    

2. with http profile- ssl client profile -- backend server is on port 80 ---- it work (no ssl server profile)

    

Are there any HTTP iRules applied to the VIP? Is the IIS8 server by chance configured differently than the IIS6 server? I see an application_data entry from the client to the server in the SSLDUMP capture, so it looks almost like the SSL handshake finishes, the client sends a request to the server, and the server resets. If you look back at that SSLDUMP reference it'll show you how to insert the IIS8 server's private key into the capture to decrypt the traffic. It might be interesting to see what that application_data is.

Expanded for readability:

New TCP connection 1: 10.1.1.x(53963) <-> 10.13.10.x(443) 
1 1 0.1468 (0.1468) C>S Handshake 
    ClientHello 
    Version 3.3 
    cipher suites 
        Unknown value 0xc00a 
        Unknown value 0xc014 
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA 
        Unknown value 0x6b 
        TLS_RSA_WITH_AES_256_CBC_SHA 
        Unknown value 0x3d 
        Unknown value 0xc007 
        Unknown value 0xc009 
        Unknown value 0xc023 
        Unknown value 0xc011 
        Unknown value 0xc013 
        Unknown value 0xc027 
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA 
        TLS_DHE_DSS_WITH_NULL_SHA 
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA 
        TLS_RSA_WITH_RC4_128_SHA 
        TLS_RSA_WITH_RC4_128_MD5 
        TLS_RSA_WITH_AES_128_CBC_SHA 
        Unknown value 0x3c 
        TLS_RSA_WITH_3DES_EDE_CBC_SHA 
    compression methods NULL 
1 2 0.1468 (0.0000) S>C Handshake 
    ServerHello 
    Version 3.1 
    session_id[32]= 
        2b 92 da 9e d0 37 46 32 a9 97 92 8e 77 12 68 96 
        82 41 cc b5 1f 23 77 d5 66 b3 0e 41 23 03 a7 a2 
    cipherSuite 
        TLS_RSA_WITH_RC4_128_SHA 
    compressionMethod NULL 
1 3 0.2942 (0.1473) S>C Handshake 
    Certificate 
1 4 0.2942 (0.0000) S>C Handshake 
    ServerHelloDone 
1 5 0.4452 (0.1509) C>S Handshake 
    ClientKeyExchange 
1 6 0.4452 (0.0000) C>S ChangeCipherSpec 
1 7 0.4452 (0.0000) C>S Handshake 
1 8 0.4482 (0.0029) S>C ChangeCipherSpec 
1 9 0.4482 (0.0000) S>C Handshake 
1 10 0.6744 (0.2262) C>S application_data 
1 0.8723 (0.1978) S>C TCP RST

Is this a client side or a server side capture? The listed cipher suites makes me think this is a client side capture. Please verify that you're listening on the correct interface and use the IIS8 server's private key to decrypt the data.

The redirect rewrite option in the HTTP profile is designed to catch 30x redirect responses and change them to the correct protocol (ie. HTTP references changed to HTTPS). The fact that altering this setting has an effect on your application somewhat suggests an application layer issue (not SSL). While it's still not totally clear what's going on, I think the next step is to get a client side HTTP capture (using something like Fiddler or IEWatch). You're looking for any responses that contain references to itself by an incorrect (potentially local) name or using HTTP:// instead of HTTPS://.