Forum Discussion
seilemor_131269
Altostratus
AltostratusSSL errno 104 through F5 (vip), directly with curl ok
Hello community,
I've the following configuration design within a virtual server configuration:
- A virtual server will route the traffic based on the hostname within the request to different pools.
- Client- and server SSL are enabled
- The routing will been done with LTM policys instead of iRules.
- There are five backend systems which should be accessible over the vs
- SNAT is enabledNow, the problem is that from the five backends are only three backend systems are accessible. The other two systems don't work. Within the LTM policy I've additional enabled the "log" option to make sure that the routing will work.
To find out what is going wrong, I've executed a curl and openssl query direct to the backend system from the F5 console. Connection can be established and I receive the 200 status code. If I do the same over the VIP of the configuration I receive the following error code:
read:errno=104These are my teststrings:
openssl s_client -connect vip:443
GET /dialin/ HTTP/1.1
Host: lyncpool1 or 2 or 3 ...With lyncpool1 as the value for the host it will work and with lyncpool2 I receive the error code from above. The client SSL settings from the vip will been displayed. It seams that there is an issue/problem while talking SSL with the backend system which will not work. If I do the same with the original IP of the system instead of the VIP, each system are working fine.
Have anybody an idea? I've just readed some threads within devcentral to similar problems but nothing helped my until now.
Regards seilemor
- Kai_Wilke
MVP
Hi Seilemor,
CURL, the F5 Monitors and the Virtual Servers are using independent SSL settings. So its not uncommon that one method may work and the other doesn't...
To troubleshoot your issue further, you may enable LTM SSL Debug logging to see if any SSL related errors are getting raised. Go to Logs\Configuration\Options and then set SSL Logging to "DEBUG". Then take a look to the Local Traffic logfile...
Note: You may also temporary switch the Server SSL profile to "serverssl" or even "serverssl-insecure-compatible" to identify server side SSL problems. If those generic SSL profiles are working for you, then you have alredy identify the cause (aka. custom SSL settings on each individual pool member.
Cheers, Kai
seilemor_131269Hey Kai,
thanks for the very fast answer. I already activaed the standard serversll profile to avoid problems. After activating the SSL debug level I see the following error within "\var\log\ltm"
Aug 29 12:44:01 o01006670 info tmm2[5320]: 01260013:6: SSL Handshake failed for TCP from 172.21.x.x:45218 to 10.235.x.x:4443The 172.21.x.x IP address is the floating one from the F5 cluster. The IP address 10.235.x.x are the backend system.
Ok - the handshake failed. What now?
Regards seilemor
- Kai_Wilke
Hi Seilemor,
the next step would be to activate the "serverssl-insecure-compatible" Profile and to TCPDUMP the SSL handshake between your Floating IP and the individual Backend servers and then inspect/compare the collected SSL handshakes using e.g. WireShark.
tcpdump -i YOUR_VLAN_LABEL -w /var/tmp/capture.cap dst host 10.235.x.x and dst port 4443Cheers, Kai
- seilemor_131269
Ok - I've used the article "how to use ssldump utility" for the next steps. After performing tcpdump and ssldump I will receive the following lines:
For description:
- x.x.x.200 is the non floating IP address which execute the monitor check. - x.x.x.201 is the floating IP address for the regular requestWithin the connection of the floating IP address I see just nothing. I already used the insecude SSL server profile, too. To check the system I've also reconfigured the vip to use the non nloating IP (x.x.x.200).
New TCP connection 2: 172.21.254.200(61407) <-> 10.235.96.29(4443) 2 1 0.1926 (0.1926) C>S Handshake ClientHello Version 3.3 cipher suites TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_DES_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 Unknown value 0xc013 Unknown value 0xc014 Unknown value 0xc012 Unknown value 0xff compression methods NULL 2 0.5735 (0.3808) S>C TCP FIN 2 0.5736 (0.0001) C>S TCP RST New TCP connection 3: 172.21.254.200(59346) <-> 10.235.96.29(4443) 3 1 0.1939 (0.1939) C>S Handshake ClientHello Version 3.3 cipher suites TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_DES_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 Unknown value 0xc013 Unknown value 0xc014 Unknown value 0xc012 Unknown value 0xff compression methods NULL New TCP connection 4: 172.21.254.201(1978) <-> 10.235.96.29(4443) 4 0.2052 (0.2052) C>S TCP FIN 3 0.5759 (0.3820) S>C TCP FIN 3 0.5760 (0.0000) C>S TCP RST 4 0.5828 (0.3776) S>C TCP FIN New TCP connection 5: 172.21.254.200(34721) <-> 10.235.96.29(4443) 5 1 0.1934 (0.1934) C>S Handshake ClientHello Version 3.3 cipher suites TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_DES_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 Unknown value 0xc013 Unknown value 0xc014 Unknown value 0xc012 Unknown value 0xff compression methods NULL 5 0.5740 (0.3806) S>C TCP FIN 5 0.5741 (0.0001) C>S TCP RST New TCP connection 6: 172.21.254.200(46619) <-> 10.235.96.29(4443) 6 1 0.1980 (0.1980) C>S Handshake ClientHello Version 3.3 cipher suites TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_DES_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 Unknown value 0xc013 Unknown value 0xc014 Unknown value 0xc012 Unknown value 0xff compression methods NULL 6 0.5788 (0.3807) S>C TCP FIN 6 0.5789 (0.0001) C>S TCP RST New TCP connection 7: 172.21.254.200(3858) <-> 10.235.96.29(4443) 7 1 0.1927 (0.1927) C>S Handshake ClientHello Version 3.3 cipher suites TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_DES_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 Unknown value 0xc013 Unknown value 0xc014 Unknown value 0xc012 Unknown value 0xff compression methods NULL 7 0.5746 (0.3818) S>C TCP FIN 7 0.5747 (0.0001) C>S TCP RST- Kai_Wilke
Hi Seilemor,
sorry my fault. Change the TCPDUMP filters to the example below. It will allow you to capture bidirectional traffic...
tcpdump -i YOUR_VLAN_LABEL -w /var/tmp/capture.cap host 10.235.x.x and port 4443But anyhow. In your last capture the F5 Floating hasn't even send a Client Hello to the problematic servers. But on the other hand the F5 Floating was able to send a Client Hello (and successfully negotiate a SSL connection) to the remaining servers using the identical pool and SSL Profiles?
Thats a somewhat strange behavior... 😞
Cheers, Kai
- seilemor_131269
Hey Kai,
I have executed the tcpdump, which was the base for the ssldump, without the "dst" option to receive also the answer packages. That the floating IP does not send correctly the clienthello I've also noticed... sadly I have no answer for these situation :(
What I've also checked is what happen if I create a virtual server with only the problem server and without policy routing. With this configuration I also have no success.
- Kai_Wilke
I guess this is more a node specific problem then? Configuration differences? Wrong Routing Tables? Bad SSL Chiphers? Some FW access-lists in transit? To get sure, I'd like to recomment to TCPDUMP directly on the affected nodes (or use a SPAN port on your switches)...
Cheers, Kai
- seilemor_131269
here comes the next frustrating thing: I've tested the configuration within my lab version with the result that it work there how it was designed. The lab version is installed with version 12 and the productive system has 11.4.1
- Kai_Wilke
Oh gosh... its getting even more weird. :-( But keep in mind, that the Lab and Live environment are not using the same Source IPs. I'd like to recommend to check the intermediate routers / firewall systems in place for missing configurations... :-(
Cheers, Kai
- seilemor_131269
Uh, my fault: the IP address x.x.x.200 is the floating one, the 201 is the non floating which execute the monitor checks.
I've now disabled the monitor and executed again the tcpdump and ssldump. Following the output which will displayed 10 times:
New TCP connection 1: 172.21.254.200(12938) <-> 10.235.96.29(4443) 1 1 0.2097 (0.2097) C>S Handshake ClientHello Version 3.3 cipher suites TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 Unknown value 0xc013 Unknown value 0xc014 Unknown value 0xc012 Unknown value 0xff compression methods NULL 1 0.5910 (0.3812) S>C TCP FIN 1 0.5911 (0.0001) C>S TCP RST - seilemor_131269
Now I have somehting positive: if I change the cipher within the monitor check to "DEFAULT:+SHA:+3DES:+kEDH", the monitor checks are working. Within a tcpdump/ssldump I see some client and server hello.
The problem: I can not use these cipher list within the SSL server profile. If I try to use it I receive the error message that the value "kEDH" is unknown.
I think that the problem between the F5 and the Backend are the ciphers which can be used and which can not be used. Additional I think that the normal openssl stack which can executed within a normal console is not the same as the stack which will been used within the bigip daemon self.
With a script I've checked which ciphers are provided from the backend system. Here is the output:
Cipher order SSLv3: DES-CBC3-SHA RC4-SHA RC4-MD5 TLSv1: ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES256-SHA DHE-RSA-AES128-SHA AES256-SHA AES128-SHA DES-CBC3-SHA RC4-SHA RC4-MD5 TLSv1.1: ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES256-SHA DHE-RSA-AES128-SHA AES256-SHA AES128-SHA DES-CBC3-SHA RC4-SHA RC4-MD5 TLSv1.2: ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-SHA DHE-RSA-AES128-SHA AES256-GCM-SHA384 AES128-GCM-SHA256 AES256-SHA256 AES128-SHA256 AES256-SHA AES128-SHA textDES-CBC3-SHA RC4-SHA RC4-MD5I've tested a little bit around with the cipher lists which the F5 can use (native, compat, default, ...). But sadly nothing has solved my issue.
- Kai_Wilke
You may read the SOL17270 to get the proper syntax of LTM chipher suites...
https://support.f5.com/kb/en-us/solutions/public/17000/300/sol17370.html
... but I'm somewhat sure this won't solve your issues, since LTM isn't even sending a CLIENT_HELLO to your backend servers, isn't it?
Cheers, Kai
- seilemor_131269
I've changed the cipher like it was described in the KB article. Unfortunately it has also not solved the problem.
After I've disabled the monitor of the pool I can see Client Hello messages if I try to reconnect to the site. The result/output from the tcpdump is somewhere in the answers above.
- Kai_Wilke
Hi Seilemor,
unfortunately I can't help you further unless you start to TCPDUMP/SSLDUMP on the application servers.
Cheers, Kai
- seilemor_131269
Hey, I will perform an tcpdump on the application server today at 2pm...
- seilemor_131269
hmmm... now I have the tcpdump from the application server. Sadly I can not see anything. If I put the pcap file into the ssldump utility I can also only see the client hello commands from the F5 :(
- seilemor_131269
Ok - here are some new tcpdumps. On the first screenshot you can see the tcpdump if I perform manually the command
openssl s_client -connect 10.235.96.29:4443The second screenshot is the tcpdump if I execute the test through the F5 vip. The connection try including only five packages. After the F5 receive the RST, ACK from the application server the F5 start a new try.
The only thing which I can see at the moment is that the length of the 4th package is different.
- seilemor_131269
Ok - I've solved the problem. I've disabled TLS features within the server SSL profile. I think it is a "feature" of the old 11.4.1 version which we are use. I'll now plan a update of the system to the current BigIP version.
Thanks for your support and your helpful comments ;)
- Kai_Wilke
Hi seilemor,
good to hear that you've managed to solve your issue! ;-)
Cheers, Kai
