Forum Discussion

Sakiy's avatar
Sakiy
Icon for Altocumulus rankAltocumulus
Oct 21, 2024

LTM SSL handshake failuer (40) with IIS SSL setting Accept

I had an issue that communication from client PC failed with one of pool members. Clinet PC can directly access to the problem member without any issue. If it is accessed through VS, the failure happened.

 

As investigated with packet capture, following error caused the communication failure.

 

Transport Layer Security
    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Handshake Failure)
        Content Type: Alert (21)
        Version: TLS 1.2 (0x0303)
        Length: 26
        Alert Message
            Level: Fatal (2)
            Description: Handshake Failure (40)

As I investigated, I found the problem member's IIS SSL setting is set as "Accept". Other working members are set as "Ignore". As I changed the setting to "Ignore", the problem was gone.

 

The IIS SSL setting "Accept" is to accept clinet certificate if it is provided by client. If client did not provide client cetificate, IIS still establish connection. On the VS, SSL server profile is used. the profile setting is almost default.

Do you know why BIG-IP fails the SSL communication if the IIS SSL setting is "Accept"?

 

LTM
ssl
    • Sakiy's avatar
      Sakiy
      Icon for Altocumulus rankAltocumulus
      Oct 22, 2024

      We only configure server certificate on SSL client profile. No client cert is set.

  • BellaAbzug1's avatar
    BellaAbzug1
    Icon for Nimbostratus rankNimbostratus
    Oct 21, 2024

    An LTM SSL handshake failure (40) indicates a problem during the SSL/TLS handshake process between the Local Traffic Manager (LTM) and an IIS server. This issue often arises when there’s a mismatch in SSL settings, such as protocol versions or cipher suites. If the IIS SSL setting is set to Accept, it might be allowing weaker protocols, which could cause compatibility issues with LTM. To resolve this, ensure that both the LTM and IIS are configured to support the same SSL/TLS protocols and cipher suites. Additionally, checking the server certificates for validity and proper installation can help eliminate handshake failures.

    • Sakiy's avatar
      Sakiy
      Icon for Altocumulus rankAltocumulus
      Oct 22, 2024

      Thank you for your reply.
      I opened the case with F5 support also. I got following reply. It was indicate the peer-cert-mode on SSL server profile caused the issue when IIS SSL setting is Accept.  But I am not sure why it cause the handshake failure.

      # tmsh list /ltm prof.ile server-ssl all-properties

      [...]

      ltm profile server-ssl /Common/XXX_XXX {
          alert-timeout 4294967295
          app-service none
          authenticate once
          authenticate-depth 9
          authenticate-name none
          ca-file none
          cache-size 262144
          cache-timeout 3600
          cert none
          chain none
          cipher-group none
          ciphers DEFAULT:@STRENGTH:!RC4-SHA:!DES:!3DES:!NONE:!SSLv3:!TLSv1
          crl-file none
          defaults-from /Common/serverssl
          description none
          expire-cert-response-control drop
          generic-alert enabled
          handshake-timeout 10
          key none
          mod-ssl-methods disabled
          mode enabled
          options { dont-insert-empty-fragments no-tlsv1.3 }
          partition Common
          passphrase ***scrubbed***
          peer-cert-mode ignore   <<<<==============
          proxy-ssl disabled
          proxy-ssl-passthrough disabled
          renegotiate-period indefinite
          renegotiate-size indefinite
          renegotiation disabled
          retain-certificate true
          secure-renegotiation require-strict
          server-name none
          session-mirroring disabled
          session-ticket disabled
          sni-default false
          sni-require false
          ssl-forward-proxy disabled
          ssl-forward-proxy-bypass disabled
          ssl-sign-hash any
          strict-resume disabled
          unclean-shutdown enabled
          untrusted-cert-response-control drop
      }