iRule to return message when SSL handshake fail

How can you return a message to the client when the transport protocol negotiation has failed?

You need to accept the TLS connection, and then reply to the HTTP_REQUEST with your required response based on the

SSL::cipher version

command.

when HTTP_REQUEST {
  log local0. "Cipher version is [SSL::cipher version]"
  if {!([SSL::cipher version] eq "TLSv1.2")} {
    HTTP::respond 500 content "You need to upgrade your client to support TLSv1.2"
    TCP::close
  }
}

However - this will be detected by external scanners (like SSLLabs) as supporting and allowing versions of SSL/TLS below TLSv1.2, and your site rating will probably be an F.