SSL error "record_overflow"

it seems this is an error msg that can show up when connecting with ssl to a HTTP service. Are the pool members and backend servers properly configured?

Which side is sending the reset? A record overflow is usually when one side sends more data than the peer can handle, for example when a server sends its certificate and a very long list of subordinate CAs in its ServerHello message.

If you do an openssl s_client from the BIG-IP shell you should be able to see what's happening:

openssl s_client -connect [server ip:port]

I'd also run an ssldump to watch data plane traffic between the BIG-IP and server:

ssldump -AdNn -i 0.0 port 443 and host [ip of server]

Yes, but:

1. Who is sending the reset?

    

2. Do an ssldump and watch the actual data plane SSL handshake. What do you see? Is the server's response too big?

    

Okay, so a few more questions.

You mentioned the backend servers are also doing SSL. Does that mean you're decrypting at the F5 with a client SSL profile and re-encrypting to the servers with a server SSL profile?

If so, is the SSL error happening on the client side (client to F5) or server side (F5 to server)?

Do you have secure renegotiation disabled in either of the SSL profiles?

Okay, so the error is happening on the client side and the F5 is sending the reset (and I'm assuming the SSL error). So is this a persistent error, or intermittent? Just for you or all users? Browers and cURL/OpenSSL s_client? The old SSL renegotiation attack could generate this error, but you'd be seeing a lot of SSL traffic hitting the box. Barring an unforeseen bug in that platform and version, the two main reasons to get that error are generally too many SSL renegotiation attempts or simply too much information in a given handshake message. Are you doing client certificate (mutual PKI) authentication?