SSL error "record_overflow"

Which side is sending the reset? A record overflow is usually when one side sends more data than the peer can handle, for example when a server sends its certificate and a very long list of subordinate CAs in its ServerHello message.

If you do an openssl s_client from the BIG-IP shell you should be able to see what's happening:

openssl s_client -connect [server ip:port]

I'd also run an ssldump to watch data plane traffic between the BIG-IP and server:

ssldump -AdNn -i 0.0 port 443 and host [ip of server]