TCP RST instead of Server Hello during SSL Handshake

That's a pretty unusual cipher list. Just to put it into perspective:

C>SV3.1(114) Handshake ClientHello 
    Version 3.1 
    random[32]= 56 1e 0a ea e4 11 03 df d1 77 92 83 da ec 1d 44 21 65 c2 20 97 25 40 53 75 d6 e5 c2 6b 1d 96 65 
    cipher suites 
        TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA 
        TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA 
        TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA 
        TLS_RSA_WITH_CAMELLIA_256_CBC_SHA 
        Unknown value 0x46 (TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA)
        Unknown value 0x45 (TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA)
        Unknown value 0x44 (TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA)
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 
        TLS_DH_anon_WITH_AES_256_CBC_SHA 
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA 
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA 
        TLS_RSA_WITH_AES_256_CBC_SHA 
        TLS_DH_anon_WITH_AES_128_CBC_SHA 
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA 
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA 
        TLS_RSA_WITH_AES_128_CBC_SHA 
        TLS_DH_anon_WITH_3DES_EDE_CBC_SHA 
        TLS_DH_anon_WITH_DES_CBC_SHA 
        TLS_DH_anon_EXPORT_WITH_DES40_CBC_SHA 
        TLS_DH_anon_WITH_RC4_128_MD5 
        TLS_DH_anon_EXPORT_WITH_RC4_40_MD5 
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA 
        TLS_DHE_RSA_WITH_DES_CBC_SHA 
        TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA 
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA 
        TLS_DHE_DSS_WITH_DES_CBC_SHA 
        TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA 
        TLS_RSA_WITH_3DES_EDE_CBC_SHA 
        TLS_RSA_WITH_DES_CBC_SHA 
        TLS_RSA_EXPORT_WITH_DES40_CBC_SHA 
        TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 
        TLS_RSA_WITH_RC4_128_SHA 
        TLS_RSA_WITH_RC4_128_MD5 
        TLS_RSA_EXPORT_WITH_RC4_40_MD5 
    Unknown value 0xff (TLS_EMPTY_RENEGOTIATION_INFO_SCSV)
    compression methods unknown value NULL

S>C TCP RST

The "unknown" cipher values can be derived from: http://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml.

In any case, since the server is sending an immediate reset after the client's ClientHello, the FIRST message in the SSL handshake, it may indicate a few things:

1. The server isn't actually doing SSL

2. The server doesn't support any of the ciphers from the client's list

What do you have in the HTTPS monitor's Cipher List? And what cipher string are you using the server SSL profile?

Possibly the best way to troubleshoot this is by simply trying to connect to the server from the BIG-IP command line using openssl s_client

openssl s_client -connect [server IP:port]

You should see what cipher is selected in this transaction (if it works at all). You can then go through your above cipher list and try each in turn with the -cipher option

openssl s_client -connect [server IP:port] -cipher 'DHE-RSA-CAMELLIA128-SHA'

Or even better, you can determine which ciphers are supported with this handy little script http://www.tuxad.de/scripts/ssltest.sh