SSL Forward proxy Bypass iRule not working

On of the primary issues here is timing. In your iRule, you're redirecting traffic to the bluecoat pool in the CLIENTSSL_SERVERHELLO_SEND event. Presumably you have another pool that is default for traffic.

When a client enters the SSL forward proxy VIP, he'll initiate the SSL handshake.

Client Hello

which is then paused. At that point, the server side goes out and performs a complete server side TCP connection and server side SSL handshake with the server, brings the server's cert back, and only then does the client side SSL session resume.

client side server hello send

So by the time you get to the CLIENTSSL_SERVERHELLO_SEND event, an initial server side connection has to have already left through the default path, so setting the pool in this event is too late. Try this:

when CLIENTSSL_CLIENTHELLO {
    set sni_exists [SSL::extensions exists -type 0]
    if { $sni_exists } {
        binary scan [SSL::extensions -type 0] @9a* tls_servername
    }
    if { [class match $tls_servername contains Bypass] } {
        pool bluecoat-pool
    }
}
when CLIENTSSL_SERVERHELLO_SEND {
    if { [class match $tls_servername contains Bypass] } {
        SSL::forward_proxy policy bypass
        catch { HTTP::disable }
    }
}