SNI and Client certificate authentication

This one is definitely unusual. For all intents and purposes, if the SNI-based profile selection is based on the CLIENTHELLO message, I would think the parts of the individual profiles shouldn't matter and certainly shouldn't have to match. I would also comment that trying to do anything in an iRule under the CLIENTSSL_ commands is too late in the process. For that reason I put together a potential workaround that relies on capturing the CLIENTHELLO server_name data in the (still unencrypted) TCP payload. Here's what it looks like:

when CLIENT_ACCEPTED {
    TCP::collect
}
when CLIENT_DATA {
    if { [info exists selected_profile] } { 
         use specified client SSL profile if set
        catch { eval "SSL::profile $selected_profile" } 
    } else {
         look for CLIENTHELLO message in the TLS handshake (SSL bypassed)
        binary scan [TCP::payload] cSSc tls_xacttype tls_version tls_recordlen tls_handshake_type
        switch -- $tls_version {
            "769" -
            "770" -
            "771" { 
                if { $tls_handshake_type equals 1 } {
                     CLIENTHELLO message
                    binary scan [TCP::payload] H* hexdump
                    if { [class match $hexdump contains my_ssl_profile_dg] } {
                         use client SSL profile from data group
                        set selected_profile [class match -value $hexdump contains my_ssl_profile_dg]
                        catch { eval "SSL::profile $selected_profile" }
                    }
                } 
            }
        }
    }
    TCP::release
    TCP::collect
}

Essentially what's happening is that I'm capturing the TCP payload and looking for a CLIENTHELLO message inside a TLS (1.0, 1.1, or 1.2) handshake. I toyed with a few different approaches, but for the sake of simplicity I hex-encoded the server names and added them to a data group with the corresponding client SSL profile name. Example:

6d79736572766572312e646f6d61696e2e636f6d := myserver1.domain.com_clientssl
6d79736572766572322e646f6d61696e2e636f6d := myserver2.domain.com_clientssl

You can get these values with the following code:

when RULE_INIT {
    binary scan "myserver1.mydomain.com" H* tmp
    log local0. $tmp
}

Another option would have been to loop through the literal server name strings in the data group and hex encode them on the fly. Ultimately, instead of trying to binary parse the CLIENTHELLO payload to find the exact server_name value, I'm simply looking for this particular hex string in the hex-encoded payload. Once I find and match the payload value to a data group entry, I extract and use the defined client SSL profile. The SSL::profile command isn't technically supported in the CLIENT_DATA event, so I wrapped in an eval command, and again in a catch command. I also set the profile name to a local variable so that subsequent requests in the same TCP session will use the same profile. You need at least one client SSL profile applied to the VIP, and all non-TLS-capable clients should simply fall through to this default profile.

The one thing it doesn't seem to support is flipping back and forth between the SSL VIPs in a single browser session. I don't imagine that'll happen often and I haven't figured out how to fix that yet. This workaround does seem to work well though. It also supports client SSL profiles with different client cert auth settings.