What is this iRule doing?

I've added some comments to the iRule so that you can see what it's doing.

when CLIENTSSL_HANDSHAKE {
     This event is triggered at the end of the client side SSL handshake

    if {[SSL::cert count] > 0} {
         The [SSL::cert count] command returns how many certificates were returned. In this case you're apparently doing mutual authentication with client certificates, so the command returns the number of certificates returned by the client during the mutual SSL handshake. A client can send its certificate, plus (optionally) its issuer and any additional issuers in the PKI chain.

        log local0. "X509 Certificate present"

        set subject_dn [X509::subject [SSL::cert 0]]
         Now that you know you have a client certificate, grab the X509 certificate subject from the first certificate [SSL::cert 0] (index of certs starts at 0).

        set cert_hash [string toupper [X509::hash [SSL::cert 0]]]
         Grab the X509 hash value inside the client cert and change it to all uppercase.

        set serial_dn [X509::serial_number [SSL::cert 0]]
         Get the X509 cert serial number

        set cSSLSubject [findstr $subject_dn "CN=" 0 ","]
         Findstr will allow you to abstract a piece of a string. In this case you're taking the cert subject that you got earlier, which is a longer distinguished name value (ex. CN=bob,ou=foo,o=bar,c=US), looking for the "CN=" value, and then grabbing everything after that up to the first comma. Based on the example, it would return "CN=bob". Consequently, if you just wanted to return "bob", you could put a skip parameter in the findstr command: [findstr $subject_dn "CN=" 3 ","].

        log local0. "Subject = $cSSLSubject, Hash = $cert_hash, Serial = $serial_dn"

        set Expected_hash [string toupper [class lookup $cSSLSubject authGroup]]
         Now you're going to look in a data group called "authGroup" for any entries that match the cSSLSubject variable (ex. CN=bob). If there's a match it'll return the corresponding "expected hash" - the value in the data group associated with that key.

        if { $Expected_hash != $cert_hash } {
             if the hash in the data group doesn't match the hash grabbed form the cert, log and reject the connection. This is a quasi-security mechanism that allows you to match a "registered" cert to an incoming cert.
            log local0. "Thumbprint mismatch. Expected Hash = $Expected_hash, Hash received = $cert_hash (Subject = $cSSLSubject, Serial = $serial_dn)"
            reject
        } else {
            log local0. "Thumbprint match. Expected Hash = $Expected_hash, Hash received = $cert_hash (Subject = $cSSLSubject, Serial = $serial_dn)"
             do nothing - let traffic through
        }
    } else {
        log local0. "X509 Certificate not present"
         There was no client certificate so reject the connection.
        reject
    }
}