What is this iRule doing?

I've added some comments to the iRule so that you can see what it's doing.

when CLIENTSSL_HANDSHAKE {
     This event is triggered at the end of the client side SSL handshake

    if {[SSL::cert count] > 0} {
         The [SSL::cert count] command returns how many certificates were returned. In this case you're apparently doing mutual authentication with client certificates, so the command returns the number of certificates returned by the client during the mutual SSL handshake. A client can send its certificate, plus (optionally) its issuer and any additional issuers in the PKI chain.

        log local0. "X509 Certificate present"

        set subject_dn [X509::subject [SSL::cert 0]]
         Now that you know you have a client certificate, grab the X509 certificate subject from the first certificate [SSL::cert 0] (index of certs starts at 0).

        set cert_hash [string toupper [X509::hash [SSL::cert 0]]]
         Grab the X509 hash value inside the client cert and change it to all uppercase.

        set serial_dn [X509::serial_number [SSL::cert 0]]
         Get the X509 cert serial number

        set cSSLSubject [findstr $subject_dn "CN=" 0 ","]
         Findstr will allow you to abstract a piece of a string. In this case you're taking the cert subject that you got earlier, which is a longer distinguished name value (ex. CN=bob,ou=foo,o=bar,c=US), looking for the "CN=" value, and then grabbing everything after that up to the first comma. Based on the example, it would return "CN=bob". Consequently, if you just wanted to return "bob", you could put a skip parameter in the findstr command: [findstr $subject_dn "CN=" 3 ","].

        log local0. "Subject = $cSSLSubject, Hash = $cert_hash, Serial = $serial_dn"

        set Expected_hash [string toupper [class lookup $cSSLSubject authGroup]]
         Now you're going to look in a data group called "authGroup" for any entries that match the cSSLSubject variable (ex. CN=bob). If there's a match it'll return the corresponding "expected hash" - the value in the data group associated with that key.

        if { $Expected_hash != $cert_hash } {
             if the hash in the data group doesn't match the hash grabbed form the cert, log and reject the connection. This is a quasi-security mechanism that allows you to match a "registered" cert to an incoming cert.
            log local0. "Thumbprint mismatch. Expected Hash = $Expected_hash, Hash received = $cert_hash (Subject = $cSSLSubject, Serial = $serial_dn)"
            reject
        } else {
            log local0. "Thumbprint match. Expected Hash = $Expected_hash, Hash received = $cert_hash (Subject = $cSSLSubject, Serial = $serial_dn)"
             do nothing - let traffic through
        }
    } else {
        log local0. "X509 Certificate not present"
         There was no client certificate so reject the connection.
        reject
    }
}

$Expected_hash is the result of a data group match of the "CN=bob" key. The data group lookup [class lookup $cSSLSubject authGroup] is looking in the data group called "authGroup" for a value of "CN=bob", and if there's a match it returns the corresponding value, which is a hash.

If by changing the cert you mean changing the client cert, then of course the hash on that certificate would change, and most certainly not match the one you have stored in the data group. As for changing the clientssl profile, I guess it depends on what you changed.

During the CLIENTSSL_HANDSHAKE event, the client is presenting a certificate to me? And that is where I am getting $cert_hash which I ultimately compare to $Expected_hash?

Yes. This iRule assumes that you're doing "mutual" SSL, which is defined in the Client Authentication section of the clientssl profile (request or require).

The data group contains CN and hash values for "known" client certificates, which would have been entered through some previous registration process.