iRule to block Excessive cookie size

This depends on a few factors. Namely:

1. Do you mean request or response? In a request, all of the cookies will come across as a single Cookie header. You can subsequently gather it's total size with something like the following:

   when HTTP_REQUEST {
       log local0. [string length [HTTP::header Cookie]]
   }
   

   If you wanted to break those down into the individual cookies, then you might do something like this:

   when HTTP_REQUEST {
       foreach x [HTTP::cookie names] {
           log local0. "cookie($x) = [string length $x[HTTP::cookie value $x]]"
       }
   }
   

2. In a response, the Set-Cookie headers would be separate, so:

   when HTTP_RESPONSE {
       foreach x [HTTP::cookie names] {
           log local0. "cookie($x) = [string length $x[HTTP::cookie value $x]]"
       }
   }
   

3. If you mean to block the entire request or response based on some threshold, then you can modify the iRule examples above accordingly. If you need to block specific cookies, then you'd need to parse and rewrite the cookie header for requests, and remove the specific cookie in a response.

Is there a way to convert the length of the [HTTP::header Cookie]] in bytes in iRule?

Well, you could technically use the (partially obsolete) string bytelength command:

when HTTP_RESPONSE {
    foreach x [HTTP::cookie names] {
        log local0. "cookie($x) = [string length $x[HTTP::cookie value $x]]"
        log local0. "cookie($x) = [string bytelength $x[HTTP::cookie value $x]]"
    }
}

But I venture that the ASCII length is going to be pretty close to the byte length.