I want restrict number of connection based on source IP if cookie exist for that source IP.

It depends.

If your requirement is "limit the number of accesses from the same source within a certain timeframe", you can use iRule table with an expiry timer to keep track of past sessions. For example, if the same user accesses the BIG-IP LTM 10 times within 100s, send back 50x response. You can determine the identity of the user by IP or cookie or any other unique values. The following sample shows IP address version. If you want to use cookie, you can use the value from

HTTP::cookie

If you are talking about "limit the currently running (connected) sessions in parallel", it would be a bit complicated: you need to decrement the count upon the CLIENT_CLOSE event. Better error handling may be needed too. HTTP sessions are usually short-lived, so you may not need to go this path (I think).

when RULE_INIT {
  set static::subtable "SatTest"
  set static::maxCount 10
}

when HTTP_REQUEST {

   Default timeout is 180(s). The entry is deleted if not touched more than 'timeout'.
  set timeout 100
   Default lifetime is 180(s). The entry is deleted after 'lifetime' from creation.
  set lifetime 100

  set ip [IP::client_addr]
  set count [table lookup -notouch -subtable $static::subtable $ip]

  if {$count == ""} {
    log local0. "$ip does not exist. Created."
    table set -subtable $static::subtable $ip 1 $timeout $lifetime
  }
  elseif {$count > $static::maxCount} {
    log local0. "$ip exeeded the max count. $count > $static::maxCount. Call rejected."
    HTTP::respond 503 content "I am overloaded."
    return
  }
  else {
    log local0. "$ip $count + 1"
    table incr -subtable $static::subtable $ip
  }
}

See also:

• Wiki: iRules API table

• Wiki: iRules API HTTP

• Wiki: iRules API CLIENT_CLOSED