Can iRules identify specific protocols?

Hi kdw313, if this is through-traffic that BIG-IP is handling and not an endpoint BGP speaker that you are referencing, you should be able to use an iRule with the binary scan command to check for BGP open messages. The value for the open message should be 1 in the 19th byte of the TCP payload. untested, but this should get you started:

when CLIENT_ACCEPTED {
    TCP::collect 19
}
when CLIENT_DATA {
    binary scan [TCP::payload] @18c bgp_message_type
    if {$bgp_message_type eq "\x01"} {
        log local0. "BGP Open message detected!"
    }
    TCP::release
}