Is there a way to detect jailbroken mobile using an irule?

The mobile app installed on customer device must run the check to identify jailbroken device. There's no way to tell the difference purely based on incoming request observation, unless your mobile application has inserted a specific cookie or header. After you have the previous in place, writing a drop/reject iRule that checks for presence of a header or cookie will not be a difficult task. Plenty of examples are available.