Hi,
BIG-IP ASM can block binary executables in 11.1+ using the magic number of the file.
https://support.f5.com/kb/en-us/products/big-ip_asm/releasenotes/product/relnote-asm-11-1-0.html
Detect File Upload Contents
ASM can now detect and block users from uploading binary executable content in a parameter’s value.
The default for this option is ON for newly created "File Upload" parameters, and this option is OFF for upgraded and imported security policies from previous versions. To change the configuration of this option, navigate to the Parameter Properties screen, set Parameter Value Type to User-input value and Data Type to File Upload, and then enable or disable the Disallow File Upload of Executables setting.
The User-input parameter Data Type that was called Binary (Length checks only) is renamed to File Upload.
We added a violation, Disallowed File Upload Content Detected that is generated when the system detects a file upload of an executable. From this violation’s learning screen you can allow file uploads of executables for each parameter the system detected.
Info on magic numbers for executable detection:
http://en.wikipedia.org/wiki/Magic_number_%28programming%29
http://catb.org/jargon/html/M/magic-number.html
You could potentially implement something similar in iRules but it would be complex and costly in terms of CPU/RAM resources.
Aaron