In response to your question about whether or not an iRule is evaluated before an ASM security policy, the answer is, "It depends." Think TCP stack on client-side and server-side connections, as well as request and response contexts. For example, if the iRule is triggered on a client-side L4 event, such as CLIENT_ACCEPTED, then the iRule will be evaluated before the ASM security policy. However, if the iRule is triggered on a server-side event, such as SERVER_CONNECTED, then the ASM policy will be evaluated for the request before the iRule. If you are just comparing at L7, then on an HTTP_REQUEST event, for example, it would appear the iRule is evaluated first. (I ran a quick test with an iRule that triggered on HTTP_REQUEST and sent a custom HTTP response. I received the response from the iRule and no ASM event log messages were recorded.) You can also trigger an iRule on certain ASM events, in which case you could say that the evaluation of the iRule is interspersed with the evaluation of the ASM policy.
This question came up recently in regards to HTTP to HTTPS redirect iRule. We know they can be abused for example using the host field inject which redirects the user to another site, so our ASM engineer and I were curious to know which comes first in this case. On the surface I'd say that the packet would never make it to ASM. I will do some testing.
If your redirect occurs on the HTTP_REQUEST event, then I am 99 and 44/100's percent sure that the HTTP_REQUEST event occurs before any ASM processing. I tested an iRule that wrote a log message on both the HTTP_REQUEST and ASM_REQUEST_DONE events, and the HTTP_REQUEST log message was consistently produced before the ASM_REQUEST_DONE log message.
