So where the iRules comes into play in all of this is actually dependant on how you code the iRule. iRules are resources of the Virtual Server but you write them to look for an "Event" before they fire. If you look at the wiki below you will see all the Events that are available and where in the process of traversing through the Big-IP each will fire.
https://devcentral.f5.com/wiki/iRules.Events.ashx
So really depending on what you need an iRule to do will depend on where it will come into play, it could before or after ASM.
The basic flow through the device would be as follows
1. VS
2. Client SSL Profile
3. Protocol and HTTP profiles
4. VS Resources (httpclass)
5. ASM Policy
6. httpclass action (LB pool or redirect)
7. Node
8. Server SSL Profile (if applicable)
9. VS
Really the iRule could come in at any point in that process it just depends on how you the write the rule and what events you are using.