APM site slowness/timeout

When you say "the site will resolve immediately", what are you referring to specifically? Do you any reason to believe that a portion of the access policy is failing? Where and how does it fail? Does it fail once in a valid session, or only for new connections? If it's only on new connections, try adding some message boxes in the APM visual policy at different points to see where it might be getting stuck.