APM site slowness/timeout

Yes, this is an address in our DMZ. We are using this for external access so it really wont be used internally, however, it does react the same going from internal. Yes, I see the requests hitting our firewall and randomly being marked as incomplete. THis seems to mean that the handshake isnt completing. However, if you try a few times eventually the site will "wake up". Also, if I go into the APM policy and just re save it, it works instantly to "wake it up". it really isnt a time of day or length of time I can pinpoint that makes it drop off. This is sitting in the same subnet in the DMZ that other sites hit and work fine. That is why this has been so tricky. You make a good point with the IP address. THat works instantly but just to bring you to the cert page (not recognized because you are using IP) Once you hit "continue to site" it behaves the same way as if I used the URL. So that proves the IP is popping up immediately, maybe DNS? THank you for your help!