Dynamically failing LTM without impact

We have 2 data centers and we use GTM and then LTM locally.

 

 

We currently have a health check (http) that we use to turn off the reals in the pool. This causes GTM to failover, which is what we want. But it also impacts clients that have cached connections. We really don't want to impact these customers because it's not a data center disaster. So I am wondering if there is any way to tell GTM to do the failover (using health check) and at the same time customers that have cached connections to the failed LTM vip directly go to the reals in other data center. Is that possible?

 

 

GTM is using iQuery.

 

 

I was thinking if we create a standby pool for the reals on other data center. But the problem is that LTM will never be marked down and there will be no failover on GTM since GTM will see LTM Vip up.