GTM issues with upgrade from 10 to 11, active/active pair

We have a pair of GTM 1600 devices running 10.2.1 Build 511.0 Hotfix HF3. Need to upgrade to 11.6.0.0.0.401. I noticed something problematic though. What I thought would be a zero downtime upgrade was not at all.

 

When I took gtm1 out of production by disabling its 1.1 interface (the interface that connects to uor prod network) and then upgraded it to 11, I noticed that the other device, gtm2, which was still active, but did not seem to be processing DNS requests. nslookup against a number of names where global availability was selected and the preferred address was up while the secondary address was down, resulted in seemingly random repsonses. Sometimes I would get address A and sometimes I would get address B, I couldn't find a pattern.

 

But it gets weirder, statistics on the active GTM device were not incrementing at all. As far as GTM was concerned, it had not returned any answers. I ran out of time on my maintenance window trying to figure this out and had to roll back. And I'll need to be able to explain what happened and how I'll prevent it before I can try again. Meanwhile we're running very out of date code with vulnerabilities that our security team is anxious to patch.

 

Anybody have any ideas?