Question with gslb

I have run into a problem, where one of our external clients are using their ISP's DNS servers, for doing a recursive query, and hence we are seeing a query come to GTMs from various different IP addresses. And hence, the GTMs resolve to different IP each time, and the session is flapping, and breaking. We have topology records setup for 'state' but the problem is not solved. Is there any better solution, when the egress paths in the client network vary and the query can come from a unpredictable source? I've checked using topology records for ISPs, but the list of ISPs available in the geoIP database is limited. Any other suggestions?