We have run across an interesting scenerio. On one of our F5 vCMP guests we recently updated the pool members in our DNS pool. The old pool members were disabled and deleted from the pool but were still in the nodes list. After the updated we were seeing DNS requests being sent to the deleted pool members. The fix we did was to reboot the F5 and then the DNS resolving failures stopped. My question is, why would F5 sent requests to the deleted pool members instead of the newly updated address? Has anyone ever seen this behavior?
@gadbekr If you used FQDNs in the pool most likely what happened is the F5 was it's own TTL for the resolution of those FQDNs rather than the TTL of the DNS record which most likely kept something in memory that continue to send traffic to those old IP addresses. When you use an FQDN the F5 will dynamically create the nodes and delete those nodes when the DNS record changes so most likely a bug or a "feature" of that configuration. I recommend that when you do use an FQDN that you go into the FQDN in the node list and click the checkbox to honor the DNS record TTL instead and for these particular DNS records I would make sure you create them with a 5 minute or less TTL so if updates need to occur in the future it will take no longer than 5 minutes to switch. Some say to put it as low as possible but that cause a significant amount of unnecessary DNS queries and something DNS tends to act odd on the client device and things don't run the way you would expect. Your checklist.
1. When adding an FQDN as a pool member make sure to go to the node list and into the FQDN node and check the box to honor the DNS TTL.
2. On the authoritative DNS server for the pool member FQDNs set your TTL to 5 minutes max.
3. Allow the FQDN to expire on its own without deleting it from the pool.
I do not see the checkbox you are referring to in the Node list for the node in question. Can you provide some guidance on where that might be located. Just as a clarification, we do not use the F5 as a DNS server. We have our DNS configured in Configuration>Device>DNS. I do notice on that screen the DNS cache checkbox is selected.
@gadbekr Understood on DNS. I mean that wherever your DNS records are configured for the FQDN that you are using as the pool member that is where I would set the TTL to 500 for future possible changes. Now in regards to the DNS TTL for the pool member using FQDN you should be able to follow these screenshots to point you in the right direction. I'm unsure when they changed it but the default use to be to use the TTL specified here in the node but it doesn't seem to be the default anymore and it now uses the TTL of the DNS record when it receives it but still worth checking. In this first screenshot you can see my pool configuration with the FQDN.
In the next screenshot you can see my node list with the two auto-generated nodes from the DNS query that happens when I added in www.example.com:80 as the pool member.
In this next screenshot you can see me going into node www.example.com with the arrow pointing at the TTL checkbox which when checked will use the TTL specified in the box to the right and if not checked will use the TTL that is returned in this case for www.example.com.
In this last window this shows you how to delete the auto nodes if you delete the FQDN node or if for some reason you still have old an new DNS records populated in the pool. You can only delete auto nodes through the CLI which is why you see that in the screenshot.
Hopefully this makes a bit more sense now and that I answered your question.