Migrations and Floating IP's

I have been going through a migration from an older HA pair running 9.x to a new pair running the latest version of 11.x. Everything has been going really well until today when I was trying to troubleshoot an issue with some systems we had migrated. I should have started by mentioning that I was not the person that originally setup the old environment and am really learning a ton as I go through this migration. That being said, I realized that the old system's floating IP's for the 'private' networks (DMZ and internal) were being used by all the hosts that had been behind it as the default gateway. I never realized this I guess because the IP's where x.x.x.1 which typically is always our default gateways for all of our subnets. So far I have not had any issues at all when migrating systems over to the new HA pair but I will note that I usually always use 'automap' on the VIP's where I never had done that on the old system. I know this should have jumped out at me right away but it didn't. So finally to my question. What is going to be the best way to move those floating IP's over to the new system? I am assuming that once I migrate the last few systems over and power that old HA pair down I will run into problems if I don't move those x.x.x.1 addresses since all of my hosts currently have that setup as the default gaetway and traffic is routing through that older HA pair? Can I move it over now and have that address on both systems or will that cause an issue? (multiple floating IP's for the same subnet) I know I am rambling here but am hoping this makes sense and I can get someone to point me in the right direction. Thank you.

Note: The issues I was troubleshooting when I figured this out was a communication issues between systems behind the F5 and some workstations. I changed the DG on the 2 systems behind the new load balancers to the floating IP of the new system and while it fixes the issue it cause a new issues where I was unable to resolve DNS.

config

design

11 Replies

What_Lies_Bene1
Cirrostratus
Apr 10, 2013
Having the same floating Self IP will definitely cause issues assuming the old and new pair are attached to the same VLAN.

Automap is definitely the way to go until the migration is complete.

Regarding the DNS issue, does the old pair have any routing/wildcard VSs that are not yet present on the new pair? Or perhaps just some routes not present on the new pair?
Soap_111722
Nimbostratus
Apr 10, 2013
Yes, both pairs of LTM's are sharing the same VLANS. Just trying to determine the best course of action. Not sure if it's to just move the last few remaining systems over to the new pair and then swap the floating IP's from old to new? If I did that, then I would have to remove the automap on everything. Just throwing stuff out there, I am at a total loss and am worried I have backed myself into a corner now. I don't have any VS's configured for the DNS issues I encountered. Still diggin through it all....

I am not as concerned about the DNS issue I had because I wasn't able to reproduce that on a windows box, the issue I had was on a linux box. It had the correct DNS entries so I was confused how changing the DG caused a problem like that. I am really concerned by the fact that all of my hosts are setup with a DG of the floating IP of the old LTM pair. When I perform a tracert on a host that I have moved to the new pair I see the self IP of the old active LTM in the route.
What_Lies_Bene1
Cirrostratus
Apr 11, 2013
Certainly completing the migration would be the best course and then yes, moving the floaters. You wouldn't have to remove the automap but you wouldn't need it anymore I'd imagine and then at least you'd see real IPs in the server logs etc.

The automap is ensuring traffic coming in through the new pair gets back to it, as far as the servers are concerned they are responding to a device on the same LAN and the default gateway isn't used. Obviously a traceroute will point to the old pair as you haven't moved the floater that is the DG.

So, complete the migration, move the floater(s) and then disable the automap one VS at a time and you're there.
Soap_111722
Nimbostratus
Apr 11, 2013
Thanks, yea the problem cropped up when I moved some systems over and those systems actaully have to communicate with some siemens workstations out in the hospital. I am not sure if those workstations have some type of approved list and they only accept connections from a specific IP or what. Still trying to get more information. I will conitnue with the migration and then schedule a downtime, unplug all of my uplinks on the old HA pair and change the floaters on the new pair to match. Thanks
What_Lies_Bene1
Cirrostratus
Apr 11, 2013
You're welcome. If you get stuck, post back.
Soap_111722
Nimbostratus
May 03, 2013
So last night I attempted to remove all of the uplinks from the old 9.x pair and then I moved the private and DMZ_private floating ip addresses to the new 11.x pair. (these ip's are what my systems being load balanced use for their default gateways) As soon as I did this I was unable to connect to my VIP's, since I was under a very small downtime for this maintenance I had to revert back. Before I did that I tried to delete the dynamic ARP cache from the 11.x active node, but that did not help. I am thinking this is an arp cache issue, but am scratching my head still over this. I am engaging with my network team to help, but wanted to report back and see if anyone had any ideas.
nitass
Employee
May 04, 2013
have you had a chance to run tcpdump on bigip when vip was not accessible? if you want bigip to send GARP, you may reload configuration (e.g. tmsh load sys config).
Soap_111722
Nimbostratus
May 06, 2013
No, I did not try that as I was caught off guard by this issue and had a very small window to troubleshoot and had to undo my changes. I am now trying to determine why this happened and what I can do to prevent it when I try again. Still leaning towards an issue related to ARP cache as I don't know how else to explain the behavior. I wish I had tried the command to see the outcome. I did attempt to failover the new 11.x nodes once they had the new floating ip addresses configured but I guess that would not give the same results as running the sys config command?
What_Lies_Bene1
Cirrostratus
May 07, 2013
Hey Soap, sorry to hear about your troubles. I'd certainly suspect stale ARP caches as the root cause (I'm feeling rather bad about forgetting to mention it tbh).

Nitass's suggestion may help but I'd suggest you manually clear the ARP cache on any surrounding L3 devices (including firewalls) next time you attempt the migration. Anything else you can do to prepare for the worst will always help.
Soap_111722
Nimbostratus
May 14, 2013
Looks like the issue was some static routes I didn't account for. I hadn't migrated over the floating IP of what I call 'public' interface (really private) so when I did the move a few weeks ago and took down the old LTM's the traffic couldn't find the new LTM's. Going to correct this Friday morning hopefully. Just wanted to report back. Thanks