GTM 10 to 11 upgrade experiences

We have performed GTM upgrades from v10 to v11.

All the objects were successfully carried over to V11.

We found out that once the GTMs in a sync group have been upgraded, there might be issues with the Sync Groups. You can circumvent by naming the SYNC group name to something else and renaming back to the original sync group name after the upgrades.

Say you have two GTMs (GTM1 and GTM2) in your sync group with syncgroup name as GTMSYNC.

1. Backup GTM configs
2. GTM1 - Rename SYNC Group name to GTMSYNC-Other1
3. GTM1 - Install V11 on a new volume and reboot to new V11 volume. Test DNS functionality by testing against GTM1.
4. GTM2 - Rename SYNC Group name to GTMSYNC-Other2
5. GTM2 - Install V11 on a new volume and reboot to new V11 volume. Test DNS functionality by testing against GTM2.
6. GTM1 and GTM2 - Rename SYNC Group name back to GTMSYNC
7. Perform iqdump from each GTM to the other GTM to verify iQuery. Also perform iqdump from GTM to all the LTMs that they are talking to, to verify iQuery is still working.
8. We had no issues with iQuery. But for some reason if iQuery is failing against the LTMs, perform bigip_add against the LTMs to re-establish iQuery connectivity.

On another note keep away from versions such as V11.3.0 HF7 and V11.4.0 HF3, etc because of a BIND vulnerability that it introduces. The downloads page has the warning in there, so avoid such releases.

"Customers that rely on BIND Zone syncing to replicate their DNS database between GTMs should not upgrade to this HF due to a problem with Zone syncing. This issue is being tracked as ID429127."

I am trying to upgrade from 10.2.4 to 11.4.1 and once booted into the 11.4.1, the Zone runner went crazy. The Daemon keep restarting over and over again, the GTM only has WideIP info, they are not running bind. Not sure what to do next. Anyone every run into that issue?

Let me know.

Wallace

Excellent information Bhanu. I was able to talk to an F5 engineer who has been through this process. He mentioned the exact same renaming of sync groups that you employed.

The engineer also mentioned these other tidbits.

1) Re-activate the existing 10.x license. Reason: If your currently running 10.x license was activated before the release of the 11.x software version that you are installing, the 11.x software ( particularly bind/zonerunner if memory serves ) will not work properly and will not bring your entries over.

2) Disable the GTM synchronization as Bhanu mentioned. This will prevent your different versions of GTM from trying to sync with each other and causing potential problems.

3) Check all zone files for non-standard entries and fix errors before running the install. named-checkconf -t /var/named -z -j /config/named.conf

4) Run the install of the software into the slot. Apply the relevant hot fixes.

5) Reboot into the new OS to test the version. Run some dns queries at listeners running on the upgraded GTM.

Don't forget to make a full UCS backup and keep it off-system before you start this. You can always reboot into another slot, but just in case it never hurts.

We'll be giving this a try in the very near future.

Much thanks to Chad of F5 for spending a lot of time going over this with me. The conversation was much more useful than anything that could have gone into a ticket.

Jason

Hi Bhanu

I'll perform upgrade BIG-IP 3900 from 10.2.4 to ... using GTM+LTM Module. (Customer want to use AVR too, so it will using 3 modules)

Which version you suggest to upgrade?

With our window approved, we finally completed our jump to 11.4.0 HF4.

When putting the two servers back into sync with each other, we ran iqdump from each GTM to its partner. We found a mismatch so we had to do big3d_install from the higher level server to the lower.

Please note: in 11.4.0 ( maybe in other versions ) your GTMs will report themselves as being 'Active' and 'standalone'. This is for the LTM status ( even with LTM receiving no provisioned resources ). Your GTM config and HA is set in the GUI through "System->Configuration->Global Traffic->General".

A test of creating an empty WideIP and watching it sync to the other node(s) will verify that GTM sync is working.

Other than the small wrinkle with the big3d version not getting upgraded somehow on one of the systems, and a few extra reboots when our systems upgraded the BIOS, HSB and AOM, the process was as advertised.

Jason

I am having trouble too. I just attempted an upgrade from 10.2.1 to 11.4.1 on a single GTM which is a part of an active-active pair. I first installed 11.4.1 to a new volume, then installed HF2 to the same volume, then booted into that volume.

My GTM came back online about 8 minutes later and seemed to working normally, but no objects (WIDEIPs, pools, etc) had carried over. All I had was partitions.

iqdump didn't tell me anything useful, big3d_install said everything was current, bigip_add wasn't needed. I spent a few hours messing with this thing before finally giving up. Anyone have any idea what was my issue?

The version upgrade was successful, but it seems that the config with all my objects wouldn't load. I am not sure why.

2014-02-03T10:29:14-08:00 localhost notice boot_marker : ---===[ HD1.3 - BIG-IP 11.4.1 Build 608.0 ]===--- 2014-02-03T10:34:13-08:00 localhost notice boot_marker : ---===[ HD1.3 - BIG-IP 11.4.1 Build 608.0 ]===--- Feb 3 10:38:16 gtm --------------------------------------------- Feb 3 10:38:16 gtm Mon Feb 3 10:38:16 PST 2014 Feb 3 10:38:16 gtm notice big3d[5665]: 012b0014:5: Current executable path is /usr/sbin/big3d. Checking for existence of /shared/bin/big3d. Feb 3 10:38:16 gtm notice big3d[5665]: 012b0017:5: Executable /shared/bin/big3d is older than /usr/sbin/big3d. Feb 3 10:38:16 gtm notice gtmd[8578]: 011a0007:5: /usr/sbin/gtmd started =============================== Feb 3 10:38:16 gtm notice big3d[5665]: 012b0016:5: Copying /usr/sbin/big3d to /shared/bin/big3d. Feb 3 10:38:16 gtm notice big3d[5665]: 012b0018:5: Respawning to run /shared/bin/big3d. Feb 3 10:38:16 gtm notice big3d[5665]: 012b0000:5: big3d started =============================== Feb 3 10:38:33 gtm Unexpected Error: Configuration cannot be saved unless mcpd is in the running phase. Save was canceled. See "show sys mcp" and "show sys service". If "show sys service" indicates that mcpd is in the run state, but "show sys mcp" is not in phase running, issue the command "load sys config" to further diagnose the problem.

I tried running the "tmsh load sys config" command but nothing changed. I still have a working GTM running 11.4.1 HF2 with a bunch of partitions but no objects inside them.

I left out one possibly pertinent line from the log, so here all of it is again. With the left-out line at the very bottom. Maybe this points me to something?

2014-02-03T10:29:14-08:00 localhost notice boot_marker : ---===[ HD1.3 - BIG-IP 11.4.1 Build 608.0 ]===--- 2014-02-03T10:34:13-08:00 localhost notice boot_marker : ---===[ HD1.3 - BIG-IP 11.4.1 Build 608.0 ]===--- Feb 3 10:38:16 gtm --------------------------------------------- Feb 3 10:38:16 gtm Mon Feb 3 10:38:16 PST 2014 Feb 3 10:38:16 gtm notice big3d[5665]: 012b0014:5: Current executable path is /usr/sbin/big3d. Checking for existence of /shared/bin/big3d. Feb 3 10:38:16 gtm notice big3d[5665]: 012b0017:5: Executable /shared/bin/big3d is older than /usr/sbin/big3d. Feb 3 10:38:16 gtm notice gtmd[8578]: 011a0007:5: /usr/sbin/gtmd started =============================== Feb 3 10:38:16 gtm notice big3d[5665]: 012b0016:5: Copying /usr/sbin/big3d to /shared/bin/big3d. Feb 3 10:38:16 gtm notice big3d[5665]: 012b0018:5: Respawning to run /shared/bin/big3d. Feb 3 10:38:16 gtm notice big3d[5665]: 012b0000:5: big3d started =============================== Feb 3 10:38:33 gtm Unexpected Error: Configuration cannot be saved unless mcpd is in the running phase. Save was canceled. See "show sys mcp" and "show sys service". If "show sys service" indicates that mcpd is in the run state, but "show sys mcp" is not in phase running, issue the command "load sys config" to further diagnose the problem. Feb 3 10:38:34 gtm err gtmd[8578]: 011a0058:3: hookOnChild: tmsh config save exited with error code 1

Before you boot into the new partition, you must do a copy of your configs.

cpcfg --source=HDx.x HDy.y

Then reboot into your new version. That may be the magic sauce you were looking for.

Also, make sure you have renewed the activation license before you copy the configs over. If your current running version of software predates the release of the new version, you will have issues.

Hopefully that fixes it.

Jason

Thanks for the suggestion. I had not seen that command before nor was I aware I should execute it. I copied the configs over from HD1.1 (running 10.2.1) to HD1.3 (11.4.1) and the copy was successful. But after booting into 1.3 I still had the same results: GTM appears functional and it retained self IPs, VLANs, partitions, but no configured objects.

I have a ticket open with F5 about this but no resolution yet.