Forum Discussion
CirrocumulusGTM Pair, one GTM incorrectly shows Virtual Server status as "unknown"
Implemenation Overview: Pair of GTMs. Both running 10.2.1. Call them GTMRIGHT and GTMWRONG. LTM HA pair ( active/passive running 10.2.0 ) containing virtual servers. These are the nodes being referenced by Wide-ips in the GTMs.
GTMRIGHT correctly reports virtual server VSA with status of 'Available'. GTMWRONG correctly reports virtual server VSA with status of 'Available'. GTMRIGHT correctly reports virtual server VSB with status of 'Available'. GTMWRONG INcorrectly reports virtual server VSB with status of 'Unknown'.
GTMWRONG INcorrectly reports several virtual servers as status 'Unknown' ( all should be available ). While it also reports correctly on other virtual servers with status of 'Available' and 'Unavailable'.
GTMRIGHT has the same code, is in sync with GTMWRONG and has the same connections ( verified with netstat looking at port 4353 ) to the LTMs. Yet GTMRIGHT reports every virtual server status correctly.
Nothing obvious in iHealth upload checks. Nothing on google, nothing in logs points to any problems. Still reviewing everything. FWIW... compared virtual server, irule, and pool settings between VIPs being reported correctly and incorrectly... no discernible difference.
3 Replies
Hem_66900Cirrus
1.If you set up iquery properly it should not report different results in GTMs. Verify iqdump < self-ip > from each other GTM to LTM;GTM-GTM as well. Fix it if iquery not set up properly.
Compare GTM synchronization setting as well.
2.Check iquery options in GTM under server objects where you define LTMs for iquery. usually it comes with the following options.Comapre them on Good & Bad GTMs. service check path SNMP
- Make sure server object settings are same.
If all 3 options looking good, You can try remove BAD GTM from sync group & add it back.
- Jason_40733The iqdump helped find the issue. All connectivity between systems was excellent. No issues here. However.... our GTMWRONG is running 10.2.1 build 297.. it's big3d shows the following version information when running iqdump. 10.2.1 big3d Version 10.2.0.297.0 linux I am guessing the 10.2.0.297.0 is probably a type-o and should say 10.2.1.297.0. The bigger news is.... GTMRIGHT is showing a different version of Big3d running despite running the same version of software. 10.2.1 big3d Version 10.2.0.1755.0 linux It appears to be running an older version of the big3d than the version of TMM. I verified the binaries available on both systems for big3d. GTMWRONG has consistent file sizes of 3875991 for the binary. GTMRIGHT has one file size in /usr/sbin/big3d of 3875991 and a file size of 3875799 for /shared/bin/big3d. I am guessing the /shared/bin/big3d is the one that might be from the older code version on GTMRIGHT. Hopefully this is the cause of the incorrect monitoring to the other GTM. We are continuing to investigate.
- Hem_66900
if you do iqdump for the self ip on a GTM locally,you'll get big3d version running on it. Check it on Good & Bad GTMs.Rule of thumb,It should match the code version on devices.
You can always remove big3d file on bad GTM and restart big3d service if the big3d versions doesnt match on good & bad GTMs.Do it after hours as you might face 2-5 seconds outage on doing it.Take the backup of wideip.conf (and or) UCS before you do anything as we can normalize the environment with just 1 file.
F5 support will walk you through how to do it all these things.Its a matter of getting 5 mins change window to fix the issue.
All the best, Hem.
