GTM not discovering LTM Virtual Server

Hello folks,

i’m really struggling to get two GTMs running together with two LTMs. I tried everything the F5 documentation said but it isn’t working.

First I’m not sure if the sync group is running like predicted and second I’m not able to discover the VS on the LTMs. Even if I try to configure the VS manually the status stays red.

What is my setup and what I’ve done so far:

GTM’s

• Both same software version (13.1.1.5-0.0.4)
• Both same big3d daemon (big3d version big3d Version 13.1.1.5.0.0.4 for linux)
• Both same sync configuration ( “Synchronize” enabled, Group Name “sync-group-gslb”
• Both connected via Management-Interface
• Open Communication Port 22 (Cert exchange) and 4353 (iQuery)
• Joined them via (tmsh run gtm gtm_add ) 
• Both exchanged Device Certificate with the other GTM as well with the LTMs (checked it more than twice)
• Both have established a communication via TCP 4353

tcp    0   0 GTM_1_IP:60068 GTM_2_IP:4353  ESTABLISHED
tcp    0   0 GTM_1_IP:4353 GTM_2_IP:42100 ESTABLISHED

• iqdump from GTM_1 to GTM_2

<!-- Local hostname: GTM_1 -->
<!-- Connected to big3d at: GTM_2_IP:4353 -->
<!-- Subscribing to syncgroup: default -->
<!-- Fri Jul 26 08:47:09 2019 -->
<xml_connection>
<version>13.1.1.5</version>
<big3d>big3d Version 13.1.1.5.0.0.4</big3d>
<kernel>linux</kernel>
<!-- Fri Jul 26 08:47:17 2019 -->
<server>
  <pkts>1678</pkts>
  <pkts_in>914</pkts_in>
  <pkts_out>764</pkts_out>
  <tot_bytes>140474</tot_bytes>
  <tot_bytes_in>82296</tot_bytes_in>
  <tot_bytes_out>58178</tot_bytes_out>
  <cpu>1</cpu>
  <uptime>84070</uptime>
  <active>1</active>
  <maintainance>0</maintainance>
  <big3d_log_level>notice</big3d_log_level>
  <monitor>0</monitor>
  <snmp>0</snmp>
  <path>0</path>
  <trace>0</trace>
  <monitor_active>0</monitor_active>
  <snmp_active>0</snmp_active>
  <path_active>0</path_active>
  <trace_active>0</trace_active>
</server>

• iqdump from GTM_2 to GTM_1

<!-- Local hostname: GTM_2 -->
<!-- Connected to big3d at: GTM_1_IP:4353 -->
<!-- Subscribing to syncgroup: default -->
<!-- Fri Jul 26 08:48:55 2019 -->
<xml_connection>
<version>13.1.1.5</version>
<big3d>big3d Version 13.1.1.5.0.0.4</big3d>
<kernel>linux</kernel>
<!-- Fri Jul 26 08:48:57 2019 -->
<server>
  <pkts>0</pkts>
  <pkts_in>0</pkts_in>
  <pkts_out>0</pkts_out>
  <tot_bytes>0</tot_bytes>
  <tot_bytes_in>0</tot_bytes_in>
  <tot_bytes_out>0</tot_bytes_out>
  <cpu>1</cpu>
  <uptime>84153</uptime>
  <active>1</active>
  <maintainance>0</maintainance>
  <big3d_log_level>notice</big3d_log_level>
  <monitor>0</monitor>
  <snmp>0</snmp>
  <path>0</path>
  <trace>0</trace>
  <monitor_active>0</monitor_active>
  <snmp_active>0</snmp_active>
  <path_active>0</path_active>
  <trace_active>0</trace_active>
</server>

• iqdump from GTM_1 to LTM_1

<!-- Local hostname: GTM_1 -->
<!-- Connected to big3d atLTM_1_IP:4353 -->
<!-- Subscribing to syncgroup: default -->
<!-- Fri Jul 26 08:50:10 2019 -->
<xml_connection>
<version>13.1.1.5</version>
<big3d>big3d Version 13.1.1.5.0.0.4</big3d>
<kernel>linux</kernel>
<!-- Fri Jul 26 08:50:17 2019 -->
<server>
  <pkts>29807203</pkts>
  <pkts_in>20448960</pkts_in>
  <pkts_out>9358243</pkts_out>
  <tot_bytes>5209260590</tot_bytes>
  <tot_bytes_in>3452551251</tot_bytes_in>
  <tot_bytes_out>1756709339</tot_bytes_out>
  <cpu>1</cpu>
  <uptime>169442</uptime>
  <active>0</active>
  <maintainance>0</maintainance>
  <big3d_log_level>notice</big3d_log_level>
  <monitor>0</monitor>
  <snmp>0</snmp>
  <path>0</path>
  <trace>0</trace>
  <monitor_active>0</monitor_active>
  <snmp_active>0</snmp_active>
  <path_active>0</path_active>
  <trace_active>0</trace_active>
</server>

• iqdump from GTM_1 to LTM_2

<!-- Local hostname: GTM_1 -->
<!-- Connected to big3d at LTM_2_IP:4353 -->
<!-- Subscribing to syncgroup: default -->
<!-- Fri Jul 26 08:50:49 2019 -->
<xml_connection>
<version>13.1.1.5</version>
<big3d>big3d Version 13.1.1.5.0.0.4</big3d>
<kernel>linux</kernel>
<!-- Fri Jul 26 08:50:57 2019 -->
<server>
  <pkts>30186423</pkts>
  <pkts_in>20678972</pkts_in>
  <pkts_out>9507451</pkts_out>
  <tot_bytes>5739874632</tot_bytes>
  <tot_bytes_in>3483570295</tot_bytes_in>
  <tot_bytes_out>2256304337</tot_bytes_out>
  <cpu>1</cpu>
  <uptime>171053</uptime>
  <active>1</active>
  <maintainance>0</maintainance>
  <big3d_log_level>notice</big3d_log_level>
  <monitor>0</monitor>
  <snmp>0</snmp>
  <path>0</path>
  <trace>0</trace>
  <monitor_active>0</monitor_active>
  <snmp_active>0</snmp_active>
  <path_active>0</path_active>
  <trace_active>0</trace_active>
</server>

LTM’s

• Both same software version like GTMs (13.1.1.5-0.0.4)
• Both same big3d daemon like GTMs (big3d version big3d Version 13.1.1.5.0.0.4 for linux)
• Both connected via Management-Interface
• Running as Active/Standby DSC
• Open Communication Port 22 (Cert exchange) and 4353 (iQuery)
• Added both LTM to both GTMs via (tmsh run gtm bigip_add)
• Both exchanged Device Certificate with the GTMs as well with the other LTM (checked it more than twice)
• Both LTM have established a communication via TCP 4353 to both GTM’s (check this here or the iqdump output above in the GTM part.
• LTM_1

o	tcp    0   0 LTM_1_IP:4353 GTM_1_IP:45348 ESTABLISHED
o	tcp    0   0 LTM_1_IP:4353 GTM_2_IP:43396 ESTABLISHED

• LTM_2

o	tcp    0   0 LTM_2_IP:4353 GTM_1_IP:35170 ESTABLISHED
o	tcp    0   0 LTM_2_IP:4353 GTM_2_IP:56806 ESTABLISHED

For me looks everything fine on this communication setup. Now we move on to the GTM configuration on the GUI.

What I have done:

• created two Datacenters, DC_1 and DC_2
• added both GTM devices (itself and the other GTM under DNS > GSLB > Servers (in their specific DC)
• added both LTM devices separately (even if they are a DSC Cluster) under Servers (in their specific DC)
• the Health Monitor is “bigip” for all of them.
• the Status LED shows GREEN
• on the LTMs I activated “Virtual Server Discovery”

That’s the setup.

My questions:

GTM:

1. I configured a sync group named “sync-group-gslb” but if I do an iqdump it shows this line

<!-- Subscribing to syncgroup: default -->

What’s wrong here?

2. How can I try the syncgroup. Which configuration should get synchronized to the other GTM? As far as I know, I need to create things like Devices and VS on every GTM separately – even in a sync group. Is that correct?

LTM:

1. No VS getting discovered automatically - why?

a. I have read tha the autodiscover feature is disabled if NAT is activated on the LTM. Are we talking about the SNAT config from a VS or some other NAT?

2. If I try to configure a VS manually, they’re going into the status RED (“no reply from big3d: timed out”) – what’s wrong here?

Any ideas beside a TAC case?