F5 ADD-VPR-ASM-C2400 Add on License - After upgrade of Viprion 2400 from version 11.6.0 Build 6.0.442 Hotfix HF6 to Version BIG-IP 12.1.2 Build 2.0.276 Hotfix HF2 we are not able to sync config between two VCMP guests Could you please help me what are the details i need to check and collect the data ...

Hi, Please let me know following details 1. What error are you getting? 2. Are both devices showing sync status as "changes pending" or disconnected? 3. Are you able to do a telnet on port 443 to configured synchronization IP? 4. Are you using admin username for sync configurations?

Are both devices showing sync status as "changes pending" or disconnected? Ans : Disconnected. Are you able to do a telnet on port 443 to configured synchronization IP? Unfortunately as VCMP guest is not acting as expected we cannot telnet it. As you can see AN0990-F5-ITBE-2-1 10.111.249.13 is not responding at all time, we do not know why? I hope that you will find something in already provided logs, as already whole network has been checked by desing team, and they didn’t find any issue also As for another box located in the same Viprion chassis vcmp guest an0990-f5-test-2-1 10.111.249.16 is running fine. This morning we didn’t do any config change in the F5 guests or Viprion chassis, the same as on any routers, switches, as still this issue happened. ping -c 100 -vpn-instance IT-Applications-Back-End 10.111.249.13 PING 10.111.249.13: 56 data bytes, press CTRL_C to break Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=3 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=43 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=44 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=51 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=52 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=53 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=54 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=55 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=56 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out --- 10.111.249.13 ping statistics --- 61 packet(s) transmitted 9 packet(s) received 85.24% packet loss round-trip min/avg/max = 1/1/1 ms ping -c 100 -vpn-instance IT-Applications-Back-End 10.111.249.16 PING 10.111.249.16: 56 data bytes, press CTRL_C to break Reply from 10.111.249.16: bytes=56 Sequence=1 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=2 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=3 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=4 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=5 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=6 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=7 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=8 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=9 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=10 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=11 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=12 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=13 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=14 ttl=255 time=1 ms --- 10.111.249.16 ping statistics --- 14 packet(s) transmitted 14 packet(s) received 0.00% packet loss round-trip min/avg/max = 1/1/1 ms dis arp all | inc 10.111.249.13|10.111.249.15 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE VLAN/CEVLAN PVC 10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End 10.111.249.15 0023-e997-8c1f 6 D-1 GE1/0/3.3261 IT-Applications-Back-End Total:886 Dynamic:802 Static:0 Interface:84 Remote:0 It looks like F5 VCMP guest doing something strange, not sure what but we cannot telnet it

Why devices are in force offline status? Please release offline one by one and verify

Post upgrade we are not able to sync config between two VCMP guests

16 Replies

RaghavendraSY
Altostratus
Nov 30, 2017
Hi,

Please let me know following details 1. What error are you getting? 2. Are both devices showing sync status as "changes pending" or disconnected? 3. Are you able to do a telnet on port 443 to configured synchronization IP? 4. Are you using admin username for sync configurations?
- pingnazer_23602
  Nimbostratus
  Nov 30, 2017
  Are both devices showing sync status as "changes pending" or disconnected?
  
  Ans : Disconnected.
  
  Are you able to do a telnet on port 443 to configured synchronization IP?
  
  Unfortunately as VCMP guest is not acting as expected we cannot telnet it.
  
  As you can see AN0990-F5-ITBE-2-1 10.111.249.13 is not responding at all time, we do not know why? I hope that you will find something in already provided logs, as already whole network has been checked by desing team, and they didn’t find any issue also As for another box located in the same Viprion chassis vcmp guest an0990-f5-test-2-1 10.111.249.16 is running fine. This morning we didn’t do any config change in the F5 guests or Viprion chassis, the same as on any routers, switches, as still this issue happened.
  
  ping -c 100 -vpn-instance IT-Applications-Back-End 10.111.249.13 PING 10.111.249.13: 56 data bytes, press CTRL_C to break Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=3 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=43 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=44 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=51 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=52 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=53 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=54 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=55 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=56 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out
  
  --- 10.111.249.13 ping statistics --- 61 packet(s) transmitted 9 packet(s) received 85.24% packet loss round-trip min/avg/max = 1/1/1 ms
  
  ping -c 100 -vpn-instance IT-Applications-Back-End 10.111.249.16 PING 10.111.249.16: 56 data bytes, press CTRL_C to break Reply from 10.111.249.16: bytes=56 Sequence=1 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=2 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=3 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=4 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=5 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=6 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=7 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=8 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=9 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=10 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=11 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=12 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=13 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=14 ttl=255 time=1 ms
  
  --- 10.111.249.16 ping statistics --- 14 packet(s) transmitted 14 packet(s) received 0.00% packet loss round-trip min/avg/max = 1/1/1 ms
  
  dis arp all | inc 10.111.249.13|10.111.249.15 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
  
  VLAN/CEVLAN PVC
  10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End
  
  10.111.249.15 0023-e997-8c1f 6 D-1 GE1/0/3.3261 IT-Applications-Back-End
  Total:886 Dynamic:802 Static:0 Interface:84 Remote:0
  
  It looks like F5 VCMP guest doing something strange, not sure what but we cannot telnet it
RaghavendraSY_7
Cumulonimbus
Nov 30, 2017
Hi,

Please let me know following details 1. What error are you getting? 2. Are both devices showing sync status as "changes pending" or disconnected? 3. Are you able to do a telnet on port 443 to configured synchronization IP? 4. Are you using admin username for sync configurations?
- pingnazer_23602
  Nimbostratus
  Nov 30, 2017
  Are both devices showing sync status as "changes pending" or disconnected?
  
  Ans : Disconnected.
  
  Are you able to do a telnet on port 443 to configured synchronization IP?
  
  Unfortunately as VCMP guest is not acting as expected we cannot telnet it.
  
  As you can see AN0990-F5-ITBE-2-1 10.111.249.13 is not responding at all time, we do not know why? I hope that you will find something in already provided logs, as already whole network has been checked by desing team, and they didn’t find any issue also As for another box located in the same Viprion chassis vcmp guest an0990-f5-test-2-1 10.111.249.16 is running fine. This morning we didn’t do any config change in the F5 guests or Viprion chassis, the same as on any routers, switches, as still this issue happened.
  
  ping -c 100 -vpn-instance IT-Applications-Back-End 10.111.249.13 PING 10.111.249.13: 56 data bytes, press CTRL_C to break Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=3 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=43 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=44 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=51 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=52 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=53 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=54 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=55 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=56 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out
  
  --- 10.111.249.13 ping statistics --- 61 packet(s) transmitted 9 packet(s) received 85.24% packet loss round-trip min/avg/max = 1/1/1 ms
  
  ping -c 100 -vpn-instance IT-Applications-Back-End 10.111.249.16 PING 10.111.249.16: 56 data bytes, press CTRL_C to break Reply from 10.111.249.16: bytes=56 Sequence=1 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=2 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=3 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=4 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=5 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=6 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=7 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=8 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=9 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=10 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=11 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=12 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=13 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=14 ttl=255 time=1 ms
  
  --- 10.111.249.16 ping statistics --- 14 packet(s) transmitted 14 packet(s) received 0.00% packet loss round-trip min/avg/max = 1/1/1 ms
  
  dis arp all | inc 10.111.249.13|10.111.249.15 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
  
  VLAN/CEVLAN PVC
  10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End
  
  10.111.249.15 0023-e997-8c1f 6 D-1 GE1/0/3.3261 IT-Applications-Back-End
  Total:886 Dynamic:802 Static:0 Interface:84 Remote:0
  
  It looks like F5 VCMP guest doing something strange, not sure what but we cannot telnet it
RaghavendraSY_7
Cumulonimbus
Nov 30, 2017
Why devices are in force offline status? Please release offline one by one and verify
- pingnazer_23602
  Nimbostratus
  Nov 30, 2017
  After Viprion chassis has been upgraded and booted we faced active/active issue for AN0990-F5-ITBE-2-1 VCMP guests, because of this all our virtual servers has been impacted for our services. To solve this issue we forced this VCMP guest to go offline, so we cannot switch it on.
RaghavendraSY
Altostratus
Nov 30, 2017
Why devices are in force offline status? Please release offline one by one and verify
- pingnazer_23602
  Nimbostratus
  Nov 30, 2017
  After Viprion chassis has been upgraded and booted we faced active/active issue for AN0990-F5-ITBE-2-1 VCMP guests, because of this all our virtual servers has been impacted for our services. To solve this issue we forced this VCMP guest to go offline, so we cannot switch it on.
RaghavendraSY
Altostratus
Nov 30, 2017
Can you please perform mcpd reload on nonworking guest and reboot and then do a test in a maintenance window.

clsh touch /service/mcpd/forceload clsh reboot
- RaghavendraSY
  Altostratus
  Nov 30, 2017
  Please take complete backup of your guest before maintenance window
- RaghavendraSY
  Altostratus
  Nov 30, 2017
  clsh touch /service/mcpd/forceload
  clsh reboot
  
  both commands are different, you need to perform one by one
- pingnazer_23602
  Nimbostratus
  Dec 01, 2017
  We are planning for downtime..
  
  One more thing which i noticed...
  
  I believe that the issue can be for the same mac address advertised by VCMP guest AN0990-F5-ITBE-2-1 10.111.249.13 on two blades at the same time, correct me if I’m wrong (or maybe it is normal behavior) for guests which is running on two blades?
  
  dis arp all | inc 10.111.249.13 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
  
  VLAN/CEVLAN PVC
  10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End
  
  dis interface description | inc f5 Interface PHY Protocol Description GE7/0/3 up up an0990-f5ci-2-1 1.4 back bl1 ---- connection to blade 1
  
  dis mac-address dynamic GigabitEthernet 7/0/3 | inc 0023-e997-b404
  MAC Address VLAN/ PEVLAN CEVLAN Port Type LSP/LSR-ID
  
  VSI/SI MAC-Tunnel 0023-e997-b404 IT-Backend-F5 3261 - GE7/0/3 dynamic 16/0
  Total matching items on slot 16 displayed = 4
  
  ================== dis arp all | inc 10.111.249.13 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
  
  VLAN/CEVLAN PVC 10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End
  Total:696 Dynamic:611 Static:0 Interface:85 Remote:0
  
  dis interface description | inc f5 Interface PHY Protocol Description GE7/0/3 up up an0990-f5ci-2-1 1.4 back bl2 ---- connection to blade 2
  
  dis mac-address dynamic GigabitEthernet 7/0/3 | inc 0023-e997-b404
  
  MAC Address VLAN/ PEVLAN CEVLAN Port Type LSP/LSR-ID
  
  VSI/SI MAC-Tunnel 0023-e997-b404 IT-Backend-F5 3261 - GE7/0/3 dynamic 16/0
RaghavendraSY_7
Cumulonimbus
Nov 30, 2017
Can you please perform mcpd reload on nonworking guest and reboot and then do a test in a maintenance window.

clsh touch /service/mcpd/forceload clsh reboot
- RaghavendraSY_7
  Cumulonimbus
  Nov 30, 2017
  Please take complete backup of your guest before maintenance window
- RaghavendraSY_7
  Cumulonimbus
  Nov 30, 2017
  clsh touch /service/mcpd/forceload
  clsh reboot
  
  both commands are different, you need to perform one by one
- pingnazer_23602
  Nimbostratus
  Dec 01, 2017
  We are planning for downtime..
  
  One more thing which i noticed...
  
  I believe that the issue can be for the same mac address advertised by VCMP guest AN0990-F5-ITBE-2-1 10.111.249.13 on two blades at the same time, correct me if I’m wrong (or maybe it is normal behavior) for guests which is running on two blades?
  
  dis arp all | inc 10.111.249.13 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
  
  VLAN/CEVLAN PVC
  10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End
  
  dis interface description | inc f5 Interface PHY Protocol Description GE7/0/3 up up an0990-f5ci-2-1 1.4 back bl1 ---- connection to blade 1
  
  dis mac-address dynamic GigabitEthernet 7/0/3 | inc 0023-e997-b404
  MAC Address VLAN/ PEVLAN CEVLAN Port Type LSP/LSR-ID
  
  VSI/SI MAC-Tunnel 0023-e997-b404 IT-Backend-F5 3261 - GE7/0/3 dynamic 16/0
  Total matching items on slot 16 displayed = 4
  
  ================== dis arp all | inc 10.111.249.13 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
  
  VLAN/CEVLAN PVC 10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End
  Total:696 Dynamic:611 Static:0 Interface:85 Remote:0
  
  dis interface description | inc f5 Interface PHY Protocol Description GE7/0/3 up up an0990-f5ci-2-1 1.4 back bl2 ---- connection to blade 2
  
  dis mac-address dynamic GigabitEthernet 7/0/3 | inc 0023-e997-b404
  
  MAC Address VLAN/ PEVLAN CEVLAN Port Type LSP/LSR-ID
  
  VSI/SI MAC-Tunnel 0023-e997-b404 IT-Backend-F5 3261 - GE7/0/3 dynamic 16/0

Forum Discussion

Post upgrade we are not able to sync config between two VCMP guests

16 Replies

Recent Discussions

About IPI check ip list

Cookie Violation - Expired TimeStamp.

About custome Response Blocking Page

Disacard SSL::cert mode to ignore when default SSLClient profile is set to request or require

CGNAT with DS-lite and LSN

Related Content

disable config sync upgrade GTM

Software Upgrade

Upgrade to 15.1.10

Getting Started with BIG-IP Next: Upgrading Central Manager

F5 LTM HA pair upgrade Automation using Python