Forum Discussion
NimbostratusPost upgrade we are not able to sync config between two VCMP guests
F5 ADD-VPR-ASM-C2400 Add on License - After upgrade of Viprion 2400 from version 11.6.0 Build 6.0.442 Hotfix HF6 to Version BIG-IP 12.1.2 Build 2.0.276 Hotfix HF2 we are not able to sync config between two VCMP guests
Could you please help me what are the details i need to check and collect the data ...
RaghavendraSYAltostratus
Hi,
Please let me know following details 1. What error are you getting? 2. Are both devices showing sync status as "changes pending" or disconnected? 3. Are you able to do a telnet on port 443 to configured synchronization IP? 4. Are you using admin username for sync configurations?
pingnazer_23602- Are both devices showing sync status as "changes pending" or disconnected?
Ans : Disconnected.
- Are you able to do a telnet on port 443 to configured synchronization IP?
Unfortunately as VCMP guest is not acting as expected we cannot telnet it.
As you can see AN0990-F5-ITBE-2-1 10.111.249.13 is not responding at all time, we do not know why? I hope that you will find something in already provided logs, as already whole network has been checked by desing team, and they didn’t find any issue also As for another box located in the same Viprion chassis vcmp guest an0990-f5-test-2-1 10.111.249.16 is running fine. This morning we didn’t do any config change in the F5 guests or Viprion chassis, the same as on any routers, switches, as still this issue happened.
ping -c 100 -vpn-instance IT-Applications-Back-End 10.111.249.13 PING 10.111.249.13: 56 data bytes, press CTRL_C to break Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=3 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=43 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=44 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=51 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=52 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=53 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=54 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=55 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=56 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out
--- 10.111.249.13 ping statistics --- 61 packet(s) transmitted 9 packet(s) received 85.24% packet loss round-trip min/avg/max = 1/1/1 ms
ping -c 100 -vpn-instance IT-Applications-Back-End 10.111.249.16 PING 10.111.249.16: 56 data bytes, press CTRL_C to break Reply from 10.111.249.16: bytes=56 Sequence=1 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=2 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=3 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=4 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=5 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=6 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=7 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=8 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=9 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=10 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=11 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=12 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=13 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=14 ttl=255 time=1 ms
--- 10.111.249.16 ping statistics --- 14 packet(s) transmitted 14 packet(s) received 0.00% packet loss round-trip min/avg/max = 1/1/1 ms
dis arp all | inc 10.111.249.13|10.111.249.15 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
VLAN/CEVLAN PVC10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End
10.111.249.15 0023-e997-8c1f 6 D-1 GE1/0/3.3261 IT-Applications-Back-EndTotal:886 Dynamic:802 Static:0 Interface:84 Remote:0
It looks like F5 VCMP guest doing something strange, not sure what but we cannot telnet it
RaghavendraSY_7Cumulonimbus
Hi,
Please let me know following details 1. What error are you getting? 2. Are both devices showing sync status as "changes pending" or disconnected? 3. Are you able to do a telnet on port 443 to configured synchronization IP? 4. Are you using admin username for sync configurations?
- pingnazer_23602
- Are both devices showing sync status as "changes pending" or disconnected?
Ans : Disconnected.
- Are you able to do a telnet on port 443 to configured synchronization IP?
Unfortunately as VCMP guest is not acting as expected we cannot telnet it.
As you can see AN0990-F5-ITBE-2-1 10.111.249.13 is not responding at all time, we do not know why? I hope that you will find something in already provided logs, as already whole network has been checked by desing team, and they didn’t find any issue also As for another box located in the same Viprion chassis vcmp guest an0990-f5-test-2-1 10.111.249.16 is running fine. This morning we didn’t do any config change in the F5 guests or Viprion chassis, the same as on any routers, switches, as still this issue happened.
ping -c 100 -vpn-instance IT-Applications-Back-End 10.111.249.13 PING 10.111.249.13: 56 data bytes, press CTRL_C to break Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=3 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=43 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=44 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out Request time out Reply from 10.111.249.13: bytes=56 Sequence=51 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=52 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=53 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=54 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=55 ttl=255 time=1 ms Reply from 10.111.249.13: bytes=56 Sequence=56 ttl=255 time=1 ms Request time out Request time out Request time out Request time out Request time out
--- 10.111.249.13 ping statistics --- 61 packet(s) transmitted 9 packet(s) received 85.24% packet loss round-trip min/avg/max = 1/1/1 ms
ping -c 100 -vpn-instance IT-Applications-Back-End 10.111.249.16 PING 10.111.249.16: 56 data bytes, press CTRL_C to break Reply from 10.111.249.16: bytes=56 Sequence=1 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=2 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=3 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=4 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=5 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=6 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=7 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=8 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=9 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=10 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=11 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=12 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=13 ttl=255 time=1 ms Reply from 10.111.249.16: bytes=56 Sequence=14 ttl=255 time=1 ms
--- 10.111.249.16 ping statistics --- 14 packet(s) transmitted 14 packet(s) received 0.00% packet loss round-trip min/avg/max = 1/1/1 ms
dis arp all | inc 10.111.249.13|10.111.249.15 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
VLAN/CEVLAN PVC10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End
10.111.249.15 0023-e997-8c1f 6 D-1 GE1/0/3.3261 IT-Applications-Back-EndTotal:886 Dynamic:802 Static:0 Interface:84 Remote:0
It looks like F5 VCMP guest doing something strange, not sure what but we cannot telnet it
- RaghavendraSY_7
Why devices are in force offline status? Please release offline one by one and verify
- pingnazer_23602
After Viprion chassis has been upgraded and booted we faced active/active issue for AN0990-F5-ITBE-2-1 VCMP guests, because of this all our virtual servers has been impacted for our services. To solve this issue we forced this VCMP guest to go offline, so we cannot switch it on.
- RaghavendraSY
Why devices are in force offline status? Please release offline one by one and verify
- pingnazer_23602
After Viprion chassis has been upgraded and booted we faced active/active issue for AN0990-F5-ITBE-2-1 VCMP guests, because of this all our virtual servers has been impacted for our services. To solve this issue we forced this VCMP guest to go offline, so we cannot switch it on.
- RaghavendraSY
Can you please perform mcpd reload on nonworking guest and reboot and then do a test in a maintenance window.
clsh touch /service/mcpd/forceload clsh reboot
- RaghavendraSY
Please take complete backup of your guest before maintenance window
- RaghavendraSY
- clsh touch /service/mcpd/forceload
- clsh reboot
both commands are different, you need to perform one by one
- pingnazer_23602
We are planning for downtime..
One more thing which i noticed...
I believe that the issue can be for the same mac address advertised by VCMP guest AN0990-F5-ITBE-2-1 10.111.249.13 on two blades at the same time, correct me if I’m wrong (or maybe it is normal behavior) for guests which is running on two blades?
dis arp all | inc 10.111.249.13 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
VLAN/CEVLAN PVC10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End
dis interface description | inc f5 Interface PHY Protocol Description GE7/0/3 up up an0990-f5ci-2-1 1.4 back bl1 ---- connection to blade 1
dis mac-address dynamic GigabitEthernet 7/0/3 | inc 0023-e997-b404MAC Address VLAN/ PEVLAN CEVLAN Port Type LSP/LSR-ID
VSI/SI MAC-Tunnel 0023-e997-b404 IT-Backend-F5 3261 - GE7/0/3 dynamic 16/0Total matching items on slot 16 displayed = 4
================== dis arp all | inc 10.111.249.13 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
VLAN/CEVLAN PVC 10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-EndTotal:696 Dynamic:611 Static:0 Interface:85 Remote:0
dis interface description | inc f5 Interface PHY Protocol Description GE7/0/3 up up an0990-f5ci-2-1 1.4 back bl2 ---- connection to blade 2
dis mac-address dynamic GigabitEthernet 7/0/3 | inc 0023-e997-b404
MAC Address VLAN/ PEVLAN CEVLAN Port Type LSP/LSR-ID
VSI/SI MAC-Tunnel 0023-e997-b404 IT-Backend-F5 3261 - GE7/0/3 dynamic 16/0
- RaghavendraSY_7
Can you please perform mcpd reload on nonworking guest and reboot and then do a test in a maintenance window.
clsh touch /service/mcpd/forceload clsh reboot
- RaghavendraSY_7
Please take complete backup of your guest before maintenance window
- RaghavendraSY_7
- clsh touch /service/mcpd/forceload
- clsh reboot
both commands are different, you need to perform one by one
- pingnazer_23602
We are planning for downtime..
One more thing which i noticed...
I believe that the issue can be for the same mac address advertised by VCMP guest AN0990-F5-ITBE-2-1 10.111.249.13 on two blades at the same time, correct me if I’m wrong (or maybe it is normal behavior) for guests which is running on two blades?
dis arp all | inc 10.111.249.13 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
VLAN/CEVLAN PVC10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-End
dis interface description | inc f5 Interface PHY Protocol Description GE7/0/3 up up an0990-f5ci-2-1 1.4 back bl1 ---- connection to blade 1
dis mac-address dynamic GigabitEthernet 7/0/3 | inc 0023-e997-b404MAC Address VLAN/ PEVLAN CEVLAN Port Type LSP/LSR-ID
VSI/SI MAC-Tunnel 0023-e997-b404 IT-Backend-F5 3261 - GE7/0/3 dynamic 16/0Total matching items on slot 16 displayed = 4
================== dis arp all | inc 10.111.249.13 IP ADDRESS MAC ADDRESS EXPIRE(M) TYPE INTERFACE VPN-INSTANCE
VLAN/CEVLAN PVC 10.111.249.13 0023-e997-b404 20 D-1 GE1/0/3.3261 IT-Applications-Back-EndTotal:696 Dynamic:611 Static:0 Interface:85 Remote:0
dis interface description | inc f5 Interface PHY Protocol Description GE7/0/3 up up an0990-f5ci-2-1 1.4 back bl2 ---- connection to blade 2
dis mac-address dynamic GigabitEthernet 7/0/3 | inc 0023-e997-b404
MAC Address VLAN/ PEVLAN CEVLAN Port Type LSP/LSR-ID
VSI/SI MAC-Tunnel 0023-e997-b404 IT-Backend-F5 3261 - GE7/0/3 dynamic 16/0