Forum Discussion

bletardph_10497's avatar
bletardph_10497
Icon for Nimbostratus rankNimbostratus
Aug 16, 2012

Strange behaviour after upgrading from 10.2.3 to 10.2.4

Hi,

 

We've started upgrading our 2 BigIP from 10.2.3 to 10.2.4...

 

Everything was ok before on 10.2.3.

 

Unfortunately, we are getting a strange result... Here's the story:

 

 

Upgrade from 10.2.3 with HF1 to 10.2.4 on LoadBalancer 1:

Here's the status before applying the update to LB1 but I had the same problem doing it on LB2 to test:

 

 

LB1 (10.2.3) Active - LB2 (10.2.4) Standby

 

 

Step 1: Boot LB1 on 10.2.4, LB2 go in Active, GUI console and ssh available

 

Step 2: LB1 finishes its reboot go in Active (GUI console an ssh available) BUT LB2 stays in Active mode and strange part is that GUI console and ssh becomes unavailable on LB2 ("The remote system refused the connection").

 

 

Step 3: Connecting in ssh on LB1, doing ssh to LB2, and executing the "bigpipe fo standby" to put LB2 in Standby mode.

 

 

The problem is the same if I force LB1 to Standby... Meaning that I'm getting back access to the GUI console and ssh on LB2, but I loose them on LB1...

 

I've checked some stuffs, restarted some process (httpd,...).

 

When I netstat on the problematic device, desired ports (443,...) are in listening mode, so...

 

Iptables seems also ok...

 

I'm able to push the configuration on both devices and synch them...

 

 

So my question, does someone experienced the same and what's the solution?

 

 

At the moment, it's not impacting the applications as the failover is ok. The only problem is that we have to check that the two LBs are not active together...

 

 

Thanks,

 

 

Phil

 

management
monitoring
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Aug 27, 2012
    Hi Phil,

     

     

    Are you trying to access the unit(s) via the management port or a TMM switch port (self IP)?

     

     

    Do you have a hardwire failover cable plugged in or are the units using network failover?

     

     

    Aaron
  • bletardph_10497's avatar
    bletardph_10497
    Icon for Nimbostratus rankNimbostratus
    Aug 29, 2012
    Hi Aaron,

     

     

    Yes I'm trying to reach the Web GUI console through the Self-IP. It was working before the upgrade, simultaneously on both devices.

     

     

    Units are using Network failover...

     

     

    Thanks for your help !

     

     

    Phil
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Sep 02, 2012
    have you checked /var/log/ltm? was there any suspicious log there?
  • bletardph_10497's avatar
    bletardph_10497
    Icon for Nimbostratus rankNimbostratus
    Sep 26, 2012
    Hi Nitass,

     

     

    Sorry for the late answer...

     

    Yes, logfiles checked but nothing suspicious...

     

    I've also tried sniffing with tcpdump on both devices to see what's going on... Previously I had verified the iptables on both devices.

     

    When I try to connect the GUI console on the 'unavailable' system (the passive one but which should answer and present the login box anyway), nothing appear in the tcpdump trace. But I can see the connection tentatives in the tcpdump trace on the first device (the active one)...

     

    Traces like :

     

    14:45:27.440241 IP mylatop.14657 > peer.https: S 3121387903:3121387903(0) win 8192

     

    14:45:27.445064 IP mylatop.12320 > LB1.ssh: . ack 35225 win 16327

     

    14:45:27.647184 IP mylatop.12320 > LB1.ssh: . ack 35357 win 16660

     

    14:45:27.676192 IP mylatop.14658 > peer.https: S 2141263268:2141263268(0) win 8192

     

    14:45:27.679175 IP mylatop.12320 > LB1.ssh: . ack 35669 win 16582

     

    14:45:27.885466 IP mylatop.12320 > LB1.ssh: . ack 35801 win 16549

     

    14:45:27.942680 IP mylatop.14657 > peer.https: S 3121387903:3121387903(0) win 8192

     

    ...
  • bletardph_10497's avatar
    bletardph_10497
    Icon for Nimbostratus rankNimbostratus
    Oct 09, 2012
    Hi,

     

     

    Found the problem... In the team, someone had configured the SNAT with the GUI IPs because of a firewall not open by another team. So, it was a misconfiguration issue...

     

    Sorry for your time :(

     

     

    Phil