Forum Discussion

Pierre_G__71801's avatar
Pierre_G__71801
Icon for Nimbostratus rankNimbostratus
Aug 05, 2011

Config sync taking a long time

Hi,

I'm runnnig a failover cluster between BIG-IP 10.2.1 473.0 (same version for both members) and was wondering if it was normal that whenever I sync the configurations, it takes more than 2 minutes to do so.

From the standby unit to the active:

[Standby] ~  time b config sync
Checking configuration on local system and peer system...
Peer's IP address: 192.168.1.2
Synchronizing Master Keys...
Saving active configuration...
tar: conf/ssl.crt/server.crt: time stamp 2035-06-29 22:13:37 is 754214057 s in the future
tar: conf/ssl.key/server.key: time stamp 2035-06-29 22:13:37 is 754214057 s in the future
tar: ssl/ssl.key/default.key: time stamp 2035-06-29 22:13:19 is 754214039 s in the future
Configsync Mode: Push
Transferring UCS to peer...
Installing UCS on peer...
Obtaining results of remote configuration installation...
Saving active configuration...
tar: conf/ssl.crt/server.crt: time stamp 2035-06-29 22:13:37 is 754214045 s in the future
tar: conf/ssl.key/server.key: time stamp 2035-06-29 22:13:37 is 754214045 s in the future
tar: ssl/ssl.key/default.key: time stamp 2035-06-29 22:13:19 is 754214027 s in the future
Current configuration backed up to /var/local/ucs/cs_backup.ucs.
Product : BIG-IP
Version : 10.2.1
Hostname: UCS   : standby_f5
          System: active_f5
Installing --shared-- configuration on host active_f5
Installing configuration...
Installing ASM configuration...
Post-processing...
Reading configuration from /config/low_profile_base.conf.
Reading configuration from /defaults/config_base.conf.
Reading configuration from /config/bigip_sys.conf.
Reading configuration from /config/bigip_base.conf.
Reading configuration from /usr/share/monitors/base_monitors.conf.
Reading configuration from /config/profile_base.conf.
Reading configuration from /config/daemon.conf.
Reading configuration from /config/bigip.conf.
Reading configuration from /config/bigip_local.conf.
Loading the configuration ...
Loading ASM configuration...
real    2m11.046s
user    0m4.795s
sys     0m2.269s

From the active unit to the standby:

[Active] config  time b config sync
Checking configuration on local system and peer system...
Peer's IP address: 192.168.1.1
Synchronizing Master Keys...
Saving active configuration...
tar: conf/ssl.crt/server.crt: time stamp 2035-06-29 22:13:37 is 754213385 s in the future
tar: conf/ssl.key/server.key: time stamp 2035-06-29 22:13:37 is 754213385 s in the future
tar: ssl/ssl.key/default.key: time stamp 2035-06-29 22:13:19 is 754213367 s in the future
Configsync Mode: Push
Transferring UCS to peer...
Installing UCS on peer...
Obtaining results of remote configuration installation...
Saving active configuration...
tar: conf/ssl.crt/server.crt: time stamp 2035-06-29 22:13:37 is 754213351 s in the future
tar: conf/ssl.key/server.key: time stamp 2035-06-29 22:13:37 is 754213351 s in the future
tar: ssl/ssl.key/default.key: time stamp 2035-06-29 22:13:19 is 754213333 s in the future
Current configuration backed up to /var/local/ucs/cs_backup.ucs.
Product : BIG-IP
Version : 10.2.1
Hostname: UCS   : active_f5
          System: standby_f5
Installing --shared-- configuration on host standby_f5
Installing configuration...
Installing ASM configuration...
Post-processing...
Reading configuration from /config/low_profile_base.conf.
Reading configuration from /defaults/config_base.conf.
Reading configuration from /config/bigip_sys.conf.
Reading configuration from /config/bigip_base.conf.
Reading configuration from /usr/share/monitors/base_monitors.conf.
Reading configuration from /config/profile_base.conf.
Reading configuration from /config/daemon.conf.
Reading configuration from /config/bigip.conf.
Reading configuration from /config/bigip_local.conf.
Loading the configuration ...
Loading ASM configuration...
real    2m4.418s
user    0m5.088s
sys     1m7.987s

This is the result after I restarted httpd (it used to take 3 to 5 minutes before I did that!)

I noticed the comand that takes most time is

/usr/local/bin/SOAPCSClient --source /var/tmp/__sync_local__.ucs --destination __sync_remote__.ucs

Restarting httpd seems to have improved that but is it normal for the members to spend that much time on syncing?

I read this post (http://devcentral.f5.com/Community/GroupDetails/tabid/1082223/asg/44/aft/1167199/showtab/groupforums/Default.aspx) but it didn't improve performance and as it is said this not recommended, i did not apply the changes to httpd configuration.

httpd log shows this when syncing from standby to active (I know I should do the opposite but the devs who wrote the automatic script that makes changes and apply them never changed the target after the active member changed):

Aug  5 16:33:24 slot1/cld0065lb err httpd[17979]: [error] [client 192.168.1.1] FastCGI: comm with server "/usr/local/www/iControl/iControlPortal.cgi" aborted: idle timeout (300 sec)
Aug  5 16:33:24 slot1/cld0065lb err httpd[17979]: [error] [client 192.168.1.1] FastCGI: incomplete headers (0 bytes) received from server "/usr/local/www/iControl/iControlPortal.cgi"

Thanks in advance!

Regards,

Pierre

management
monitoring
  • L4L7_53191's avatar
    L4L7_53191
    Icon for Nimbostratus rankNimbostratus
    Aug 08, 2011
    I would check the duplex on the management port and confirm that there's not a mismatch - do this on both systems. I've seen this come up many times, and it can cause this behavior.

     

     

    --Matt
  • Pierre_G__71801's avatar
    Pierre_G__71801
    Icon for Nimbostratus rankNimbostratus
    Aug 09, 2011
    All the ports are full duplex, on both F5.

     

    I still don't see why this is taking so long and why it reduced by half he time of syncing when restarting httpd. :-(
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Aug 09, 2011
    All the ports are full duplex, on both F5.

     

    have u ever tried to use another interface i.e. selfip?

     

     

    additionally, is there any weird file including in the ucs i.e. tar tzvf ...?
  • Pierre_G__71801's avatar
    Pierre_G__71801
    Icon for Nimbostratus rankNimbostratus
    Aug 09, 2011
    No I haven't tried to use another interface and I wouldn't take the chance to break something 🙂

    Here are the results of a b conf save and the list of file in the ucs:

    (will repost it later, didn't work, the list is too long :))

    Size of the archive: 

     ls -lh /tmp/test_pg_20110809.ucs
-rw-r--r-- 1 root webusers 1.6M Aug  9 10:55 /tmp/test_pg_20110809.ucs
 ls -l /tmp/test_pg_20110809.ucs
-rw-r--r-- 1 root webusers 1669332 Aug  9 10:55 /tmp/test_pg_20110809.ucs
  • nathe's avatar
    nathe
    Icon for Cirrocumulus rankCirrocumulus
    Aug 10, 2011
    Pierre

     

     

    Something to help troubleshoot - I've referenced this doc before and it's been of use:

     

     

    http://support.f5.com/kb/en-us/solutions/public/7000/000/sol7024.html?sr=15954430

     

     

    In there is says that these 3 daemons mcpd, cssd and httpd are used in the configsync process. It sounds like from your initial post that restarting httpd helped, I wonder if it was hogging the memory and a restart has gone some way to correct this?!?

     

     

    I'd try to see how much cpu / mem all 3 are currently using on both boxes (ps -aux |grep httpd etc...)

     

     

    Hope it sheds some light on the matter.

     

     

    N