Forum Discussion
Pierre_G__71801
Nimbostratus
NimbostratusConfig sync taking a long time
Hi,
I'm runnnig a failover cluster between BIG-IP 10.2.1 473.0 (same version for both members) and was wondering if it was normal that whenever I sync the configurations, it takes more than 2 minutes to do so.
From the standby unit to the active:
[Standby] ~ time b config sync
Checking configuration on local system and peer system...
Peer's IP address: 192.168.1.2
Synchronizing Master Keys...
Saving active configuration...
tar: conf/ssl.crt/server.crt: time stamp 2035-06-29 22:13:37 is 754214057 s in the future
tar: conf/ssl.key/server.key: time stamp 2035-06-29 22:13:37 is 754214057 s in the future
tar: ssl/ssl.key/default.key: time stamp 2035-06-29 22:13:19 is 754214039 s in the future
Configsync Mode: Push
Transferring UCS to peer...
Installing UCS on peer...
Obtaining results of remote configuration installation...
Saving active configuration...
tar: conf/ssl.crt/server.crt: time stamp 2035-06-29 22:13:37 is 754214045 s in the future
tar: conf/ssl.key/server.key: time stamp 2035-06-29 22:13:37 is 754214045 s in the future
tar: ssl/ssl.key/default.key: time stamp 2035-06-29 22:13:19 is 754214027 s in the future
Current configuration backed up to /var/local/ucs/cs_backup.ucs.
Product : BIG-IP
Version : 10.2.1
Hostname: UCS : standby_f5
System: active_f5
Installing --shared-- configuration on host active_f5
Installing configuration...
Installing ASM configuration...
Post-processing...
Reading configuration from /config/low_profile_base.conf.
Reading configuration from /defaults/config_base.conf.
Reading configuration from /config/bigip_sys.conf.
Reading configuration from /config/bigip_base.conf.
Reading configuration from /usr/share/monitors/base_monitors.conf.
Reading configuration from /config/profile_base.conf.
Reading configuration from /config/daemon.conf.
Reading configuration from /config/bigip.conf.
Reading configuration from /config/bigip_local.conf.
Loading the configuration ...
Loading ASM configuration...
real 2m11.046s
user 0m4.795s
sys 0m2.269sFrom the active unit to the standby:
[Active] config time b config sync
Checking configuration on local system and peer system...
Peer's IP address: 192.168.1.1
Synchronizing Master Keys...
Saving active configuration...
tar: conf/ssl.crt/server.crt: time stamp 2035-06-29 22:13:37 is 754213385 s in the future
tar: conf/ssl.key/server.key: time stamp 2035-06-29 22:13:37 is 754213385 s in the future
tar: ssl/ssl.key/default.key: time stamp 2035-06-29 22:13:19 is 754213367 s in the future
Configsync Mode: Push
Transferring UCS to peer...
Installing UCS on peer...
Obtaining results of remote configuration installation...
Saving active configuration...
tar: conf/ssl.crt/server.crt: time stamp 2035-06-29 22:13:37 is 754213351 s in the future
tar: conf/ssl.key/server.key: time stamp 2035-06-29 22:13:37 is 754213351 s in the future
tar: ssl/ssl.key/default.key: time stamp 2035-06-29 22:13:19 is 754213333 s in the future
Current configuration backed up to /var/local/ucs/cs_backup.ucs.
Product : BIG-IP
Version : 10.2.1
Hostname: UCS : active_f5
System: standby_f5
Installing --shared-- configuration on host standby_f5
Installing configuration...
Installing ASM configuration...
Post-processing...
Reading configuration from /config/low_profile_base.conf.
Reading configuration from /defaults/config_base.conf.
Reading configuration from /config/bigip_sys.conf.
Reading configuration from /config/bigip_base.conf.
Reading configuration from /usr/share/monitors/base_monitors.conf.
Reading configuration from /config/profile_base.conf.
Reading configuration from /config/daemon.conf.
Reading configuration from /config/bigip.conf.
Reading configuration from /config/bigip_local.conf.
Loading the configuration ...
Loading ASM configuration...
real 2m4.418s
user 0m5.088s
sys 1m7.987sThis is the result after I restarted httpd (it used to take 3 to 5 minutes before I did that!)
I noticed the comand that takes most time is
/usr/local/bin/SOAPCSClient --source /var/tmp/__sync_local__.ucs --destination __sync_remote__.ucsRestarting httpd seems to have improved that but is it normal for the members to spend that much time on syncing?
I read this post (http://devcentral.f5.com/Community/GroupDetails/tabid/1082223/asg/44/aft/1167199/showtab/groupforums/Default.aspx) but it didn't improve performance and as it is said this not recommended, i did not apply the changes to httpd configuration.
httpd log shows this when syncing from standby to active (I know I should do the opposite but the devs who wrote the automatic script that makes changes and apply them never changed the target after the active member changed):
Aug 5 16:33:24 slot1/cld0065lb err httpd[17979]: [error] [client 192.168.1.1] FastCGI: comm with server "/usr/local/www/iControl/iControlPortal.cgi" aborted: idle timeout (300 sec)
Aug 5 16:33:24 slot1/cld0065lb err httpd[17979]: [error] [client 192.168.1.1] FastCGI: incomplete headers (0 bytes) received from server "/usr/local/www/iControl/iControlPortal.cgi"Thanks in advance!
Regards,
Pierre
nitassEmployee
have u checked size of ucs file? how big is it?
Pierre_G__71801Hi,
The UCS file is 1.6MB.
Regards,
Pierre
L4L7_53191I would check the duplex on the management port and confirm that there's not a mismatch - do this on both systems. I've seen this come up many times, and it can cause this behavior.
--Matt- Pierre_G__71801All the ports are full duplex, on both F5.
I still don't see why this is taking so long and why it reduced by half he time of syncing when restarting httpd. :-( - nitassAll the ports are full duplex, on both F5.
have u ever tried to use another interface i.e. selfip?
additionally, is there any weird file including in the ucs i.e. tar tzvf ...? - Pierre_G__71801No I haven't tried to use another interface and I wouldn't take the chance to break something 🙂
Here are the results of a b conf save and the list of file in the ucs:
(will repost it later, didn't work, the list is too long :))
Size of the archive:ls -lh /tmp/test_pg_20110809.ucs -rw-r--r-- 1 root webusers 1.6M Aug 9 10:55 /tmp/test_pg_20110809.ucs ls -l /tmp/test_pg_20110809.ucs -rw-r--r-- 1 root webusers 1669332 Aug 9 10:55 /tmp/test_pg_20110809.ucs - nathe
Cirrocumulus
Pierre
Something to help troubleshoot - I've referenced this doc before and it's been of use:
http://support.f5.com/kb/en-us/solutions/public/7000/000/sol7024.html?sr=15954430
In there is says that these 3 daemons mcpd, cssd and httpd are used in the configsync process. It sounds like from your initial post that restarting httpd helped, I wonder if it was hogging the memory and a restart has gone some way to correct this?!?
I'd try to see how much cpu / mem all 3 are currently using on both boxes (ps -aux |grep httpd etc...)
Hope it sheds some light on the matter.
N