Forum Discussion
Jason_40769
Nimbostratus
NimbostratusF5 10.1 Offline???
I had an issue where i needed to reboot one of my F5 LTM's the other day. After it came back up it has sat in "Offline" mode and i cant figure out how to get it back to "Online" mode. Thoughts?
8 Replies
Jason_40769Here is the error i get when running a configsync... it looks like its possible the mysql database is corrupt.
Operation Status
Configsync Mode: Pull
Saving temporary UCS on peer...
Downloading UCS from peer...
Checking configuration on local system and peer system...
Peer's IP address: 192.168.208.5
Saving active configuration...
mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) when trying to connect
Failed to execute /usr/bin/mysqldump --defaults-file=/usr/local/wa/scripts/wa_configsync_my.cnf -uroot -pPASSPHRASE --protocol=socket --socket=/var/lib/mysql/mysql.sock --databases --result-file=/var/tmp/wa_db.sql.cstmp WA - Bad file descriptor
The database daemon exited with error code 256.
Failed to save the WA database. at /usr/local/wa/scripts/wa_configsync.pl line 466.
Error creating package
Error backing up current configuration.
Product : BIG-IP
Version : 10.1.0
Hostname: UCS : bigip1.musl.lan
System: bigip2.musl.lan
Installing --shared-- configuration on host bigip2.musl.lan
Installing configuration...
Post-processing...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Failed to load: print to piped cmd 'mysql': Broken pipe at /usr/local/wa/scripts/wa_configsync.pl line 466, <$in_fh> line 197.
ERROR: problems during loading WA configuration.
Reading configuration from /config/low_profile_base.conf.
Reading configuration from /defaults/config_base.conf.
Reading configuration from /config/bigip_sys.conf.
Reading configuration from /config/bigip_base.conf.
Reading configuration from /usr/share/monitors/base_monitors.conf.
Reading configuration from /config/profile_base.conf.
Reading configuration from /config/daemon.conf.
Reading configuration from /config/bigip.conf.
Reading configuration from /config/bigip_local.conf.
Loading the configuration ...
WARNING: There were one or more errors detected during installation.
Check the error messages and take the proper actions if needed.
ERROR: UCS installation failed.
Operation aborted.
/tmp/configsync.spec: Error installing package
Config install aborted.
Error running config sync pull.
BIGpipe parsing error:
01110001:3: Error running config sync pull
hoolioCirrostratus
Can you run bigstart stop mysql, bigstart start mysql and check again? If that fails, try checking the mysql err logs under /var/lib/mysql/.
Aaron- Jason_40769Here is what is listed..
120123 12:32:22 mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
120123 12:32:25 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
120123 12:32:26 InnoDB: Started; log sequence number 0 43715
120123 12:32:26 [Note] Recovering after a crash using mysqld-bin
120123 12:32:26 [ERROR] Error in Log_event::read_log_event(): 'read error', data_len: 306, event_type: 2
120123 12:32:26 [Note] Starting crash recovery...
120123 12:32:26 [Note] Crash recovery finished.
/usr/sbin/mysqld: Disk is full writing './mysqld-bin.000021' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
/usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
/usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
/usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
/usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs
120123 13:10:17 mysqld started
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
120123 13:10:17 InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB - Jason_40769If i run a "bigstart stop mysql" command the unit comes back online.
- Jason_40769of course i cant config sync the unit however. =(
- Jason_40769Do i need to free up space? Notice my sql is 100% used.
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg--db--hdc-set.2.root
253871 136201 104563 57% /
/dev/mapper/vg--db--hdc-set.2._config
507748 19545 461989 5% /config
/dev/mapper/vg--db--hdc-set.2._usr
1290112 922524 302052 76% /usr
/dev/mapper/vg--db--hdc-set.2._var
3096336 128096 2810956 5% /var
/dev/mapper/vg--db--hdc-dat.share.1
30963708 5175380 24215464 18% /shared
/dev/mapper/vg--db--hdc-dat.log.1
7224824 179536 6678288 3% /var/log
none 2030632 324 2030308 1% /dev/shm
none 2030632 6900 2023732 1% /var/tmstat
prompt 4096 12 4084 1% /var/prompt
none 2030632 684 2029948 1% /var/run
/dev/mapper/vg--db--hdc-app.ASWADB.set.2.mysqldb
12385456 11756768 0 100% /var/lib/mysql - Jason_40769Fixed it!
This solution ended up helping me.
http://support.f5.com/kb/en-us/solutions/public/12000/700/sol12759.html?sr=18886874 
Philippe_Page_2Cirrus
Hello Jason,
I'm facing the same issue having my /var/lib/mysql @ 100%
Could you help me how did you resolve this issue? when I accessed the link you posted I was prompted with "Page Not Found"
Thank you in advance.
![\"F5](\"https://www.f5.com/content/dam/f5/f5-logo.svg\"){kind=logo}
