Forum Discussion

Jason_40769's avatar
Jason_40769
Icon for Nimbostratus rankNimbostratus
Jan 20, 2012

F5 10.1 Offline???

I had an issue where i needed to reboot one of my F5 LTM's the other day. After it came back up it has sat in "Offline" mode and i cant figure out how to get it back to "Online" mode. Thoughts?
  • Here is the error i get when running a configsync... it looks like its possible the mysql database is corrupt.

     

     

     

    Operation Status

     

    Configsync Mode: Pull

     

    Saving temporary UCS on peer...

     

    Downloading UCS from peer...

     

     

    Checking configuration on local system and peer system...

     

    Peer's IP address: 192.168.208.5

     

    Saving active configuration...

     

    mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) when trying to connect

     

    Failed to execute /usr/bin/mysqldump --defaults-file=/usr/local/wa/scripts/wa_configsync_my.cnf -uroot -pPASSPHRASE --protocol=socket --socket=/var/lib/mysql/mysql.sock --databases --result-file=/var/tmp/wa_db.sql.cstmp WA - Bad file descriptor

     

    The database daemon exited with error code 256.

     

    Failed to save the WA database. at /usr/local/wa/scripts/wa_configsync.pl line 466.

     

    Error creating package

     

    Error backing up current configuration.

     

    Product : BIG-IP

     

    Version : 10.1.0

     

    Hostname: UCS : bigip1.musl.lan

     

    System: bigip2.musl.lan

     

    Installing --shared-- configuration on host bigip2.musl.lan

     

    Installing configuration...

     

    Post-processing...

     

    ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

     

    Failed to load: print to piped cmd 'mysql': Broken pipe at /usr/local/wa/scripts/wa_configsync.pl line 466, <$in_fh> line 197.

     

    ERROR: problems during loading WA configuration.

     

    Reading configuration from /config/low_profile_base.conf.

     

    Reading configuration from /defaults/config_base.conf.

     

    Reading configuration from /config/bigip_sys.conf.

     

    Reading configuration from /config/bigip_base.conf.

     

    Reading configuration from /usr/share/monitors/base_monitors.conf.

     

    Reading configuration from /config/profile_base.conf.

     

    Reading configuration from /config/daemon.conf.

     

    Reading configuration from /config/bigip.conf.

     

    Reading configuration from /config/bigip_local.conf.

     

    Loading the configuration ...

     

     

    WARNING: There were one or more errors detected during installation.

     

    Check the error messages and take the proper actions if needed.

     

    ERROR: UCS installation failed.

     

    Operation aborted.

     

    /tmp/configsync.spec: Error installing package

     

    Config install aborted.

     

    Error running config sync pull.

     

    BIGpipe parsing error:

     

    01110001:3: Error running config sync pull
  • Can you run bigstart stop mysql, bigstart start mysql and check again? If that fails, try checking the mysql err logs under /var/lib/mysql/.

     

     

    Aaron
  • Here is what is listed..

     

     

    120123 12:32:22 mysqld started

     

    InnoDB: The log sequence number in ibdata files does not match

     

    InnoDB: the log sequence number in the ib_logfiles!

     

    120123 12:32:25 InnoDB: Database was not shut down normally!

     

    InnoDB: Starting crash recovery.

     

    InnoDB: Reading tablespace information from the .ibd files...

     

    InnoDB: Restoring possible half-written data pages from the doublewrite

     

    InnoDB: buffer...

     

    120123 12:32:26 InnoDB: Started; log sequence number 0 43715

     

    120123 12:32:26 [Note] Recovering after a crash using mysqld-bin

     

    120123 12:32:26 [ERROR] Error in Log_event::read_log_event(): 'read error', data_len: 306, event_type: 2

     

    120123 12:32:26 [Note] Starting crash recovery...

     

    120123 12:32:26 [Note] Crash recovery finished.

     

    /usr/sbin/mysqld: Disk is full writing './mysqld-bin.000021' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)

     

    /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

     

    /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

     

    /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

     

    /usr/sbin/mysqld: Retry in 60 secs. Message reprinted in 600 secs

     

    120123 13:10:17 mysqld started

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    120123 13:10:17 InnoDB: Retrying to lock the first data file

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB: using the same InnoDB data or log files.

     

    InnoDB: Unable to lock ./ibdata1, error: 11

     

    InnoDB: Check that you do not already have another mysqld process

     

    InnoDB
  • Do i need to free up space? Notice my sql is 100% used.

     

     

     

    Filesystem 1K-blocks Used Available Use% Mounted on

     

    /dev/mapper/vg--db--hdc-set.2.root

     

    253871 136201 104563 57% /

     

    /dev/mapper/vg--db--hdc-set.2._config

     

    507748 19545 461989 5% /config

     

    /dev/mapper/vg--db--hdc-set.2._usr

     

    1290112 922524 302052 76% /usr

     

    /dev/mapper/vg--db--hdc-set.2._var

     

    3096336 128096 2810956 5% /var

     

    /dev/mapper/vg--db--hdc-dat.share.1

     

    30963708 5175380 24215464 18% /shared

     

    /dev/mapper/vg--db--hdc-dat.log.1

     

    7224824 179536 6678288 3% /var/log

     

    none 2030632 324 2030308 1% /dev/shm

     

    none 2030632 6900 2023732 1% /var/tmstat

     

    prompt 4096 12 4084 1% /var/prompt

     

    none 2030632 684 2029948 1% /var/run

     

    /dev/mapper/vg--db--hdc-app.ASWADB.set.2.mysqldb

     

    12385456 11756768 0 100% /var/lib/mysql

     

  • Fixed it!

     

     

    This solution ended up helping me.

     

     

    http://support.f5.com/kb/en-us/solutions/public/12000/700/sol12759.html?sr=18886874
  • Hello Jason,

     

    I'm facing the same issue having my /var/lib/mysql @ 100%

     

    Could you help me how did you resolve this issue? when I accessed the link you posted I was prompted with "Page Not Found"

     

    Thank you in advance.