Forum Discussion

William_J_Gonza's avatar
William_J_Gonza
Icon for Nimbostratus rankNimbostratus
Apr 27, 2018

MySQL Cause Faiover in ASM administration

I was doing the elimination of learned traffic, elimination of the suggestion of some violations, when unexpectedly the machine is locked in GUI blocking and additionally is placed in "Offline" mode causing failover, additionally a high CPU consumption is observed over 100% Once the theme is reestablished, the ASM policy generates access blocking to the associated Virtual Server, and as a solution to this, the ASM policy "policy_new_www" is restored, restoring the service.

This are some logs i can see:

/var/log/daemon.log

 

Apr 26 11:15:52 F5BIGIP warning kernel: mysqld invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
...
Apr 26 11:15:52 F5BIGIP err kernel: Out of memory: Kill process 21037 (bd) score 64 or sacrifice child
Apr 26 11:15:52 F5BIGIP err kernel: Killed process 21037, UID 0, (bd) total-vm:2270172kB, anon-rss:760212kB, file-rss:29852kB
Apr 26 11:30:21 F5BIGIP warning kernel: avrd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
...
Apr 26 11:30:22 F5BIGIP err kernel: Out of memory: Kill process 18113 (mysqld) score 26 or sacrifice child
Apr 26 11:30:22 F5BIGIP err kernel: Killed process 18113, UID 98, (mysqld) total-vm:506428kB, anon-rss:198096kB, file-rss:3280kB

 

/var/log/ASM.log

 

Apr 26 11:15:53 F5BIGIP crit ControlPlane.CorrelationEngine[21066]: ControlPlane.CorrelationEngine|CRIT|Apr 26 11:15:53.651|21141| [read_message() at CorrelationEngine.cpp:1845] flwctl_read() - Failed to receive event from flow control socket. Program terminated.

Apr 26 11:16:19 F5BIGIP crit perl[18881]: 01310027:2: ASM subsystem error (asm_start,F5::NwdUtils::Nwd::log_failure): Watchdog detected failure for process. Process name: bd, Failure: Insufficient number of threads (required: 8, found: 0).
Apr 26 11:16:33 F5BIGIP info perl[18881]: 01310053:6: Going to wait for bd_agent to finish pushing configuration to bd
Apr 26 11:16:54 F5BIGIP info perl[18881]: 01310053:6: bd_agent has finished pushing configuration to bd
Apr 26 11:16:56 F5BIGIP crit perl[18881]: 01310027:2: ASM subsystem error (asm_start,F5::NwdUtils::Nwd::log_failure): Watchdog detected failure for process. Process name: pabnagd, Failure: Insufficient number of threads (required: 2, found: 0).
Apr 26 11:16:57 F5BIGIP crit perl[18881]: 01310027:2: ASM subsystem error (asm_start,F5::NwdUtils::Nwd::log_failure): Watchdog detected failure for process. Process name: correlation, Failure: Insufficient number of threads (required: 1, found: 0).

Apr 26 11:30:20 F5BIGIP crit ControlPlane.CorrelationEngine[31036]: ControlPlane.CorrelationEngine|CRIT|Apr 26 11:30:20.066|31040| [batch_all() at CorrelationEngine.cpp:1572] mysql_commit() - Failed to commit the current transaction: Lost connection to MySQL server during query. Program terminated.
Apr 26 11:30:25 F5BIGIP crit g_server.pl[21148]: 01310027:2: ASM subsystem error (asm_config_server.pl,F5::ASMConfig::Handler::handle_error): Failed update for device_group_member [host = '/Common/F5BIGIP.runt.com.co']
Apr 26 11:30:25 F5BIGIP crit g_server.pl[21148]: 01310027:2: ASM subsystem error (asm_config_server.pl,F5::ASMConfig::Handler::handle_error): Code: 999, Error message = DBD::mysql::st execute failed: MySQL server has gone away at /usr/local/share/perl5/F5/ASMConfig/Entity/Base.pm line 1296.
Apr 26 11:30:25 F5BIGIP crit g_server.pl[21148]: 01310027:2: ASM subsystem error (asm_config_server.pl,F5::ASMConfig::Server::start): Relay handler shutdown: Can't use string ("DBD driver has not implemented t") as a HASH ref while "strict refs" in use at /usr/local/share/perl5/F5/ASMConfig/Handler.pm line 330.
Apr 26 11:30:27 F5BIGIP crit g_server.pl[21148]: 01310027:2: ASM subsystem error (asm_config_server.pl,): ASM Config server died unexpectedly
Apr 26 11:30:27 F5BIGIP crit g_server.pl[21148]: 01310027:2: ASM subsystem error (asm_config_server.pl,): Can't use string ("DBD driver has not implemented t") as a HASH ref while "strict refs" in use at /usr/local/share/perl5/F5/ASMConfig/Handler.pm line 330.

 

  • are you sharing information, reporting an issue, looking for help?

     

  • to me it seems that your actions led to high cpu and memory exhaustion, the last one causing the mysql process to crash. that crash causes a ASM failure, which causes a fail over.

     

    please make a qkview while there is still enough data and open a ticket with F5 support, this should not happen and only they can point you to a probable cause.