Forum Discussion
smp_86112
Cirrostratus
CirrostratusGUI Operations Time Out
LTMs with 9.3.1HF8
All of a sudden our GUI operations seem to be timing out. For example with ConfigSync, there is no evidence in the LTM log indicating an attempt was made. Updating an iRule and mananging a user account appear in the LTM log, but the GUI simply times out. I have attempted an orderly restart of httpd and tomcat4.
The both the /var/log/webui.log and /var/log/tomcat4/catalina.out have errors like this:
2010-01-25 08:29:07,050 ERROR [Thread-18] io.MCPConnectionBroker:freeConnection - ----> Error: Could not free connection!!!
2010-01-25 08:29:07,053 ERROR [Thread-18] user.UserCreateHandler:afterTrigger - common.error.MCPErrorNull message reply.
java.sql.SQLException: null in statement [SET COMMITTOFILE NOW]
at org.hsqldb.Database.execute(Database.java:566)
at org.hsqldb.jdbcConnection.executeStandalone(jdbcConnection.java:3061)
at org.hsqldb.jdbcConnection.execute(jdbcConnection.java:2629)
at org.hsqldb.jdbcStatement.fetchResult(jdbcStatement.java:1840)
at org.hsqldb.jdbcStatement.execute(jdbcStatement.java:764)
at com.f5.model.db.DBConnection.executeNoRS(DBConnection.java:294)
I have already opened a case. Seems like tomcat can't talk with mysql? I was thinking about restarting mysql with bigstart restart mysql, but I'm concerned that will interrupt traffic processing and I am not in a position to failover.
hoolioRestarting mysql for a ASM/WAM licensed unit would most likely cause an outage. Your best bet might be to wait for an official response from F5 or wait until you can fail over and restart mysql on the failing unit.
Aaron
smp_86112Thanks for the response hoolio. We have ASM licensed, but are not actively using it. I don't see anything wrong on the Standby. If I fail over, I am going to reboot the unit. But that would be a very bad thing to do from a business perspective. I need to try and figure this out without affecting traffic processing.
I am waiting for F5 now.- hoolioIf you're not using ASM, then restarting mysql shouldn't have much affect. But better to be safe than sorry.
You could check the /var/lib/mysql directory for the mysql.err file and see if there's anything interesting there. Do you see any errors in the GUI when accessing the web applications page? If so, do you see anything in /var/log/asm, /var/log/webui.log or /var/log/httpd/*?
Aaron - smp_86112I thought ASM was licensed, but it's not - we own the license, but apparently have not installed it yet. Thanks for the references - the webui.log has similar info to catalina.out:
2010-01-25 07:56:41,233 ERROR [Thread-20] io.MCPConnectionBroker:freeConnection - ----> Error: Could not free connection!!!
2010-01-25 07:56:41,244 ERROR [Thread-20] util.Syscall:call - Exception trying perform a system call.
java.sql.SQLException: Null message reply. in statement [SELECT SINGLE * FROM shell_call WHERE command = 2 AND args = ' users']
at org.hsqldb.Database.execute(Database.java:566)
at org.hsqldb.jdbcConnection.executeStandalone(jdbcConnection.java:3061)
I see some stuff in the httpd_errors log, but no reference on ask:
Jan 25 09:22:09 BigIP1 httpd[29145]: [error] Error ajp_process_callback - write failed
Jan 25 09:22:09 BigIP1 httpd[29145]: [error] ajp13.service() ajpGetReply recoverable error 3
Jan 25 09:22:09 BigIP1 httpd[29145]: [error] ajp13.service() Error forwarding ajp13:localhost:8009 1 0 - smp_86112Just to close the loop on this, F5 advised we were affected by two CRs: CR91634 and CR37431, both relating to the size of the config. The recommendation is to full_box_reboot, or upgrade to 9.4.5.