Forum Discussion
CraigM_17826
Altocumulus
AltocumulusWeb managment interface drops connection after 36hrs or so
Hi everyone,
just wondering if anyone else has run into this issue or not. Due to our BIGIPs crashing randomly (due to what we now appears to be a possible memory leak issue) we decided to leave a web session to the unit's active, displaying the performance stats, specifically ram usage. The PC running the session to the two units displays on a central 52" LCD monitor so everyone in support can see the status.
What is happening is that after 36hrs or so the connection to one of the unit is lost. If I attempt a refresh it just states that an internal error occured and to check the logs. If I close the browser and reconnect it is fine.
I've looked at the logs on the offending unit and in the webui.log I found this
2009-08-04 15:25:12,255 ERROR [Thread-21] controller.F5WebController:processWebResponse - Exception thrown in processWebResponse: java.lang.NullPointerException
java.lang.NullPointerException
at web.common.formredirect_jsp._jspService(formredirect_jsp.java:123)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:432)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:356)
at com.f5.controller.F5WebController.processWebResponse(F5WebController.java:157)
at com.f5.controller.F5WebController.processWebRequest(F5WebController.java:123)
at com.f5.controller.ControlServlet.doGet(ControlServlet.java:172)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:261)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:360)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:604)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:562)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:679)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:619)
at java.lang.Thread.run(Unknown Source)
Going by the date/time of this error, this occured when I tried to refresh the page.
Now this does not happen on the 2nd BigIP unit so it seems odd the first unit is doing this. The 1st unit is in standby mode.
Has anyone else had this happen?
We are running the latest 9.4 build of the firmare on both units.
tia
Craig
hoolioCirrostratus
I expect F5 Support will be your best option for troubleshooting this as it would help to see the full logs/metrics in a tech.out when the failure occurs.
Aaron
Haarith_DevarajNimbostratus
my two cents. * Caution: consult with support before making changes.
We have seen very significant benefit by increasing the allocated host memory. So much that 3 years back, we pushed for a DB change to be included in new software versions instead of the old grub based change. The default linux alloc is way too low. It however depends on what platform and software version you are running. If you are running redundant 3600's for eg, you can allocate like 2Gigs to your host. Very very simple change.
** AGAIN CAUTION: Check with support. This is just for information
9.4.6
> do "b memory show all " and see reserved host vs tmm if you have the default ~300M
> bigpipe db Provision.ExtraMB
( should show 0)
> bigpipe db Provision.ExtraMB (whatever you want - current)
> grubmem
for older releases
grub_open /mnt/tmp/install/boot/hda/grub/grub.conf
vi /mnt/tmp/install/boot/hda/grub/grub.conf
grub_close
Note: You must not be in the /tmp parent directory when you run the grub_close
command; the command will fail because the mounted file system is still busy.
edit tmm
/etc/bigstart/scripts/tmm