Selectively Disabling Keep-Alives

tcpdump was run on the LTM. we were sniffing traffic between "everything" and the LTM as well as traffic between the LTM and the web servers. "Everything" is in quotes because we were testing against an identically configured stack as our production site. (Same LTM. Different web servers and app servers--although the servers in the lab are a bit beefier than production. We have a swing environment set up of two identical silos of servers. Identical numbers of web servers, app servers, database servers, etc. and we use shell scripts on the LTM to detach and attach pools to point traffic to a particular silo but that's another long story)

 

 

At the same time we ran the trace, the mod_jk log level on the web servers was set to 'debug'. While my colleague was running the tcpdump, I was staring at httpfox on my local machine as well as tailing the logs on the web servers so I neglected to run wireshark locally as well.

 

 

On our test bed, when the problem occurs, the transaction never completes. The nature of this particular bug places so much data into the jvm's memory that the jvm may very well perform full garbage collections forever. In production, due to the volumes of traffic we have, this bug ultimately results in the web servers hitting MaxClients and no longer being able to service any kind of request. The weird thing about this particular problem, which is outside the scope of this particular thread but is in-scope for a ticket with JBoss, is that even though only 1 of the 6 app servers is in Full GC mode, threads on all application servers quickly pile up until we're out of threads on all app servers, the web servers are at MaxClients, and users are down. It's a nasty bug to say the least!