Forum Discussion
hc_andy_35682
Nimbostratus
NimbostratusHigh connection rate (millions conns/sec)
Hi All,
Last night at close to 10.30pm, our cacti graphs showed a hugh spike in the number of connections/sec hitting the majority of our inside_vlans (forwarding vips) and virtual servers. This lasted just a few mintues and I have been unable to track it down to what might have caused this.
The only thing I saw in the logs were:
Mar 31 22:33:15 local/f5-1-manage emerg logger: Re-starting tmm1
Mar 31 22:33:15 local/f5-1-manage emerg logger: Re-starting tmm2
Mar 31 22:33:15 local/f5-1-manage emerg logger: Re-starting tmm3
Does anyone have a clue what might have happened? At first I thought it was a ddos attack but this isn't evident by looking at our router graphs which shows normal traffic levels.
Here's an example of the cacti graphs I'm talking about. Propel is a dial up acceleration application that our dial up users use. Port 80 is barely used. We'd be lucky to get 1-2 people hitting that page a day, but last night it took a hammering. Not sure if this might have just been a cacti or LTM bug???
Thanks.
Andy
hc_andy_35682The only other link I could find relating to the issue was that ICMP traffic for our wildcard VIP shot through the roof at the same time.
Cspillane_18296Hello hc_andy,
it's possible that if there were a sudden burst of traffic this is what caused the tmm process to restart. However, you mention that associated routers show normal traffic levels....which doesn't seem to match.
What version of TMOS are you running?
Are there any other devices using the forwarding VS's that might have been involved (aside from the routers), maybe a proxy server for instance?- L4L7_53191An aggressive port scanner could have caused this as well.
-Matt - L4L7_53191Sorry - just noticed the ICMP bit. That's interesting for sure.
- JRahm
Admin
sounds like an infrastructure event, perhaps a mistimed recovery in l2/l3 convergence? - hc_andy_35682Thanks to all who chipped in with a reply.
I've tracked the issue down to the active unit failing and the backup unit taking over (we have a HA setup). I can only assume that cacti goes a bit "crazy" when this sort of event happens hence why the connection rate is so high. Atleast the graphs pointed me to an issue.
I went searching the ltm logs and found that the F5 interfaces had gone down (maybe due to duplex mismatch).
Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160016:6: Interface 2.1, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down
Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160010:6: Link 2.1 removed from aggregation
Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160016:6: Interface 2.2, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down
Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160010:6: Link 2.2 removed from aggregation
Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160016:6: Interface 2.3, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down
Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160010:6: Link 2.3 removed from aggregation
Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160016:6: Interface 2.4, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down
Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160010:6: Link 2.4 removed from aggregation
Cheers.
Andy - hc_andy_35682We're running 10.1.0 so hopefully that bug you mentioned is no longer present.
We were advised by F5 support that the version we were using had 2 bugs affecting SSL and FTP that can cause the LTM to restart. We applied an engineering hotfix so hopefully the problem has gone away.
