Troubleshooting traffic spike on loadbalancer

[SivaYenduri]

Hello All,

Can someone help me how can i troubleshoot this issue?

Customers are accessing a VIP:8080 and there are around 2k connections on the VIP.

But at 10:00AM CET everyday, connections are going form 2k to 5k+ for 10mins and then going to back to normal.

Traffic flow is as below

Intranet > DC WAN router > Firewall > router > Loadbalancer > Backend servers.

Please help me how to identify this sudden spike in traffic.

[rafaelbn · Accepted solution]

Hello Siva!

Do you happen to know the source IP of all those connections?

I would start by looking at the connection table during that spike. Something like this...

tmsh show sys connection cs-server-addr <VIP> cs-server-port 8080

If everything comes from the same IP it could possibly narrow your tshoot down to a specific client.

Best of luck!

[SivaYenduri]

Thank you for the reply. Unfortunately source is entire corp network.. But this command is helpful.

[F5SJ_]

Please run bigtop -n from bash, it will show you VIPs with highest number of connections. Then please check the timeout value of tcp profile assigned to the VIPs. I faced the same issue and I reduced the timeout value of tcp profile and I saw that all long running connections deleted from the connection table.