Forum Discussion

SivaYenduri's avatar
Feb 28, 2020

Troubleshooting traffic spike on loadbalancer

Hello All,

 

Can someone help me how can i troubleshoot this issue?

 

Customers are accessing a VIP:8080 and there are around 2k connections on the VIP.

But at 10:00AM CET everyday, connections are going form 2k to 5k+ for 10mins and then going to back to normal.

Traffic flow is as below

Intranet > DC WAN router > Firewall > router > Loadbalancer > Backend servers.

 

Please help me how to identify this sudden spike in traffic.

 

  • Hello Siva!

    Do you happen to know the source IP of all those connections?

    I would start by looking at the connection table during that spike. Something like this...

    tmsh show sys connection cs-server-addr <VIP> cs-server-port 8080

    If everything comes from the same IP it could possibly narrow your tshoot down to a specific client.

    Best of luck!

  • Hello Siva!

    Do you happen to know the source IP of all those connections?

    I would start by looking at the connection table during that spike. Something like this...

    tmsh show sys connection cs-server-addr <VIP> cs-server-port 8080

    If everything comes from the same IP it could possibly narrow your tshoot down to a specific client.

    Best of luck!

    • SivaYenduri's avatar
      SivaYenduri
      Icon for Cirrus rankCirrus

      Thank you for the reply. Unfortunately source is entire corp network.. But this command is helpful.

  • Please run bigtop -n from bash, it will show you VIPs with highest number of connections. Then please check the timeout value of tcp profile assigned to the VIPs. I faced the same issue and I reduced the timeout value of tcp profile and I saw that all long running connections deleted from the connection table.