Latency on switch

[dbuckley77]

We have several 2960 switch connected to our network with 1g fiber that we're seeing latency on.  I'm pining continuously from a desktop on the network and getting mostly <1ms but every so often I will see spike from 500-1000ms or more.

What would be a good starting point to address this?

[Cristian Matei]

Hi, 

 @dbuckley77 Couple of questions, to first try and isolate RCA as well as understand the E2E path.

Do you ping to a switch IP, or to an end host? Is the destination in the same VLAN (so E2E path is layer 2) or is the destination I another VLAN (so E2E path is layer 2 + layer 3)? If it's layer 3, do you have routers, firewall, or both in the path? Do you encounter the same behaviour when changing either the destination, either the source of the ping?

  Can you enable, on all switches spanning-tree logging and check the logs to see if you have any constant or sporadic STP port / role changes? What SW version do you run on these switches?

Thanks,

Cristian.

[dbuckley77]

I'm pinging from a different vlan but using the same source IP.  I could try a different one from a different vlan and one formt he same vlan as the destination IP.  We have a Cisco 6807 running VSS that  does our routing between networks.  No firewalls.  This is all inside with no ACLs or anything, it's all in the same routing table on our 6807.

We have a dedicated vlan/network for all of our switches and out of close to 100 switches we're only seeing the latency issue on a few of them so the issue does appear to be on the destination side.  I just tried pinging from the same source to a different destination in the same vlan/network as the one that's experiencing the latency and I'm not seeing the issue.

Also, we're running VOIP and data  across all our switches and not experiencing any real issues with either of these.  The reason we're looking into this is because we have a host monitoring system that uses ping to determine downed hosts and it's sending us a lot of false alerts.  I'm trying to tweak the settings on that but is it normal on most networks to see occasional spikes when doing continuous pings?  I've checked the trunk ports on the switches with latency and am not seeing any errors/drops at all.  

[Giuseppe Larosa]

Hello @dbuckley77 ,

when you ping a switch SVI IP address the switch processes it with no special priority.

I mean that what you see is more likey caused by switch CPU usage with its own peaks rather then link saturation on the uplinks even if they are just 1 Gbps

Hope to help

Giuseppe

[Mark Elsen]

  - @dbuckley77    I don't consider 'user pings' as being a definitive judge on latency , as it may get lower priority if the
                             switch and or switch-chain (network)  is under high load.  ICMP  ping may get lower priority.
                             Application impact should for   instance be monitored with tools such as iperf. 

 M.

[Cristian Matei]

Hi,

  @dbuckley77 Try to isolate the physical path and / or the switches that seem to be related to the issue you're seeing. Enable STP logging, as suggested, on all switches, and see if you have any STP logs. Collect and post outputs of following commands from switches you've isolated to be part of the problem:

show buffers
show buffers old 
show buffers failures 

     Errors / drops would result in packet drops, not increased latency.

Thanks,

Cristian. 

[balaji.bandi]

It's hard to say what the issue is - you need to find where the latency is added in the Hop.

is this Latency any specific time? Or peak hours of the network? Do you have any tool which capture this information and recorded format weekdays vs weekend ?

You need to start gathering information :

show processes cpu sorted | exclude 0.00%

Check any drops :

show interfaces counters errors and show interfaces | include drops|input rate|output rate

Check STP topology change :

show spanning-tree detail | include from|occurrence

Check any Broadcast and multicast stroms :

show interfaces stats

[M02@rt37]

Hello @dbuckley77 

Check if control-plane is busy:

show processes cpu sorted

Check also interfaces, if there are CRC errors, input drop, underruns...

Also, check STP events... if you see frequent topology changes...

[roseanglina221]

This kind of behavior can be really frustrating. Seeing sub-1ms latency most of the time and then random 500–1000ms spikes makes it feel unpredictable and hard to trust the network. On a wired 1G fiber setup, those sudden jumps definitely don’t feel “normal,” especially when everything looks fine in between

[Mark Elsen]

  - @roseanglina221     I conclude the reverse ; it could be perfectly normal if application traffic get's priority.

   M.
                                 

[Joseph W. Doherty]

---

@dbuckley77 wrote:

What would be a good starting point to address this?

---

Perhaps starting with the understanding (as some have already mentioned), ping RTT isn't guaranteed to be accurate.  This because, a host receiving an ICMP echo request, doesn't have to "promptly" perform an ICMP echo reply, or, actually, reply at all.  My understanding (?) is, the ICMP echo feature was principally for determining if a host is on-line and reachable.

So, perhaps the best starting point, if you're going to monitor RTTs, would be to use something that works to better insure accuracy.  (Like Cisco's [formerly router] IP SLA responder.)  (In theory, the ICMP Timestamp echo and its reply, could provide improved RTT accuracy, but as it's optional [I believe] and as many hosts won't support if for security reasons, and for RTT monitoring, you would probably want both hosts on NTP, you might do better using something else.)

I will say, I don't ever recall seeing such a legitimate variance, i.e. from less than 1 ms to up to 1 second.

Are you also seeing any total ping loss, i.e. timeouts on the reply?