2960x really slow ping response ms time

[Amafsha1]

Hello,

I have a branch that has a router, which when I ping it,  on average the response time is 10ms.  Attached to that router via 1gig trunk is a 2960x switch with some ip phones and ip cameras.  When I ping this switch, I'm getting 600ms respone time on pings and sometimes it even misses pings.  When I ssh into it, it's unbearably slow....really slow I can barley type. When I SSH into router, it's perfectly fine.

CPU usage on switch looks normal

2960x#sh processes cpu sorted | ex 0.00

CPU utilization for five seconds: 32%/0%; one minute: 31%; five minutes: 31%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
161 84375406 11813787 7142 15.41% 15.28% 15.22% 0 Hulc LED Process
6 744904 81099 9185 1.25% 0.17% 0.12% 0 Check heaps
131 3724403 472824 7876 0.65% 0.75% 0.77% 0 hpm counter proc
167 1665 697 2388 0.41% 0.08% 0.16% 1 SSH Process
172 1467630 94252 15571 0.23% 0.26% 0.24% 0 HQM Stack Proces
89 195356 21626024 9 0.05% 0.03% 0.05% 0 RedEarth Tx Mana

How does someone recommend me to tshoot this?

[Reza Sharifi]

Hi,

Can you check the interface between the router and the switch for any issue like packet drops?

Also, did this just started? Was there a config change recently on the router and/or the switch?

HTH

[Amafsha1]

The interface counter error was actually the first thing I checked and it's all clear.  No configuration changes have been made other than a couple of switchport changes.   The problem also comes and goes.  It's fine now, but in a couple hours it will be slow again. 

[Reza Sharifi]

Ok, that is good info. Since this is a remote site, I am assuming you are accessing it using an ISP. Has there been any issue with the provider?

HTH

[Amafsha1]

Yes, it's a remote site and there has been no issue with the ISP.  Also we have a router here that connects to the ISP and that router has no latency issues, but the 2960x switch that connects to the router does.  If there was an ISP issue I'm pretty sure the router would have latency issues too. 

[sathvik k v]

Hello,

If there is no problems reaching WAN router but you have slowness reaching the switch connected to router i would check the physical path between route and switch. Have you looked at interfaces for error/drops etc on both sides ?

I see CPU utilization is around 30%, was this taken while you were facing latency issue? Have you looked at memory of the switch?


-Sathvik

[Amafsha1]

yeah I checked to make sure there is no counter error on the switch end, but on the router end there is no such command....there is, but it doesn't show anything.

CPU Utilization is pretty much always 30% regardless of the issue or not.  What command should I use to check for memory problems?

[sathvik k v]

If health of switch is good, we need to check connectivity between switch and router. Though we don't see any errors on the link i would go ahead and change the port on the switch for testing. 

Would you be able to pull a packet capture spanning the uplink of the switch?

[Amafsha1]

I believe you are right and it probably will end up being the trunk between the 2, but it's hard for me to get someone to switch that out.  It's happening right now.  Is their any way you know that I can troubleshoot the memory to make sure that's not the issue before I finalize for someone to switch out the port?  I just want to be sure before dispatching someone to do that.

[sathvik k v]

Memory should not be a problem unless there is some process which pulls a lot of memory, you can check using show memory usage  and look for free memory.

Check CPU utilization while the issue re occurs and monitor for some time. 

[Amafsha1]

These are the only commands I have under memory:

allocating-process      Show allocating process name
dead                       Memory owned by dead processes
debug                     Memory debugging commands
failures                            Memory failures
fast                    Fast memory stats
fragment                          Summary of memory fragment information
free                        Free memory stats
io                  IO memory stats
lite-chunks                  Malloc lite info
multibus                        Multibus memory stats
overflow                        memory overflow corrections
pci                     PCI memory stats
processor                   Processor memory stats
statistics                 Mempool Statistics
summary                    Summary of memory usage per alloc PC
transient                Transient memory stats

[sathvik k v]

processor  and summary can be handy.

[Amafsha1]

Do you think the problem could be that ICMP packets aren't prioritized properly by the switch? So the switch doesn't look to process the ICMP packets right away?

[sathvik k v]

Well if it was rate limiting  you should have seen packet loss and not latency. 

[Andrew Woolman]

Sounds like a cable problem to me. Check out the TDR commands

https://community.cisco.com/t5/network-architecture-documents/how-to-use-time-domain-reflectometer-tdr/ta-p/3119327

and check for duplex issues

I would probably upgrade the IOS or at the very least reboot the switch