Connectivity to switch is ok, but show commands slow output

[ricardo1831]

I wonder if anyone has come across this issue - I have recently upgraded a 3750E with and 3750X which is currently up and running in production environment.

Connectivity of services connected to the switch are reported as good.

The switch sends back a continuous and steady ping response.

The issue is when trying to look at show commands - the screen output is deadly slow. It takes a good few minutes to scroll through the running config. the output displays half of what it should hangs then reluctantly displays some more.

This is causing havoc for my backup schedule as the TFTP server is just seeing it timeout, as a result no back-ups of the config

Any suggestions would be appreciated.

[Yogesh Ramdoss]

Hello,

> The switch sends back a continuous and steady ping response.

I am wondering, the switch sends ping response, to which destination address ? Are we sure specific device not sending continuous ping request ?

For slowness with "show commands", did you check the CPU usage ? If possible, send:

show process cpu sorted | exclude 0.00

Regards,

Yogesh

[ricardo1831]

Hi Yogesh,

Conitnuous ping is from my management server and also from a switch directly connected to the switch in question.

I can see actually that the CPU usage is fairly high, more than I would expect for a 3750X. The new 3750 was taken from a stack of switches as an emergency replacement for the original 3750E. The configuration is basic, switching only with maybe 16 live interfaces.

switch-3750X#show process cpu sorted | exclude 0.00

CPU utilization for five seconds: 94%/1%; one minute: 61%; five minutes: 52%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

193     5762883    865373       6659 64.07% 39.50% 35.03%   0 IP Input

307        2056      8625        238 16.39%  1.41%  0.29%   1 Virtual Exec

  71     6564133   1181077       5557  2.10%  2.16%  2.20%   0 RedEarth Tx Mana

113     4107236    242950      16905  1.05%  1.35%  1.43%   0 hpm counter proc

152      597273   5805383        102  1.05%  0.86%  0.80%   0 Hulc LED Process

  70     1316678   1792190        734  0.75%  0.54%  0.60%   0 RedEarth I2C dri

208     2744716   9069458        302  0.60%  0.70%  0.77%   0 Spanning Tree

  16        9702     56113        172  0.60%  0.08%  0.03%   0 EEM ED Syslog

162      929699     49922      18623  0.45%  0.35%  0.35%   0 HQM Stack Proces

243        9383      2105       4457  0.15%  2.24%  0.49%   0 TCP Timer

221     1322684    242949       5444  0.15%  0.25%  0.25%   0 PI MATM Aging Pr

163       41047     99824        411  0.15%  0.05%  0.03%   0 HRPC qos request

Any idea what could be causing this high CPU?

Regards,

Ric

[Yogesh Ramdoss]

Ricardo,

The process that has highest CPU usage is "IP Input". This indicates hgih number of IP packets are sent to the CPU.

"Virtual Exec" process is due to Telnet/SSH.

First thing I would do is to stop the ping requests from the management station, and see the CPU usage goes down.

If the CPU usage is still high, then we need to find the traffic hitting the CPU and find it is legitimate.

Catalyst 3750 Series Switches High CPU Utilization Troubleshooting:

http://www.cisco.com/en/US/partner/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml

Troubleshooting High CPU Utilization

http://www.cisco.com/en/US/partner/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

Regards,

Yogesh

[ricardo1831]

Yogesh,

I have no ping requests running at present. The problem appears to be getting worse - it has taken nearly 5 minutes to display the output to 'sh processes cpu | exc 0.00' command.

I have just noticed that the ping response has increased...

C:\>ping switch-3750

Pinging switch-3750 [192.168.50.X] with 32 bytes of data:

Reply from 192.168.50.X: bytes=32 time=149ms TTL=249

Reply from 192.168.50.X: bytes=32 time=910ms TTL=249

Reply from 192.168.50.X: bytes=32 time=633ms TTL=249

Reply from 192.168.50.X: bytes=32 time=365ms TTL=249

Ping statistics for 192.168.50.X:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 149ms, Maximum = 910ms, Average = 514ms

C:\>

The switch I'm having problems with connects back to another switch via a 4 port Etherchannel which is fully operational. I'm starting to worry that the switch may eventually fall over. This would be worrying as for the spec of switch as it is nowhere near full capacity.

Any suggestions appreciated.

[Antonio Knox]

What kind of utilization are you seeing on the etherchannel (show int po# | i t rate)

[ricardo1831]

switch-3750X#show int po20 | i t rate

  5 minute input rate 638000 bits/sec, 562 packets/sec

  5 minute output rate 77000 bits/sec, 30 packets/sec

[Antonio Knox]

Is there a low-capacity WAN connection in front of this switch?  If so, that may explain the slow response from commands.

[Yogesh Ramdoss]

Ricardo,

If you have no ping requests sent to the Switch, and still have high CPU due to "IP Input" process, then you need to find out the traffic hitting the switch CPU.

First, you need to find out through which interface the CPU is getting all these traffic.

Use " show interfaces | inc line pro|Input " command (exaclty as given). Look for high number of packets in the queue, drops and flushses.

Also, you can use " show controllers cpu-int " to find which CPU queue has high number of packets received/dropped.

You may want to consider opening a TAC Service Request, so that the switch in issue can be accessed and troubleshot, for timley resolution.

Regards,

Yogesh