I wonder if anyone has come across this issue - I have recently upgraded a 3750E with and 3750X which is currently up and running in production environment.
Connectivity of services connected to the switch are reported as good.
The switch sends back a continuous and steady ping response.
The issue is when trying to look at show commands - the screen output is deadly slow. It takes a good few minutes to scroll through the running config. the output displays half of what it should hangs then reluctantly displays some more.
This is causing havoc for my backup schedule as the TFTP server is just seeing it timeout, as a result no back-ups of the config
Any suggestions would be appreciated.
> The switch sends back a continuous and steady ping response.
I am wondering, the switch sends ping response, to which destination address ? Are we sure specific device not sending continuous ping request ?
For slowness with "show commands", did you check the CPU usage ? If possible, send:
show process cpu sorted | exclude 0.00
Conitnuous ping is from my management server and also from a switch directly connected to the switch in question.
I can see actually that the CPU usage is fairly high, more than I would expect for a 3750X. The new 3750 was taken from a stack of switches as an emergency replacement for the original 3750E. The configuration is basic, switching only with maybe 16 live interfaces.
switch-3750X#show process cpu sorted | exclude 0.00
CPU utilization for five seconds: 94%/1%; one minute: 61%; five minutes: 52%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
193 5762883 865373 6659 64.07% 39.50% 35.03% 0 IP Input
307 2056 8625 238 16.39% 1.41% 0.29% 1 Virtual Exec
71 6564133 1181077 5557 2.10% 2.16% 2.20% 0 RedEarth Tx Mana
113 4107236 242950 16905 1.05% 1.35% 1.43% 0 hpm counter proc
152 597273 5805383 102 1.05% 0.86% 0.80% 0 Hulc LED Process
70 1316678 1792190 734 0.75% 0.54% 0.60% 0 RedEarth I2C dri
208 2744716 9069458 302 0.60% 0.70% 0.77% 0 Spanning Tree
16 9702 56113 172 0.60% 0.08% 0.03% 0 EEM ED Syslog
162 929699 49922 18623 0.45% 0.35% 0.35% 0 HQM Stack Proces
243 9383 2105 4457 0.15% 2.24% 0.49% 0 TCP Timer
221 1322684 242949 5444 0.15% 0.25% 0.25% 0 PI MATM Aging Pr
163 41047 99824 411 0.15% 0.05% 0.03% 0 HRPC qos request
Any idea what could be causing this high CPU?
The process that has highest CPU usage is "IP Input". This indicates hgih number of IP packets are sent to the CPU.
"Virtual Exec" process is due to Telnet/SSH.
First thing I would do is to stop the ping requests from the management station, and see the CPU usage goes down.
If the CPU usage is still high, then we need to find the traffic hitting the CPU and find it is legitimate.
Catalyst 3750 Series Switches High CPU Utilization Troubleshooting:
Troubleshooting High CPU Utilization
I have no ping requests running at present. The problem appears to be getting worse - it has taken nearly 5 minutes to display the output to 'sh processes cpu | exc 0.00' command.
I have just noticed that the ping response has increased...
Pinging switch-3750 [192.168.50.X] with 32 bytes of data:
Reply from 192.168.50.X: bytes=32 time=149ms TTL=249
Reply from 192.168.50.X: bytes=32 time=910ms TTL=249
Reply from 192.168.50.X: bytes=32 time=633ms TTL=249
Reply from 192.168.50.X: bytes=32 time=365ms TTL=249
Ping statistics for 192.168.50.X:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 149ms, Maximum = 910ms, Average = 514ms
The switch I'm having problems with connects back to another switch via a 4 port Etherchannel which is fully operational. I'm starting to worry that the switch may eventually fall over. This would be worrying as for the spec of switch as it is nowhere near full capacity.
Any suggestions appreciated.
switch-3750X#show int po20 | i t rate
5 minute input rate 638000 bits/sec, 562 packets/sec
5 minute output rate 77000 bits/sec, 30 packets/sec
Is there a low-capacity WAN connection in front of this switch? If so, that may explain the slow response from commands.
If you have no ping requests sent to the Switch, and still have high CPU due to "IP Input" process, then you need to find out the traffic hitting the switch CPU.
First, you need to find out through which interface the CPU is getting all these traffic.
Use " show interfaces | inc line pro|Input " command (exaclty as given). Look for high number of packets in the queue, drops and flushses.
Also, you can use " show controllers cpu-int " to find which CPU queue has high number of packets received/dropped.
You may want to consider opening a TAC Service Request, so that the switch in issue can be accessed and troubleshot, for timley resolution.