RSM CPU freezes with small burst of traffic

[daniel.bowen]

I have a Catalyst 5500 with integrated RSM that is responsible for routing between 16 VLANs and routing our Internet bound traffic. The three busiest interfaces handle around 1,000 to 2,000 pps 95% of the time with no problem. Then, due to a burst of traffic coming from VLAN 88, the interface for that VLAN encounters thousands of throttles, dropped packets and causes the CPU to hit 100% and freeze. On analysing the traffic, there is nothing that great that sticks out as going between VLAN 88 and 80. Does anybody know of any problems that can cause this.

Thanks for your time

[paquino]

From what you say a small burst intraffic causes the switched to see thousands of throttles, dropped packets etc. My first question would be is the burst of traffic equal to the packets counters on the ports. You could get a feel for this by using a packet sniffing device and compare capture through the net ( knowing where traffic is originating from and destined ). Some data traffic patterns may be causing errors or flooding through the net. Examples of this would be spanning tree loops,trunks forwarding vlan traffic which supposedly should be blocked, and port buffers erroring out (hardware). What kind of traffic is this and what is the path from A to Z?

You may also want to check out the release notes for your version of code just in case this is a known issue.

Sincerely,

Paul Aquino

CSE

Cisco Systems

[daniel.bowen]

The show proc cpu says 20% IP Input when this happens. Adding all the individual percentages adds up to about 30%, meaning 70% is traffic. I have fast switching configured on both subnets. When I do a packet sniff, there is no amount of traffic that stands out as high, there is a lot of NCP and NBURST traffic, and a bit of NetBIOS but nothing that adds up to enough to kill the RSM.

Any ideas?

PS - thanks for your help

[glen.grant]

You don't have "no ip route-cache" configured on either interface that is forcing all packets to be process switched do you ? This can make a big difference if you are process switching every packet . Verify this by doing "show ip interface" and make sure fast switching is on for both subnets . We saw a problem like this when we had to process switch decnet and it would drive the CPU to 90-100 % , we had it turned off due to a known software bug . Once we got the box upgraded and turned decnet switching turned back on it never climbs above 10-15 % . I would check for something like this if you feel the traffic does not justify how high the CPU is running . What does show proc cpu show you as the cause of the high cpu ?

[daniel.bowen]

The show proc cpu says 20% IP Input when this happens. Adding all the individual percentages adds up to about 30%, meaning 70% is traffic. I have fast switching configured on both subnets. When I do a packet sniff, there is no amount of traffic that stands out as high, there is a lot of NCP and NBURST traffic, and a bit of NetBIOS but nothing that adds up to enough to kill the RSM.

Any ideas?

PS - thanks for your help