Input drops on VLAN's, output drops on switchports for that VLAN

[richard_tufaro]

Hi Group. Been working on a problem, im hoping to get some good advice.

I have just installed 2 new 6000's both with dual MSFC2's, PFC2's etc. They are doing HSRP using the same group number (to get around that unique 16 group limitation). For simplicity, let’s say I have an access switch, dual homed to both switches.

My problem is, on each of the router interfaces on both primary and secondary chassis im getting HUGE amounts of INPUT DROPS on VLAN interfaces, and on the access layer switches im getting huge amounts of OUTPUT DROPS on switch ports.

My first thought, was there is either a VLAN that is crossed in the network and the router is seeing packets not for there respective VLAN and dropping them, or I have a broadcast storm somewhere.

Neither seems to be the case. I put a sniffer on the VLAN's that are having the issue, and I don’t see any broadcast packets for any subnet except for the subnet that I plugged the sniffer into.

The VLAN's are not crossed anywhere, iv looked at spantree roots for each VLAN and verified that the root bridge mac is not the same. Meaning that VLAN's are not hearing each others BPDU's.

So anyone have any other suggestions?

[skarundi]

I would troubleshoot the input drops and output drops as 2 separate issues.

troubleshooting input drops on the msfc:

If you are running in hybrid mode, i.e. Catos on the supervisor and IOS on the MSFC, then you can do 2 things to determine the source of the traffic causing the input drops

a) Issue the "show interface switching" command ( type out the whole command..) to see the breakdown of traffic received on the msfc interfaces. How often does it go up ? It is continuous or it goes up in bursts ?

b) Next, sniff the MSFC interconnect ( 15/1 or 16/1) to determine the source of the traffic causing the drops. You can use the "set span" command to achieve that.

troubleshooting output drops:

If you are running CatOS on the switches reporting output drops, issue the "show counters " command to get a complete breakdown of all ethernet counters. This will show you why the switch port is report output drops. Normally it is due to congestion, but in rare cases, it could be due to something else.

If it is due to congestion, check the xmit unicast/broadcast/multicast traffic to make a guess at what traffic may be causing the congestion. Next take a sniffer trace of the affected port and wait for the output drops to increase. Analyse the sniffer trace to determine the source of the congestion.

[richard_tufaro]

Hi. Thanks for your reply.

Let me preface my reply by saying that, this is a network upgrade from 5000's with sup1's.

That being said. I have issued the "show interface switching" command and seen that traffic is very bursty...but yet at a current rate of burst-e-ness.

There are no cache-misses on the interfaces so i figure that the traffic is destined FOR the router not going through it.

There is no way that i can sniff the interconnect, my sniff connection would be way to slow. We have a network of about 100 switches (5000's/6000's).

I did sniff an access layer port, and did not see anything out of the ordinary.

The only thing that i could find that is strange, is when issuing the "show ip traffic" command on the core router. I have ALOT of "encapsulation failed" messages.

Any other thoughts on this?

IP statistics:

Rcvd: 95267975 total, 53503074 local destination

0 format errors, 0 checksum errors, 9777982 bad hop count

0 unknown protocol, 0 not a gateway

0 security failures, 170 bad options, 2699937 with options

Opts: 12 end, 0 nop, 0 basic security, 0 loose source route

0 timestamp, 0 extended security, 12 record route

0 stream ID, 0 strict source route, 2699755 alert, 0 cipso

0 other

Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble

2 fragmented, 0 couldn't fragment

Bcast: 26147579 received, 12894 sent

Mcast: 15611091 received, 11224217 sent

Sent: 17623617 generated, 728368 forwarded

Drop: 38190253 encapsulation failed, 0 unresolved, 0 no adjacency

12579 no route, 0 unicast RPF, 0 forced drop