Strange traffic to switches...maybe loop?

[Nils Sandborg]

Hi!

I have some strange traffic-flow in my Cisco 4500.

First of all, I have two C4500. One STP root and one secondary root.

In the network I got both C2960 and C2950. These are connected as a triangle.

We have about 40 switches in the network.

Pri Root      Sec Root

4500_1    4500_2

      \         /

       \       /

       C2960

If i look on the secondary stp-root switch, I got some strange traffic-pattern on the interfaces:

GigabitEthernet4/1 is up, line protocol is up (connected)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 43975000 bits/sec, 15586 packets/sec
GigabitEthernet4/2 is up, line protocol is up (connected)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 43975000 bits/sec, 15586 packets/sec
GigabitEthernet4/3 is up, line protocol is up (connected)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 43975000 bits/sec, 15586 packets/sec
GigabitEthernet4/4 is up, line protocol is up (connected)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 43975000 bits/sec, 15586 packets/sec
GigabitEthernet4/5 is down, line protocol is down (notconnect)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
GigabitEthernet4/6 is up, line protocol is up (connected)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 43975000 bits/sec, 15586 packets/sec
GigabitEthernet4/7 is up, line protocol is up (connected)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 43977000 bits/sec, 15586 packets/sec

As you see, there is only output traffic, and a lot of it!

Where should i begin to look?

I have some trouble with cpu-spikes on the 4500 too.

I can provide you with logs/output if you want to!

Regards

Nils

[Bert Gevers]

Hi Nils,

looking at the counters being almost equal, this looks like either broadcast/multicast or unknown unicast flooding.

The easiest next step would probably be to perform a span session from one of the interfaces to determine which traffic is being flooded.

Using this data, you can then determine where it's coming from and why it's being flooded.

Best regards,

Bert

[Antonio Knox]

How many vlans do you have?  Is this switch secondary for ALL vlans or certain ones?

Message was edited by: Antonio Knox

[Nils Sandborg]

I'll try to look at the traffic with a trafficsniffer, like wireshark?

We have about 60 Vlans and the root bridge is root for all of them.

So the secondary root should be passive in my opinion..?

Here is the CPU history of the pri root. (a bit twisted but you can see the spikes)

100       *

90       *                                           *

80  **  ** *  * * *                           *  * *** ***  ** *****    **

70  **********************************************************************

60  **********************************************************************

50  **********************************************************************

40  **********************************************************************

30  ######################################################################

20  ######################################################################

10  ######################################################################

    0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

              0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

[Giuseppe Larosa]

Hello Nils,

secondary root bridge ports are in STP forwarding state on its side?

you can check this with

show spanning-tree interface

if so this a form of flooding as noted by Gert and might be normal in your scenario.

Hope to help

Giuseppe

[Nils Sandborg]

Hi Giuseppe!

Do you mean that 40Mbit/s is normal flood traffic on a blocked port?

Here is the output from the pri root/bkp root bridge.

4500_1 (Primary)

4500_1#sh spanning-tree blockedports

Name                 Blocked Interfaces List
-------------------- ------------------------------------

Number of blocked ports (segments) in the system : 0

-------------------------------------------------------------------------------------------------

4500_1#sh spanning-tree summary totals 

Switch is in pvst mode
Root bridge for: "ALL"
Extended system ID           is enabled
Portfast Default             is disabled
PortFast BPDU Guard Default  is disabled
Portfast BPDU Filter Default is disabled
Loopguard Default            is disabled
EtherChannel misconfig guard is enabled
UplinkFast                   is disabled
BackboneFast                 is disabled
Configured Pathcost method used is short

Name                   Blocking Listening Learning Forwarding STP Active
---------------------- -------- --------- -------- ---------- ----------
70 vlans                     0         0        0       2328       2328

4500_2 (Backup)

4500_2#sh spanning-tree blockedports

Name                 Blocked Interfaces List
-------------------- ------------------------------------
VLAN0001             Gi1/1

Number of blocked ports (segments) in the system : 1

---------------------------------------------------------------------------------------------------

4500_2#sh spanning-tree summary totals 
Switch is in rapid-pvst mode
Root bridge for: none
Extended system ID           is enabled
Portfast Default             is disabled
PortFast BPDU Guard Default  is disabled
Portfast BPDU Filter Default is disabled
Loopguard Default            is disabled
EtherChannel misconfig guard is enabled
UplinkFast                   is disabled
BackboneFast                 is disabled
Configured Pathcost method used is short

Name                   Blocking Listening Learning Forwarding STP Active
---------------------- -------- --------- -------- ---------- ----------
70 vlans                     1         0        0       2170       2171

[Nils Sandborg]

I've checked all the switches in the network, they are all blocking the port to 4500_2 and the port to 4500_1 is unblocked.

So spanning tree seems to be OK in the network...

[Bert Gevers]

Hello Nils,

I do not believe 40Mb is normal and given that the speeds do not reach linerate, I don't expect this to be a loop.

Would it be possible to perform a short sniffer capture to determine which traffic is being flooded?

Cheers,

Bert

[Nils Sandborg]

Thanks for your answares

I've used a sniffer to check the traffic.

It seems to be ordinary traffic, a lot of traffic from different ip-addresses.

It's strange that the traffic flows on the blocked STP port...?!

[Nils Sandborg]

Hi

If I look "at the other side" the counters match.

So there is traffic at both sides.

[Calin C.]

GigabitEthernet4/1 is up, line protocol is up (connected)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 43975000 bits/sec, 15586 packets/sec

Just an idea beside the ones already said by other members.

Let's take the port Gi4/1. I assume that in the other end of this port, there is switch. On that "remote" switch, the counters on the corresponding interface are showing appropiate values with the above ones? (e.g. a lot of input and very less output packets).

If the values are aprox. the same, then you know traffic is really flowing there. Otherwise, I don't know, check for some bugs on your IOS version that might cause this behavior.

Cheers,

Calin

[Antonio Knox]

Have you checked Netflow for a heavy talker

sh ip cache flow

If someone is pushing 43975000 bits/sec they should stick out like a sore thumb.  Find that ip, and if it's an ip that shouldn't be pushing that sort of traffic, then find it in arp and track it down by mac address so you can find out what is going on.

[Nils Sandborg]

I noticed that the traffic is addressed to machines that doesn't exist behind that switch.

So the traffic should not go to that switch at all...

[Giuseppe Larosa]

Hello Nils,

>> I noticed that the traffic is addressed to machines that doesn't exist behind that switch.

the question is : the backup distribution switch that is sending traffic out those ports knows the destination MAC addresses of the forwarded traffic or not?

if it does not, then this is flooding:  unknown unicast are sent out of all ports except the one the frame is received.

so you should check with

show mac address-table  address

  

or

show mac-address-table  address

 

on the

backup distribution switch

the port is blocked on the access layer switch side not on the backup distribution switch side.

This is also a point to consider.

Hope to help

Giuseppe

[Jon Marshall]

Nils

I think Giuseppe is right. STP only blocks one end of the link and it will usually block the access-layer end. So the 4500 end is still forwarding. However because no mac-addresses will be learnt from the access-layer switch on that port under normal conditions you would see minimal traffic.

But if you have a lot of traffic being forwarded from the 4500 switches to the access-layer switches with unknown mac-adddresses they will simply be broadcast out of all ports on the 4500 and i think this is what you are seeing.

Jon