05-11-2011 03:29 AM - edited 03-06-2019 05:00 PM
Hi!
I have some strange traffic-flow in my Cisco 4500.
First of all, I have two C4500. One STP root and one secondary root.
In the network I got both C2960 and C2950. These are connected as a triangle.
We have about 40 switches in the network.
Pri Root Sec Root
4500_1 4500_2
\ /
\ /
C2960
If i look on the secondary stp-root switch, I got some strange traffic-pattern on the interfaces:
GigabitEthernet4/1 is up, line protocol is up (connected)
5 minute input rate 1000 bits/sec, 1 packets/sec
5 minute output rate 43975000 bits/sec, 15586 packets/sec
GigabitEthernet4/2 is up, line protocol is up (connected)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 43975000 bits/sec, 15586 packets/sec
GigabitEthernet4/3 is up, line protocol is up (connected)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 43975000 bits/sec, 15586 packets/sec
GigabitEthernet4/4 is up, line protocol is up (connected)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 43975000 bits/sec, 15586 packets/sec
GigabitEthernet4/5 is down, line protocol is down (notconnect)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
GigabitEthernet4/6 is up, line protocol is up (connected)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 43975000 bits/sec, 15586 packets/sec
GigabitEthernet4/7 is up, line protocol is up (connected)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 43977000 bits/sec, 15586 packets/sec
As you see, there is only output traffic, and a lot of it!
Where should i begin to look?
I have some trouble with cpu-spikes on the 4500 too.
I can provide you with logs/output if you want to!
Regards
Nils
05-11-2011 03:33 AM
Hi Nils,
looking at the counters being almost equal, this looks like either broadcast/multicast or unknown unicast flooding.
The easiest next step would probably be to perform a span session from one of the interfaces to determine which traffic is being flooded.
Using this data, you can then determine where it's coming from and why it's being flooded.
Best regards,
Bert
05-11-2011 05:18 AM
How many vlans do you have? Is this switch secondary for ALL vlans or certain ones?
Message was edited by: Antonio Knox
05-11-2011 12:00 PM
I'll try to look at the traffic with a trafficsniffer, like wireshark?
We have about 60 Vlans and the root bridge is root for all of them.
So the secondary root should be passive in my opinion..?
Here is the CPU history of the pri root. (a bit twisted but you can see the spikes)
100 *
90 * *
80 ** ** * * * * * * *** *** ** ***** **
70 **********************************************************************
60 **********************************************************************
50 **********************************************************************
40 **********************************************************************
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
05-11-2011 01:33 PM
Hello Nils,
secondary root bridge ports are in STP forwarding state on its side?
you can check this with
show spanning-tree interface
if so this a form of flooding as noted by Gert and might be normal in your scenario.
Hope to help
Giuseppe
05-12-2011 01:09 AM
Hi Giuseppe!
Do you mean that 40Mbit/s is normal flood traffic on a blocked port?
Here is the output from the pri root/bkp root bridge.
4500_1 (Primary)
4500_1#sh spanning-tree blockedports
Name Blocked Interfaces List
-------------------- ------------------------------------
Number of blocked ports (segments) in the system : 0
-------------------------------------------------------------------------------------------------
4500_1#sh spanning-tree summary totals
Switch is in pvst mode
Root bridge for: "ALL"
Extended system ID is enabled
Portfast Default is disabled
PortFast BPDU Guard Default is disabled
Portfast BPDU Filter Default is disabled
Loopguard Default is disabled
EtherChannel misconfig guard is enabled
UplinkFast is disabled
BackboneFast is disabled
Configured Pathcost method used is short
Name Blocking Listening Learning Forwarding STP Active
---------------------- -------- --------- -------- ---------- ----------
70 vlans 0 0 0 2328 2328
4500_2 (Backup)
4500_2#sh spanning-tree blockedports
Name Blocked Interfaces List
-------------------- ------------------------------------
VLAN0001 Gi1/1
Number of blocked ports (segments) in the system : 1
---------------------------------------------------------------------------------------------------
4500_2#sh spanning-tree summary totals
Switch is in rapid-pvst mode
Root bridge for: none
Extended system ID is enabled
Portfast Default is disabled
PortFast BPDU Guard Default is disabled
Portfast BPDU Filter Default is disabled
Loopguard Default is disabled
EtherChannel misconfig guard is enabled
UplinkFast is disabled
BackboneFast is disabled
Configured Pathcost method used is short
Name Blocking Listening Learning Forwarding STP Active
---------------------- -------- --------- -------- ---------- ----------
70 vlans 1 0 0 2170 2171
05-12-2011 02:11 AM
I've checked all the switches in the network, they are all blocking the port to 4500_2 and the port to 4500_1 is unblocked.
So spanning tree seems to be OK in the network...
05-12-2011 02:17 AM
Hello Nils,
I do not believe 40Mb is normal and given that the speeds do not reach linerate, I don't expect this to be a loop.
Would it be possible to perform a short sniffer capture to determine which traffic is being flooded?
Cheers,
Bert
06-08-2011 01:50 AM
Thanks for your answares
I've used a sniffer to check the traffic.
It seems to be ordinary traffic, a lot of traffic from different ip-addresses.
It's strange that the traffic flows on the blocked STP port...?!
06-08-2011 01:52 AM
Hi
If I look "at the other side" the counters match.
So there is traffic at both sides.
05-12-2011 02:31 AM
GigabitEthernet4/1 is up, line protocol is up (connected)
5 minute input rate 1000 bits/sec, 1 packets/sec
5 minute output rate 43975000 bits/sec, 15586 packets/sec
Just an idea beside the ones already said by other members.
Let's take the port Gi4/1. I assume that in the other end of this port, there is switch. On that "remote" switch, the counters on the corresponding interface are showing appropiate values with the above ones? (e.g. a lot of input and very less output packets).
If the values are aprox. the same, then you know traffic is really flowing there. Otherwise, I don't know, check for some bugs on your IOS version that might cause this behavior.
Cheers,
Calin
05-12-2011 05:11 AM
Have you checked Netflow for a heavy talker
sh ip cache flow
If someone is pushing 43975000 bits/sec they should stick out like a sore thumb. Find that ip, and if it's an ip that shouldn't be pushing that sort of traffic, then find it in arp and track it down by mac address so you can find out what is going on.
06-08-2011 01:57 AM
I noticed that the traffic is addressed to machines that doesn't exist behind that switch.
So the traffic should not go to that switch at all...
06-08-2011 08:14 AM
Hello Nils,
>> I noticed that the traffic is addressed to machines that doesn't exist behind that switch.
the question is : the backup distribution switch that is sending traffic out those ports knows the destination MAC addresses of the forwarded traffic or not?
if it does not, then this is flooding: unknown unicast are sent out of all ports except the one the frame is received.
so you should check with
show mac address-table address
or
show mac-address-table address
on the
backup distribution switch
the port is blocked on the access layer switch side not on the backup distribution switch side.
This is also a point to consider.
Hope to help
Giuseppe
06-08-2011 09:35 AM
Nils
I think Giuseppe is right. STP only blocks one end of the link and it will usually block the access-layer end. So the 4500 end is still forwarding. However because no mac-addresses will be learnt from the access-layer switch on that port under normal conditions you would see minimal traffic.
But if you have a lot of traffic being forwarded from the 4500 switches to the access-layer switches with unknown mac-adddresses they will simply be broadcast out of all ports on the 4500 and i think this is what you are seeing.
Jon
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide