Heavy Traffic on all Ports in Specific Vlan

[ahmad82pkn]

Hi, i am currently facing strange issue, but unable to find the cause.

i observed 55Mbps traffic out on all port that are part of comman vlan 10.

i tried to find out if any VLAN 10 machine is transmitting same 50Mbps  traffic , so that might be source of that traffic but didnt find any  such port.

i installed sniffer on one of the machine to see what 50Mbps traffic switch is sending to that machine, and i could see irrelevent communication packets in wireshark logs, means what ever communication is happening in my network, i can see that traffic on my sniffer machine.

i am unable to understnd why its happening. why would switch send complete network communication info out to all PC in my particular VLAN.

i have verified there are to SPAN(Monitor session configured) but they are targetted to different ports, so its not SPAN issue as well.

any suggestion? how to narrow down the issue?

[Reza Sharifi]

Hi,

So, all interfaces on the switch are receiving 50Mbps of traffic at all the time?

[ahmad82pkn]

All Switch port are transmitting 50Mbps traffic approx to Host machines in a particular vlan ( in my case vlan 10 ) unable to find from where this traffic is coming in.

[sgouldbo]

Hello,

This sounds to me like Unicast flooding described here:

Cause 2: Spanning-Tree Protocol Topology Changes

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801d0808.shtml#cause2

Could check if any port without portfast is flapping?

What´s the output of the following command:

show span det | i is exec|from|occurr

[ahmad82pkn]

No port flapping occuring in network. ( right now since my production hours are almost finish , so all ports are not transmitting approx 20Mbps , and sniffer shows unicast communication between different hosts, not sure why on earth Cisco switch is sending unicast communication between different hosts informaiton out to all ports of my particular vlan)           

i will read your article, here is log from my core switch. its also transmitting same data to host directly connected to it in that particular vlan.

6509CORE#show span det | i is exec|from|occurr

VLAN0001 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 27 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0088 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 9 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0800 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 61 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0810 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 25 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0812 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 9 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0814 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 21 last change occurred 2w6d ago

          from FastEthernet2/2

VLAN0815 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 8 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0816 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 8 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0817 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 14 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0819 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 8 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0820 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 8 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0821 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 8 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0822 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 8 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0823 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 8 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0825 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 8 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0830 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 8 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0834 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 26 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0840 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 82 last change occurred 1w6d ago

          from GigabitEthernet1/40

VLAN0850 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 7 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0851 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 5 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0854 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 5 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN0900 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 9 last change occurred 5w2d ago

          from TenGigabitEthernet7/1

VLAN1000 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 374 last change occurred 1d05h ago

          from Port-channel6

[sgouldbo]

Ahmad,

Are you using HSRP?

Can you take one of the destination MAC addresses that you are seeing on the ports that shouldn´t be seeing there.

Take that MAC address and see if it shows in the MAC address table of  your CORE device or devices?

[ahmad82pkn]

not using HSRP. i can find few MAC and Few not, on this outcome , now i am able to narrow down the issue, it looks to be all traffic is for my email servers which include exchange DB servers, CAS servers, and Webmail servers. i guess they are running in NLB enviroment, i thought issue might be there Virtual IP or MAC binding not present in Core switch, but thats not the case i can see their virtual IP and MAC hard binded in my core switch. so traffic shouldnt be broadcasted right??

[Leo Laohoo]

On you core switch, please post the following commands:

1.  sh int | i protocol | txload

2.  sh controller util

3.  sh version (I hope your access switches are running 15.0 IOS.)

[ahmad82pkn]

1-TX/RX load is normal ( not posting output, since its too long, i have 7 blades )

2- command not available.

6509CORE#sh controller util?

% Unrecognized command

3- nops here is core s72033-ipservicesk9_wan-mz.122-18.SXF17.bin

Can you also review my second last response, above this one? looks like NLB mis configured issue?

[ahmad82pkn]

Looks like i have found the problem. i can see mac address for mail server communication starting with 02bf, that confirm mail admins configured it as unicast NLB, and as per Cisco document, it will cause Flooding in that vlan since MAC will never get learned on switch port

http://www.cisco.com/en/US/products/hw/switches/ps708/products_configuration_example09186a0080a07203.shtml

so now checking with Server admins to plan/evaluate to put servers behind a dedicated switch or hub.