I am experiencing strange behavior in my network. It occurs in Layer2 part of the network (it consists of Cisco 7600, 6500, 3560, 2960 and some wireless antennas). The MPLS and L3 part of network are fine. It seems that some packets are replicated in network. I have occasional storms of unicast flooding. Some unicast packets are multiplicated and due to CAM table flashing they are flooded to all ports.
What could cause multiplicating packets in layer 2 netowrk (1 sent unicast packet replicated 10000 times). Has anyone had problem like that? If there is layer 2 loop in my network than I would have broadcats storm, which is not the case.
Whit sniffer i.e. I can see 170000 DNS replys to same IP address (the same vallid src and dst MAC and IP address so it shouldn't be DOS attack). Also from DNS server to network are sent only one DNS reply.
Also I send 1 ping and then receive back 1000 replys!. I can see on traffic graph that some wireless antennas are sending back traffic that they received.
any help would ne appreciated
Thanks in advance.
This seems like a rather odd issue. I am familiar with unicast flooding in a dual-aggregation topology where MAC tables are flushed before ARP tables, but have not heard of anything like you described (with actual duplicate data packets).
The only thing I can suggest is that you run a number of captures in parrallel. If you identify the traffic that is being flooded or duplicated you should be able to capture at both endpoints and verify that no source or destination is causing the issue. If your sender is not generating the duplicates then you can pull the monitoring location out further into the path. At some point in time you should begin seeing the duplicate traffic and then scruitinize the specific portion of the network where the problem is occuring.
Even with things like etherchannel or ECMP you are not generating duplicates but distributing individual frames.
As Jeremy suggested, putting sniffer captures in several network point is best.
Also, you can try to ping each hop by hop to see if PING pkts are duplicated.
For example, PC----DEF GW ROUTER-----R1----R2---PC.
I would ping PC to DEF GW first and see if PKT are duplicated or not.
Then, next hop.
So, you will see who causes the duplicated packet or at least getting close to the root cause.
Take a look at the documents below if you haven't already
Do you have any Etherswitch modules in your routers? I saw "similar" weird behavior when I had etherswitch module on different routers both attached at layer 2 to the same 3560 L2 switch. I ended up just having to break up my broadcast domains and move the 3560 to routed ports on the routers. We chalked it up to eitherswitch modules not playing well with stand alone switches.
yes I have some etherswitch modules in some 1812, 1841 routers, but ususally they are not facing the core network,
I'll have to check it out.
A very quick way to narrow down the problem on the 6500 and 7600 is to use TopN reports. ie
Type the following to start the TopN reports
# collect top 10 counters interface all sort-by utilization
After a short time view the reults using
# show top counters interface report 1
This will tell you which ports the repliacted/looped traffic is coming from/to and help you narrow down the issue.
And then when you're finished
# clear top counters interface report