How to identify broadcast traffic source

[drstokes76]

We have 2 switches split across 2 datacentres connected via an interconnect.  Over the past couple of days the interconnect provider's Cisco kit has shut down our port (err-disabled) due to a broadcast storm.  They had the level set at 1 which I thought was a bit low.  They say they tried to set to 2, then 5 but still kept tripping the storm-control feature so they set at 10.  They say they've always had it set at 1% (on a 100Mb switch) and so we must be generating more broadcast traffic.

I'm trying to identify where the broadcast traffic is coming from.  On our Cisco 3750 I've clear interface counters and when I do a sh run | i broadcasts there are a few ports which have what seems like a high broadcast count.  The one port that is especially high and the only one tripping the storm-control feature (I've enabled on all our ports to try to identify where the traffic is coming from) is the port connected to the 100Mb interconnect.  I've mirrored that port to another port and connected a server with wireshark so I can capture all the traffic across that port. 

What I'm struggling to find is the source of the broadcast traffic.

I have a few questions are these broadcasts layer 3 or layer 2 broadcasts.  Also in the output below when it says broadcasts received is this inbound to the port i.e. from the connected device or is this a total of inbound and outbound broadcasts.

When I use wireshark and filter the capture on broadcasts (ff:ff:ff:ff:ff:ff) I see only 200-300 compared to the thousands the switch is reporting.

If I filter on the broadcast IP address I also don't see the numbers corresponding to what I see in the show interface output.

GigabitEthernet1/0/1 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 0014.a93f.7401 (bia 0014.a93f.7401)

  Description: Interconnect

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 4/255, rxload 44/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 17w0d, output 00:00:00, output hang never

  Last clearing of "show interface" counters 00:12:16

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 17623000 bits/sec, 2054 packets/sec

  5 minute output rate 1894000 bits/sec, 1419 packets/sec

     1818136 packets input, 1818336989 bytes, 0 no buffer

     Received 104971 broadcasts (0 multicast)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 97965 multicast, 0 pause input

     0 input packets with dribble condition detected

     1269637 packets output, 241905137 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

Oh also I'm currently doing : monitor session 1 source int g1/0/1 both, and also tried just rx incase I just need to be looking at receive traffic but still nothing is standing out.

Any help would be appreciated.

[John Blakley]

It's difficult to see what the source is with wireshark. You can try to do a capture based on destination ip of 255.255.255.255 (ip.dst==255.255.255.255), but you may not be able to find what's doing it still. Do you have storm control configured across your ports? I'd start there so you can control what's getting sent to the provider and they don't shut you down.

John

[drstokes76]

Hi John

I've enabled storm-control on all our ports and am trapping the event.  The only port that activates the feature is the port connected to the interconnect, so this says to me that traffic is coming across the interconnect to this port.  I've also enabled storm control on the other side of the interface but not yet on all ports because I wanted to be sure that the ports wouldn't get shutdown if they exceeded the level.

But with all that said I thought I should be able to capture the traffic with wireshark.

Darren

[fb_webuser]

You can caprure traffic with wireshark and then look statistic - conversations, it's the siplest method

---

Posted by WebUser Aleksandr Yanovskiy

[drstokes76]

Hi Aleksandr, I've been doing that and whilst I can see a lot of packets they all look like unicast packets I cant seem to find hundreds of broadcasts.

Darren

[John Blakley]

You should be able to see with your original capture of 0xffffffff. Has anything been added or changed on either side of your 2 sites? Added switches for redundancy or anything like that?

[drstokes76]

We recently replaced an ASA in each site running v7 with a new one, more RAM and with v8 (we don't have Cisco maintenance so not entitled to upgrade).

They've been configured with pretty much the same config.  The only really difference is that the outside and internal interfaces on the new firewalls have sub-interfaces as we plan to add extra VLANs to them in the near future.  Previously they were in a single vlan each and all servers on the same VLAN. 

We've added an extra virtual server which isn't doing anything currently and thought that maybe we were just under the 1% threshold before and now we're just over but the fact that they had to raise it by a factor of 10 would wipe that theory out.

[drstokes76]

Oh also whats that capture filter 0xffffffff?  How would I enter that?

thanks

Darren

[John Blakley]

That's your eth.dst==ff:ff:ff:ff:ff:ff filter in wireshark that you said you had used. Did you try Aleksandr's method of finding your broadcast with statistics/conversations? You should be able to see any abnormalities in that as well.

[drstokes76]

Hi John, yes that was a display filter.

Even if I capture everything I can't seem to find any sort of broadcasts in the numbers that the switch is reporting.

[drstokes76]

Still not having much joy with this. At the moment I'm not seeing large numbers of layer 2 or layer 3 broadcasts so not sure where the switch is getting this information.  Is there any more information I can get from the switch using debugging?

Thanks

Darren