Packet Capture Shows Only Broadcast Traffic on Firewall Inside Interface

Nathan Farrar · ‎09-26-2016

Working with a Cisco Meraki firewall. These things don't give you much as far as options go for troubleshooting, I really wish it was an ASA!

Here is the issue that I'm hoping to get some clarity on. At some point this morning a small network consisting of about 15 users connected to an unmanaged switch which has a Meraki firewall as the gateway stopped working. The firewall is accessible from the outside but no clients are able to get out of the network. The network has a small Mitel phone system some printers and desktops. Nothing special.

When running a packet capture on the inside interface of the Meraki all that returns is ARP/Broadcast traffic. The rate of traffic isn't high at all, maybe 6 - 10 ARP requests per second at most. Many of them are looking for a single address inside the network, not getting responses. It appears that these clients are related to the phone system in some way. I'm not seeing how all TCP traffic is just gone. I suppose that something else in the network is now blackholing the traffic and sitting in as the default gateway... so I tested this. I changed the IP address of the firewall and reset the DHCP pool. Had a PC rebooted to see if it would get a new address with a different default gateway. It did, and nothing changed. I'm not even seeing DHCP requests.

I'm not physically onsite so I can only ask questions as I go. Asked them to check the topology and there doesn't seem to be any rouge devices in path. I can ping some devices from the firewall. I've confirmed subnets haven't been misaligned somehow as well as any static routes. Like I said, basic basic network.

My assumption is that the phone system is somehow causing an issue. Maybe it is a duplicate IP address and it is getting all the egress traffic instead of the firewall but it's hard to diagnose that remotely... especially with an unmanaged switch in there.

Just fishing for some thoughts. I'm going to have them remove the phone system from the picture to see if that changes anything..

Thanks

pwwiddicombe · ‎09-26-2016

How are you actually tapping into the link to do the capture? Some of these small unmanaged switches that you connect there to tap off a wireshark connection are just that - switches. Remember the function of a switch - send ONLY to the port where the traffic goes, unless it happens to be an unlearned destination or a broadcast. So, if you tap in that way, you only get the broadcast traffic.

If you can find a real HUB you can do it that way; or a true "network tap" (we have a few specifically for this purpose) you can also capture traffic on a link. Otherwise, you can possible SPAN traffic on a cisco switch attached to the firewall. I think Fluke has a network analysis unit that can capture traffic on a link - but it's not cheap ! (although it also does the capture, analysis, and lots of other goodies).

Nathan Farrar · ‎09-27-2016

One of the nifty features of the Meraki equipment is that they have a built in packet capture feature. From the cloud management portal, I can begin a packet capture on the LAN interface and output a wireshark .pcap file. This is really the only way to do it since I am not able to get onsite. So in this case, I should be seeing any data that is destined for the default gateway which is the Meraki in this case. Otherwise you are totally correct, I wouldn't see all data unless it was destined to what ever was in the CAM table for that switchport/broadcast traffic.

So that is why it was weird, the device was telling me that it was only receiving broadcast type traffic when there should have been something unicast.

A follow up to this when I ran another capture this morning and I am now seeing TCP data at the inside interface of the Meraki, most of which is a backup going across a VPN connection to another site.....

So, needless to say, this isn't satisfying! I hate it when things fix themselves and we never know what caused the issue in the first place. Although I think this will happen again, my network spidey senses are tingling :) I'm still leaning toward the phone system but I'll update. Can you think of any other reasons why we would see something like this? The only reason I can think of is that either 1. The firewall is being buggy or 2. Something else has the same IP address and all unicast traffic toward the outside is being sent elsewhere.