Can something on the network block ARP replies?

[InquiringTech]

We've noticed there is a problem with reaching hosts by ping sometimes on our network (mostly some Linux machines), and the issue is not always consistent. Sometimes if someone pings the gateway, they can then ping the host they need, but not before then. Sometimes it seems to work one way. There wasn't a setting or firewall on the local machines blocking icmp traffic. I did a Wireshark capture when trying to ping a certain IP, and I can see the ARP broadcast packet go out, but there is no reply.

My question is, what should I look for on the Cisco firewall and switch configuration to see if ARP is being affected by something? I checked the access-lists and didn't see much at all. On the Catalyst 9200 switch, under show run I see

ip arp entry learn 10240

and on the firewall I see

arp timeout 14400
no arp permit-nonconnected
arp rate-limit 16384

Not sure what that 'no arp permit-nonconnected' is all about or if that is relevant to issues on the internal network (heard it may have something to do with multiple ISP networks being configured on the gateway). Our equipment was configured by our HQ but we're supposed to manage it on our own now.

Also not sure if something would be configured on one of the ports on the switch that's affecting things.

[balaji.bandi]

no arp permit-nonconnected - There is good explanation here :

https://community.cisco.com/t5/network-security/arp-permit-nonconnected/td-p/2226198

Regarding your issue - we need to look the config and some small network diagram to understand.

we can only guess what could be the problem with many reason, but that may be not true, so provide the requested information so we can assists better.

by the way all same IP address Block ? same VLAN ?  ping loss from PC to PC ? PC to internet ?

[InquiringTech]

So we have a stack of three C-9200 switches connected to both a FirePower 1140 and an ISR 4331, both of which are then connected to the ISP router on different ports (the router goes to a DMVPN across to our corporate network). We have several VLANS, with subnets corresponding to each. So like VLAN 33 is data and 10.10.33.0 /24 network, voice is 34, etc. These are only on logical interfaces, not physical. There doesn't seem to be any trunking though. With the exception of the separated guest VLAN subnet, the rest can freely talk to each other (can ping VoIP phones on its respective subnet and also devices on the AnyConnect VPN subnet). I also noticed for some reason some of the switchports are also on VLAN 1, particularly the ones that are also on the VoIP VLAN (34), but I'm not sure why exactly.

As far as ping loss from PC to PC...

By default most of our Windows computers have incoming ICMP traffic blocked by their local firewalls (this is set up by corporate when they send the laptops out). I enabled receiving pings on some of our computers because they needed that, although this is kind of unrelated to the issue here since we know the cause of that.

But since this is a lab we also have a lot of atypical Linux based machines and other devices that complicate things. Some of those are the ones that are having problems pinging and communicating. Like we even reinstalled a clean Ubuntu on one with the problems, and somehow it not only caused ping to start working but also caused the other similar machine (which was not reinstalled) to begin accepting pings successfully. So that's what's confusing me. I also checked to see if our Wifi access points were doing something, but that wasn't the case.

Also, there isn't an issue pinging from hosts to the internet. However, you can't ping an external internet address from the switch itself if you are on there with Putty or something.

[Georg Pauwen]

Hello,

with the issue being intermittent, you can be pretty sure that it is not an access rule causing this. What (Cisco) firewall do you have ? I don't know how busy the firewall is, but try and increase the 'arp rate-limit' value to the maximum (32768)...

[InquiringTech]

Thanks. What are the possible repercussions of doing that though? Now even some of our Windows machines are having problems pinging each other, even with Windows firewalls down, same WiFi network and subnet. They can both ping certain hosts like servers, especially those that are hardwired. Pings between two hard-wired clients always seem to work, while ones on the Wifi tend to be more iffy. We have non-Cisco, Ruckus wireless APs for now, but there doesn't seem to be anything security-wise on there that would be blocking/disrupting this functionality, at least that I can see.

The ARP entries seem to be in the proper table in the switch, but direct contact between some hosts isn't happening for some reason. This is also not always consistent. I notice sometimes when a computer with the problem goes to sleep or is rebooted, the functionality starts or stops working. Just when I think I see a pattern, there isn't one anymore.

I feel like something is still screwy with the ARP table. I see a lot of the clients have an age of 0 minutes all the time. Is this normal, meaning they keep renewing? How would I check the relevant timers and if there isn't some kind of conflict.

[Georg Pauwen]

Hello,

--> Now even some of our Windows machines are having problems pinging each other, even with Windows firewalls down

If this happens even with the firewall down, the issue is elsewhere. What is the uptime of the switch ? Can you post the output of 'sh ver' from the switch ?

[InquiringTech]

Switch uptime is 6 weeks, so that seems fine at least

[MHM Cisco World]

You ping and you find arp broadcast in wireshark? 

Do you config GW in host ? If not then you must enable arp proxy in both fw and router ( which is bad idea) or you must config gw in host.

[olivertwist0404]

Some Linux machines on our network have trouble reaching hosts via ping, and the problem isn't always consistent. Pinging the gateway can sometimes result in pinging the host, but not before. It sometimes only works one way. There wasn't a setting or firewall on the local machines blocking icmp traffic. I did a Wireshark capture when trying to ping a certain IP, and I can see the ARP broadcast packet go out, but there is no reply.

In order to see if ARP is affected by something on the Cisco firewall and switch configuration, what should I look for? As far as access-lists are concerned, there isn't much to see. Under show run, I see the following on the Catalyst 9200 switch

ip arp entry learn 10240

and on the firewall I see

arp timeout 14400
no arp permit-nonconnected
arp rate-limit 16384

The 'no arp permit-nonconnected' error may have something to do with multiple ISP networks configured on the gateway. I don't know what that means. We are now responsible for managing our equipment, which was set up by our HQ.

Additionally, I'm unsure if something is configured on one of the ports on the switch that is affecting things.

[MHM Cisco World]

from my view the following is happened, 
SW connect to Router and ASA 
the host connect to SW have IP address in subnet different than ASA 
host need to connect to outside it Subnet 
the host send arp (arp ask GW MAC), this arp receive from ASA here the no arp permit-nonconnect is play role, this command prevent the ASA reply to ARP, but why ? because the ARP is send to GW in different subnet than ASA interface receive that ARP, this make ASA drop arp and connection failed. 
so my view is there is something wrong in mask
here there are multi mask location we must check 
1- mask of ASA interface 
2- mask of host and mask of dhcp pool supply host with IP
3- mask of GW 

waiting your reply.

[InquiringTech]

I'm a little confused. Is the ASA supposed to be giving out the ARP replies, or the layer 3 Catalyst switch? The ASA is where NATing is configured to go out onto the ISP router and then the internet. The router on the other hand, is mostly for our DMVPN link back to our HQ site, and our voice gateway.

If this helps though, here you go:

1. mask of ASA - the inside interface connecting to the switch is 2.2.2.1 /24, connected to 2.2.2.3 /24 on the switch. And the router interface connected to switch is 2.2.2.2 /24. The outside interface of the ASA is a public IP address from the ISP.

2. mask of host - 10.10.33.48 /24, on the DHCP supply pool of 10.10.33.0 /24

3. mask of GW - it's just 10.10.33.1 /24 for the data pool on the switch. And for voice 10.10.34.1 /24, etc.

Thanks

[InquiringTech]

What do you mean precisely by configuring the gateway in the host? Like manually? When the hosts get DHCP, they have the gateway listed as the virtual switch interface for the data subnet, in this case 10.10.33.1, as per an ipconfig or ifconfig. Is there more to it than that?

[MHM Cisco World]

OK the host get GW IP (10.10.33.1) from the DHCP,  are the 10.10.33.1 is within range of IP of host subnet ??
you mention 10.10.33.0/24  so 
the host and GW in same range IP of subnet. ? am I right 
now the IP of FW and router interface is in same subnet ??

[InquiringTech]

This is a rough diagram of the network (the IPs are of course changed from our real ones, but just examples). 

But yes the inside IP of the FW and router interfaces are in the same 2.2.2.0 /24 subnet as the switch interface. The actual host pools are just on the switch and each has a VLAN (although routes are also defined on the ISR). There are no VLANs present on the FW/ASA.

I still think that this may have something to do with the wireless APs (non-Cisco) attached to the switch. Since the directly ethernet connected hosts don't seem to really have the problem, from what I can tell.

Would the ports on the switch that the APs are connected to need special configuration?