ICMP Issues to Certain Addresses

[brandonbittinger]

Hello, I have a very strange issue I was looking for some direction on because at this point I have about exhausted my ideas. We have a subnet at another building connected through metro Ethernet. The metro Ethernet connects into Nexus switches we have at the site. We have several subnets at this site but one in particular (as far as I can tell so far) seems to be having some very strange issues. The subnet is question has 3 devices that are giving me the issue a 1941 router, a 2500 wireless LAN controller, and an APC battery backup. The issue is that we have a few select servers that are unable to ping specific addresses in these remote subnets. We have a server that runs PRTG for monitoring and this is one of the servers that is unable to ping across to these devices, this is how we discovered the problem. 

The PRTG server fails to ping all three of these devices and if you run a traceroute it stops at the NEXUS switches at the remote location. If you ping any of these devices from the NEXUS at that building you do get a reply but if you traceroute to it on the that same switch it times out. So far in my troubleshooting steps I have tried rebooting several devices in between and the affected devices themselves. I have also went as far as changing the IP address of the PRTG server. After I change the address I got a strange result, I was now able to ping all of the devices. I then rebooted the 1941 router and after it came back up I was unable to ping the device again... I have not tried to reboot either of the other two devices as of now but I am assuming I will get the same result. The even stranger part is the issue is just ICMP traffic I can get to the web interface of the APC and WLC or SSH into the 1941 from the PRTG server without issue.

I have checked on all the devices in between there are no firewalls, and none of the devices have any ACL that would block ICMP or an ACL that would block anything for that matter. Also Windows firewall if not enabled on the PRTG server and everything was working fine. As far as I knwo no changes have been made either. If anybody has any ideas they feel may be helpful I am all ears. Let me know if you need any clarification. Thanks!

[Julio E. Moisa]

Hi

Have you verified if there are duplicated addresses? do you have any dhcp scope for the servers?

[brandonbittinger]

I had thought about he duplicate address at first as well. I did verify that there was not a duplicate address. I later also thought that if the issue was a duplicate address then how would I be able to consistently use the web interface of them or SSH to the router? So at this point I do not believe that is the issue. Thank you for the suggestion though! 

[Raja_D]

Hi,

Execute the route PRINT from your PRTG server and check the routing table entry for the mentioned subnet to identify if the gateway is set properly or not from server side.

[brandonbittinger]

Everything looks to be fine there as well. One persistent route tot he default gateway and a few other routes to the local link. Nothing out of the ordinary or referencing any networks outside its own.  

[johnd2310]

Hi,

Have you gone through the configuration of the NEXUS switch, especially the svi of the trouble subnet. Do the ip addresses and subnet masks match? Can the WLC and the 1941 ping/traceroute to the PRTG Server?

Thanks

John

[brandonbittinger]

I have gone through the config on the NEXUS and as far as I can tell everything is fine. I did verify that all of the subnet masks are correct and yes I can ping/traceroute from the affected devices to the PRTG server without issue. I should have had that in the original post. My apologies! 

[communities@paessler.com]

Do you happen to have a Firepower SFR module between the PRTG server and the subnet that has issues?  

If so, then it *might* be the same problem as in this discussion: https://supportforums.cisco.com/discussion/13294256/how-exclude-network-monitoring-system-ip-sfr-configurationrules, where the SFR seems to be dropping ICMP packets.

[brandonbittinger]

We do have Firepower but it is not between these subnets. Thank you for the thought though. 

The issue seems to have worked itself out somehow.... Not sure what the root of the issue was or if it will pop back up again.