End device can ping its default gateway but no other interfaces

[Jerry_Lee]

Good Morning,

     I'm currently experiencing an issue with one of my Cisco 3825s with a NME-XD-24ES-1S-P mounted on it. The current set up consists of this stack, with the router and switch being connected by a /30 link (fast ethernet 1/0/1 on switch), with the router housing the connections to our modem and encryption devices. The switch handles our end devices as well as a RSAM that handles our DSN traffic. The issue is that our RSAM, which is currently configured with an IP address of 214.43.28.230 /30. It is directly connected to VLAN 5 which is configured with .229. The RSAM can ping its gateway, however when it tries to hit the first part of the link to the router, 214.43.28.34 /30, it times out. Our data traffic is received on VLAN 2, then proceeds through the same .34 link to the router.

     This is where it gets interesting for my colleagues and myself. IP routing is obviously enabled. The switch's default gateway, which isn't necessary anyway, is set to 214.43.28.33. The switch is configured with the route 0.0.0.0 0.0.0.0 214.43.28.33. The port is up up EIGRP is broadcasting correctly as our router can ping 214.43.28.229 and shows the .228 network in its routing table. There are no ACLs forbidding this traffic. When I try and ping the router link IP address with debugging on, no messages are generated.

     This issue is driving me crazy! I'm sure I'm missing a slight detail, any help in this would be greatly appreciated! Thank you!

[Richard Burts]

Can you verify what is configured on RSAM as its default gateway? And can you check and verify whether any routes have been configured on RSAM?

HTH

Rick

[Jerry_Lee]

Good afternoon Rick, thank you for your response!

The gateway on the RSAM is the vlan interface of .229. The route configured on the RSAM is a very basic one of 0.0.0.0 0.0.0.0 214.43.28.229.

I've tried putting the device on a different vlan, as well as removing the vlan completely and just configuring the port it was on with a /30, with the same address. As stated earlier, even with these alternate configurations, the end result is still the same. A ping on the switch with the source address of 214.43.28.229 can reach any destination, specifically the 214.43.28.34 interface that is on the same device. A ping going vice versa, with the source address being .34, can reach the address of .229. All this makes perfect sense. However any ping with the destination of .230 times out.

[Richard Burts]

Thanks for the additional information. I would like to verify that my understanding of some things is correct.

- the RSAM is able to ping the 230 but is not able to ping anything else.

- you were running debug icmp on the router, attempted ping from RSAM and got no debug output.

- did you also ping the router from the switch and did get debug output?

- any ping to the RSAM at 230, including from the 229 address fails.

And I have a couple of suggestions.

- check the arp table entry for 230. Can you verify that it is the correct mac address for RSAM?

- perhaps posting the config might help us find the issue.

HTH

Rick

[Jerry_Lee]

-The RSAM is .230 with .229 as its gateway. The gateway was the only address that it was able to reach. While the gateway could reach anything, as well as be reached by external devices.

-While on the switch, pinging any address with the source address of 214.43.28.34 (link to the router) with icmp debugging enabled regardless of whether the ping timed out or not, displayed output. However pings to .230 (RSAM) would not provide any output upon failure.

     When we initially stumbled upon this issue, checking our arp table was one of the first things we did, along with EIGRP, routing tables, ACLs, etc. Everything looked just fine. This was even kinda validated about an hour or so ago, when we turned the device back on (we had shut it off for about an hour), and somehow it worked just fine. This didn't make sense, because the previous day we had it shut off for about 10 hours.

     So in the end, the result seems to be 48 hours of troubleshooting this issue, with about a 10 hour period of the device being inactive in between days. The device was shut off for about a hour more recently, then re-enabled without any configuration or hardware changes from what was already in place. It resumed normal operation.

[Richard Burts]

Thanks for the update. This resolution of the problem is pretty weird. But while I always want to find a rational explanation for issues, I have learned to accept that sometimes things that were broken just start to work for weird reasons and I have learned to be grateful for these.

HTH

Rick