Single Address In A Subnet Stopped Routing

[usmc_9398]

We have a very strange issue happening where a single address in a subnet will disappear from the network. It is happening at random times to different devices, some in the middle of the night when no one is making any changes. One example is a domain controller that is now no longer reachable from outside if its network, yet all other devices in that same network remain available. I have checked the arp table on the layer 3 switch and the correct mac address is listed for the correct port. The DC can talk to all devices within its network, but anytime it needs to go across layer 3 (to talk to Solarwinds for example), the traffic is lost. Trace routes show that the packets are leaving the layer 3 switch and returning from Solarwinds, but never arrive back at the DC. This is a production environment so I cant share much information, but I am hoping someone might have seen a similar situation and found a resolution. I have work for weeks with Cisco support and they dont have a solution yet, and it looks like the next step is to bring someone onsite to investigate. Thanks in advance for any help. 

[Giuseppe Larosa]

Hello ,

you should at least provide some info like

switch model , IOS version and if this a modular switch a show module .

You should check the CEF table for the affected host.

The ARP table can be correct pointing to the correct MAC address.

The CEF entry holds the packet rewrite information if that entry is somewhat corrupted the forwarding to the host fails.

show ip cef

or

show cef  <host-address> detail

verify the info of packet rewrite (note it will be shown in hexadecimal )

Hope to help

Giuseppe