I know what you're thinking... IP conflict, right? Not exactly....
So we have this CentOS 6.5 linux server plugged into a 2960G access switch trunked to a 3750 as the core LAN router in our DC. The linux server was working until just recently. The problem is that it can't ping it's local gateway (the core switch) or anything beyond it, however it can ping other hosts on the same subnet. Tried re-kicking machine with same IP stack information, tried clearing arp on the upstream switches, flapping ports on the upstream ports. Nothing changed the fact that this host just seemed to lose it's routing functionality.
Now, as a hunch, I changed the hosts IP address from 10.1.1.126 to 10.1.1.196 and VOILA! The host suddenly started pinging the GW and past the gateway now. I still can't ping the old IP (.126) or see any traces of it in any of the switches. No IP or MAC remnants exist on any of the other switches.
I then tried to change the IP back to .126 and it stopped working again. Changed back to .196 and started working again. These are /21 subnets (don't ask - this was setup before my time.) The first two /24 in each are pretty used, the rest is empty IPs.
What gives? Has anyone seen anything like this before? My gut tells me it's something with the server NIC, but I'm not positive.
We may try sniffing off the access switch and see what the ARP looks like. We do see the core and the access switches with the correct IP and MAC information.
