Bizarre IP

[Ajai George]

Hi All,

I have a very strange issue with one remote subnet IP address.

I have a Datacenter Backbone which comprises of 2 Core Switches and 10 Rack Switches(2960Gs). We have 2 NTP servers, one at our Main DC and the other at the DR Site.

Recently, I’ve been seeing hits on our internal firewall( default gateway) for traffic destined to the DR NTP Server.

Did some digging and noticed that I am unable to ping the DR NTP server from a few rack switches. I am able to ping other IPs in that DR NTP Server subnet from the Rack Switches. I am also able to ping it from all other subnets.

Now, it would seem the issue is with the NTP server which is running RHEL. But, I've noticed that I can ping the DR NTP server from 2 Rack switches in the same backbone vlan. There is no difference in the switch configs other than the interface vlan IP. This would rule out any routing issue with this subnet. I have also checked the NTP server and the routing table on this server looks fine.

If I do a traceroute it just times out. Just don’t know what could be the issue. Routing looks fine, switch configurations is fine and the server routing table is also good.

Let me know if anyone has faced a similar issue and how I can get to the root cause.

Many Thanks,

Ajai

[garapoglou]

Hi,

Since everything else seems to be fine, I would have a look at the firewall. It could be caused by an access rule which blocks access to specific IP addresses, like the ones of the switches which don't get ping replies.

Best regards,

Giorgos

[Ajai George]

Hi Giorgos,

The Firewall is the default gateway for the core switches.

The setup is like this.

Rack switch --> Core Switch-->Core Router<--WAN LINK--> DRS L3 Switch-->DR NTP Server.

                              |

                         Firewall

                              |

                         Internet

The Rack switch has the Core switch as default gateway.and the core switch has an entry for the remote subnet in the routing table.

Hope this clarifies.

Thanks,

Ajai

[garapoglou]

OK, Ajai I've figured it out. Thanks for the information.

Where does traceroute time out?

Giorgos

[Ajai George]

The ping/ traceroute to DR NTP is hitting the internal firewall instead for going to the core router.

Ping and traceroute to any other IP in the DR NTP Server subnet goes to the core switch and then to core router.

[garapoglou]

Ajai,

What about the ARP table of the rack switch? Did you check it as well?

Giorgos

[Ajai George]

Hi Giorgos,

The rack switch is got the arp entries for the core switch and core router.

Regards,
Ajai

[paolo bevilacqua]

Check all relevant ARP entries, or try clear ARP.

Also try clear mac-address-table.

[Ajai George]

Hi Paolo,

The Rack switch is got the right arp entries for the core switch and core router.

Regards,

Ajai

[Dennis Olvany · Accepted solution]

Show ip redirect

[Ajai George]

Hi Dennis,

Noticed that the Host IP is using the firewall as the gateway from the show ip redirects output. Cleared ip redirects and i am now able to ping the NTP server.

Thanks a lot for the help.

Regards,

Ajai

[paolo bevilacqua]

I had a same case recenty too. 5 points awarded!

[Ajai George]

But still not sure what could have caused the host to be redirected to the wrong gateway.

I do have the Core switch configured as the default gateway.

[Dennis Olvany]

If it is a dynamic route to the destination on the default gateway and that route goes away, then the default gateway may send icmp redirects for whatever route now matches the destination. Some switches fail to properly timeout redirects.