Unable to ping until arp cache cleared

[samirshaikh52]

Hello Experts

I have 3 servers connected to Cisco Catalyst Switch  C2960 and this switch has uplink to one of the access switch and  ultimately this access switch connected to our 2 Core Switch

We are running HSRP and core switches has direct link between them.

Today  I encountered an issue these server are unreachable from vlans other  that its own.  I just cleared arp-cache and it started pinging.

Please can you help since this happening repeadetly.

Thanks

[samirshaikh52]

Hello,

I have to check on server team for that.

I make continous to VIP.

Thanks

[Jon Marshall]

When you clear the arp cache and do a continuous ping what do you see in the arp cache ie. do you see the real IPs or just the VIP.

It sounds like what is happening is that when you run a continuous ping from a client to the VIP the server(s) respond to those packets with the VIP address. But when you stop and the server(s) then try to communicate they are using their real IPs instead of the VIP and therefore there is no entry in the core switch arp table or if there is there are also the other entries.

This is just guesswork at the moment, but i suspect the issue is to do with how the HA has been setup.

Jon

[samirshaikh52]

One more thing to be noted again

I was not able to ping the ip addresses 10.1.1.14, 15 and 17

I cleared the cache

then I can ping to 10.1.1.14 but to 15 failed. Then I ping 17 and again came back to 15 it also started pinging.

[Jon Marshall]

What IP do end clients connect to ?  is it the VIP ?

Can you find out exactly how the servers have been setup ie. what is the HA software in use, how are the servers connected to the switch eg. one NIC per server, dual NICs etc.

Are we talking about physical servers or is it VMware ?

Jon

[samirshaikh52]

In the arp-cache I can see all three IP's

[samirshaikh52]

They see VIP.

They are physical servers.

[samirshaikh52]

but why suddenly they are stop communicating with other vlans and unless i clear the cache. Now it stopped again.

[Jon Marshall]

but why suddenly they are stop communicating with other vlans and unless i clear the cache. Now it stopped again.

That's what we are trying to find out

It may be that something was changed on the server(s) or the switch you do not have access to. 

We can't tell but what we can do is look at it from the core switch perspective and to do that we need to understand how the HA sofware works, what mac addresses we should be seeing etc..

Jon

[Jon Marshall]

And in your arp cache on the core switch does it show the same mac address for both the two real IPs and the VIP ?

Are the servers that are working in the same vlan running any HA software ? If not then it definitely looks like this is the issue.

If you could find out what HA software they are running we could look at the docs to see what should be happening and how it should be setup in terms of the switch connections.

Jon

[samirshaikh52]

On core it shows same mac for 1 real and VIP. For other its different

10.1.1.15

10.1.1.17

Other servers are not working with HA

[Jon Marshall]

If the other servers are not running HA and are working then i would say it is very likely the HA setup that is causing the problem.

We just need to understand how it should work.

Jon

[JohnTylerPearce]

On the server, is it possible to install Wireshark, and take a look at the ARP replies, for the VIP when you ping from the server?

[samirshaikh52]

Well my time is +3 GMT I can only contact the person in the morning.

May when you guys might be available

BTW, I appreciate your help Thanks a lot

Thanks

[Jon Marshall]

It's 10:00pm in the UK but there are always people around on these forums to help out.

I think the first thing you need to do is find out what HA software they are running and then we can try and work out what the core switch should be seeing in terms of mac addresses/IP addresses.

Jon

[samirshaikh52]

The problem is guess now with only 2 IP addressing 10.1.1.15 and 17 which shows same arp entry on core switch.

10.1.1.14 is working fine upto now.