Unable to ping until arp cache cleared

[samirshaikh52]

Hello Experts

I have 3 servers connected to Cisco Catalyst Switch  C2960 and this switch has uplink to one of the access switch and  ultimately this access switch connected to our 2 Core Switch

We are running HSRP and core switches has direct link between them.

Today  I encountered an issue these server are unreachable from vlans other  that its own.  I just cleared arp-cache and it started pinging.

Please can you help since this happening repeadetly.

Thanks

[samirshaikh52]

do you need arp-a or arp-n result ??

[samirshaikh52]

Please let me know your question that will help us to identify the problem I will forward to the server team.

[JohnTylerPearce]

I just took a look at this document, and I'm a little confused on a few things. I'm sure you have said this before, but I just want to verify.

Which IP Addresses are assoicated with Server 1 and which IP Addresses are associated with Server 2?

I understand the VIP of the load balancing between the servers is 10.1.1.17 (I'm assuming this is correct?

From the word document, I also see you are running HSRP and the VIP fo VLAN 2 is 10.1.1.1.

All of the bold IP addresses you have highlighed, go to the same MAC address.

10.1.1.15 > 0014.5ebc.7466

10.1.1.17 > 0014.5ebc.7466

10.1.1.23 > 0014.5ebc.7466

10.1.1.21 > 0014.5ebc.7466

[samirshaikh52]

Server 1

10.1.1.14

10.1.1.18

Server 2

10.1.1.15 > 0014.5ebc.7466

10.1.1.17 > 0014.5ebc.7466

10.1.1.23 > 0014.5ebc.7466

10.1.1.21 > 0014.5ebc.7466

VIP - 10.1.1.17

Yes HSRP VIP 10.1.1.1

Please let know if you need furhter information.

[JohnTylerPearce]

So I'm assuming Server 1 and Server 2 are combined to create one cluster?

And you're having trouble pinging devics out of your local vlan and in your local vlan correct?

[samirshaikh52]

Yes Sir.

I'm having trouble pinging server 2 IP addressed out of local vlan and but I can ping within same vlan.

10.1.1.15 > 0014.5ebc.7466

10.1.1.17 > 0014.5ebc.7466

10.1.1.23 > 0014.5ebc.7466

10.1.1.21 > 0014.5ebc.7466

[JohnTylerPearce]

Ok I gotcha.

So outside of the local vlan, if you have a computer, it will check it's local ARP cache first, for it's default gateway, if it's not in the local ARP cache (Which I'm sure it most likely is), it will ARP out for the default gateway. Once this packet gets to the default gateway (If that's on your core), I'm assuming it's going to have a destination IP address of 10.1.1.17, is this correct, since that's the VIP?

[samirshaikh52]

Yes the client looks for a destination 10.1.1.17.

[samirshaikh52]

Sorry but I couldn't understand your reply

Thanks

[JohnTylerPearce]

So with the load balancing of Server 1 and Server 2 is this a Active/Standby cluster or Active/Active cluster?

[samirshaikh52]

Its Active/Active

[JohnTylerPearce]

Ok, so has the ever worked without any problems before?

So, if that's the case, let's say you have User-A, who wants to access 10.1.1.17 which is the VIP of this server cluster. This User-A is on a different subnet, he will eventually get to his default gateway, which is probably on the Core. From your word document, during the problem, and then when it's working, the following IP addresses in the ARP table have the same mac address associated with it.

Server1

10.1.1.14 > 0014.5ebc.0c84

10.1.1.18 > 0014.5ebc.0c84

Server 2

10.1.1.15 > 0014.5ebc.7466

10.1.1.17 > 0014.5ebc.7466

10.1.1.23 > 0014.5ebc.7466

10.1.1.21 > 0014.5ebc.7466

If users on different subnets, are getting to the server-cluster via 10.1.1.17, then it should get the MAC Address of 0014.5ebc.7466, which goes to Server 2. So I'm really curious how people are succesfully getting to Server 1 via 10.1.1.17, if the ARP table has a MAC Addresses for 10.1.1.17 associated with Server 2.

Between the Core Switch and these two different servers, are there other switches in between?

[samirshaikh52]

Yes it was working before without any issues.

This is the setup we have.

(3 Servers)<--->(2960 SW)<--->(Access Switch)

                                          |               | 

                                          |               |

                                     Core 1         Core2

[JohnTylerPearce]

I'm a little curious as to how the server is doing it's clustering. Because users are going to 10.1.1.17, which on the core has an ARP entry of 10.1.1.17 > 0014.5ebc.7466, which is on Server 2. So how exactly is Server 1 working? I don't know how the clustering is working, so that's a mystery to me right now. If it's doing Active/Active, I'm not sure how that's working if the MAC is going to Server 2 in the ARP entry.

Do you guys have any static ARP and or MAC entries anywhere?

The fact that on Server 2 you ping 10.1.1.17, and Users can now get to server 2 based upon 10.1.1.17 is kinda odd.

Sounds like it's updating the MAC table and or ARP table or both somewhere.........

I have another question.... How does the server team know, that they can't get to Server 2 when users are using 10.1.1.17? because users are just using the VIP?

[samirshaikh52]

hi,

Actually when the problem occurs we started receive calls from the user that they are unable to access the application.

Then I clear the cache on Core and generate ping packet out of this vlan and started working. meaning started pinging to 10.1.1.17, 10.1.1.21, 10.1.1.15, 10.1.1.23