Unable to ping until arp cache cleared

[samirshaikh52]

Hello Experts

I have 3 servers connected to Cisco Catalyst Switch  C2960 and this switch has uplink to one of the access switch and  ultimately this access switch connected to our 2 Core Switch

We are running HSRP and core switches has direct link between them.

Today  I encountered an issue these server are unreachable from vlans other  that its own.  I just cleared arp-cache and it started pinging.

Please can you help since this happening repeadetly.

Thanks

[JohnTylerPearce]

To go with Jon Marshall's recent post you can do that by doing the following.

1. Ping the IP Address

2. 'show arp | i IP-Address

3. 'show mac-address table | i IP-Address' *** Command for show mac address table may vary on the switch ***

This will show you the port it was learned from and you can do 'show cdp neighbor' and trace it form switch to switch that way.

[samirshaikh52]

1.show arp | i IP-Address

Core Switch

Internet  10.1.1.17              16   0014.5ebc.7466  ARPA   Vlan2

2. show mac-address table | i IP-Address

no mac address found

3. Core 1

sh mac-address-table address 0014.5ebc.7466

2  0014.5ebc.7466   dynamic  Yes         35   Gi1/1 ( connected to access switch)

Access Switch

sh mac-address-table address 0014.5ebc.7466

2    0014.5ebc.7466    DYNAMIC     Gi1/0/44 ( connected to 2960 switch)

HSRP Config

Core 1

interface Vlan2
 ip address 10.1.1.254 255.255.255.0
 ip route-cache flow
 standby delay minimum 20 reload 25
 standby 1 ip 10.1.1.1
 standby 1 priority 110
 standby 1 preempt

Core 2

interface Vlan2
 ip address 10.1.1.253 255.255.255.0
 ip route-cache flow
 standby delay minimum 20 reload 25
 standby 2 ip 10.1.1.1
 standby 2 priority 95
 standby 2 preempt

Sh standby

Core 1

Vlan2 - Group 2
  Local state is Active, priority 110, may preempt
  Hellotime 3 sec, holdtime 10 sec
  Next hello sent in 1.143
  Virtual IP address is 10.1.1.1 configured
  Active router is local
  Standby router is 10.1.1.253 expires in 9.844
  Virtual mac address is 0000.0c07.ac02
  1 state changes, last state change 8w1d
  IP redundancy name is "hsrp-Vl2-2" (default)

Core 2


Vlan2 - Group 2
  Local state is Standby, priority 95, may preempt
  Hellotime 3 sec, holdtime 10 sec
  Next hello sent in 1.931
  Virtual IP address is 10.1.1.1 configured
  Active router is 10.1.1.254, priority 110 expires in 7.440
  Standby router is local
  43 state changes, last state change 8w1d
  IP redundancy name is "hsrp-Vl2-2" (default)

[samirshaikh52]

1. are the non working servers in the same vlan as the working servers ?

No. they are in same vlan

2.how do you know the servers can't ping the VIP, is someone else doing  that for you ? If so can they do an "arp -a" on the server and look for  the VIP entry ?

Actually this is a medical server, the vendor logged in during the issue and discovered that they cannot ping the default gateway i.e VIP.

3. can you pick one of the non working servers and trace it's mac address  from the core switch ie. on each switch up to the 2960 can you look in  the mac address tables and make sure the mac for that server is going  out of the right port.

Yes they are coming on right port

[JohnTylerPearce]

Can you post the HSRP config for this vlan from each router? Also can you post the results from 'show standby' with the specific HSRP number from each router?

[Jon Marshall]

When you cannot connect to the server ie. before you clear the arp cache on the core switch, is the mac address that is showing against the server IP the correct one or does it change after you clear the arp cache.

Jon

[samirshaikh52]

It remains the same.

[samirshaikh52]

One thing to be noted

2 servers showing same mac

0014.5ebc.7466

[samirshaikh52]

2 IP addresses

10.1.1.15 and 17 show mac address entry in the mac table on core

[Jon Marshall]

Are these two of the servers not working ?

Jon

[Jon Marshall]

Okay, well that could well stop communication working ie. one server sends ping to VIP but the return packet is sent to the other server.

So i suspect you need to get that fixed.

Are both the servers showing the same mac address not working ?

Jon

[samirshaikh52]

I have just communicated with server team and came to some information that

they have 2 servers 10.1.1.14 and 15 they are working as HA. and IP 17 is working like a virtual IP. and its working like active IP.

[JohnTylerPearce]

I noticed something on your HSRP configuration.

HSRP Config

Core 1

interface Vlan2
 ip address 10.1.1.254 255.255.255.0
 ip route-cache flow
 standby delay minimum 20 reload 25
 standby 1 ip 10.1.1.1
 standby 1 priority 110
 standby 1 preempt

Core 2

interface Vlan2
 ip address 10.1.1.253 255.255.255.0
 ip route-cache flow
 standby delay minimum 20 reload 25
 standby 2 ip 10.1.1.1
 standby 2 priority 95
 standby 2 preempt

You have a priority of 110, if there is an issue, it wil go down by a default priority of 10, which will leave it at a priority of 100 which is higher than core2 HSRP config for this vlan.

[samirshaikh52]

But the whole is not going down, the other servers and clients in the same vlan can ping the VIP.

[samirshaikh52]

server 10.1.1.14 and 15 not working and 17 virtual on server not working as well.

Thanks

[Jon Marshall]

So if you generate a ping from one of those servers what should be the source IP ie. it's real IP or it's VIP ?

When you said you run a continous ping from a client which IP are you pinging ?

Jon