other subnets unable to ping to certain IP

[Sharin Taib]

I have a vlan10 configured on 2 different 6509 with HSRP and the configuration is as such

interface Vlan10
 ip address 1.1.1.130 255.255.255.128
 no ip redirects
 ip flow monitor FlowMonitor1 input
 ip flow monitor FlowMonitor1 output
 standby 10 ip 1.1.1.129
 standby 10 priority 105
 standby 10 preempt

the problem i am facing now is there are certain IP's within this subnet that can't be ping by other vlans

but other IPs are ok.

6509#ping 1.1.1.141
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.141, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

6509#ping 1.1.1.141 so vlan9
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.141, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.2
.....
Success rate is 0 percent (0/5)

6509#ping 1.1.1.133 so vlan9
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.133, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.2

!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

i found a temp solution by clearing the arp cache for the IP's that are unpingable but i am unable to explain why this is happening. does anyone have a clue? routing between vlan9 and vlan10 does not pass through any firewalls, its just inter-vlan.

[Seb Rupik]

Hi there,

On the two different 6509's, what is the output of: sh standby

Alos, on host say 1.1.1.141, are you able to run tcpdump/ wireshark and filter on ARP and ICMP packets. Can you see host sending a response? Where to?

cheers,

Seb.

[Sharin Taib]

the main switch is the one that is showing as active, i am clearing the ARP from this switch.

physically, the device is connected to the standby switch.

can't run wireshark though as i am not the owner.

[Seb Rupik]

It would still be useful to see the output of sh standby .

Are you able to monitor capture on the 6509's? filtering for the two endpoints (ping source and destination)?

cheers,

Seb.

[Sharin Taib]

6509#sh standby vlan10
Vlan10 - Group 10
  State is Active
    2 state changes, last state change 1y5w
  Virtual IP address is 1.1.1.129
  Active virtual MAC address is 0000.0c07.ac0a
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 2.192 secs
  Preemption enabled
  Active router is local
  Standby router is 1.1.1.131, priority 100 (expires in 9.392 sec)
  Priority 105 (configured 105)
  Group name is "hsrp-Vl10-10" (default)

[Seb Rupik]

For the devices that are unreachable what does their arp entry look like when unreachable, eg:

sh ip arp vlan 10 | inc 1.1.1.141

If the ARP entry is complete what is the output for :

sh mac add | inc <mac_address_from_arp_table>

If the source interface correct?

cheers,

Seb.

[Sharin Taib]

the unreachable devices all have mac-addresses, when traced to physical port its up and running.

[Richard Burts]

A common source of symptoms like this where a device is pingable within the local subnet but not pingable from remote subnets is that the device at 141 does not have the correct default gateway configured. Can you verify what is the gateway for 141?

HTH

Rick

[Sharin Taib]

yup, default gateway is configured properly on the servers. its set to the HSRP IP

[Sharin Taib]

the same thing happened again today. i cleared the ARP and compared the before/after of the ARP detail and it was the same.

i suspect that due to the refresh timing, it takes 4 hours before the connection re-establishes.

i decided to try a static ARP to see if it will resolve the issue as i can't find a way to modify the refresh timing just for this IP.

ICES04-DIS01-6509#sh arp 1.1.1.237 det
ARP entry for 1.1.1.237, link type IP.
  Dynamic, via Vlan10, last updated 0 minute ago.
  Encap type is ARPA, hardware address is 0015.5dc4.9420, 6 bytes long.
  ARP subblocks:
  * Dynamic ARP Subblock
    Entry will be refreshed in 264 minutes and 45 seconds.
    It has 2 chances to be refreshed before it is purged.
    Entry is complete.
  * ARP HA
    ARP entry is a new entry and has not been synchronized to standby RP.
  * IP ARP Adjacency
    Adjacency (for 1.1.1.237 on Vlan10) was installed.
    Connection ID: 0
  * IP ARP VLAN ID
    Subblock data size is 4 bytes.
    VLAN IN ID: 10
    VLAN OUT ID: 10

[Richard Burts]

It is interesting and probably helpful that the problem happened again and that you solved it by clearing the arp table. It is especially helpful to know that you checked the before and after and there was no difference. I believe that this suggests that the problem is not on the switch but is on the host at 141.

As I mentioned when a device is able to communicate with locally connected addresses but not with remote addresses then it suggests that there is some issue with the gateway for this host. You have told us that the host at 141 does have the correct gateway configured so the isue must be something else. I am guessing that this may be what is going on:

- when you do a normal ping from the switch the source address is in vlan 10 and when the host at 141 responds it just uses the entry in its arp table for the switch address in vlan 10.

- when you do a ping from the switch specifying that the source address is in vlan 9 then when the host at 141 prepares to send its respionse it needs to send the packet to its default gateway. I am guessing that at that point the arp table on the host does not have an entry for the HSRP address and this causes the response to fail.

Is there a way to check on the content of the arp table for the host at 141 and compare it in operation when thigns are working to what it has then the problem occurs?

HTH

Rick

[Sharin Taib · Accepted solution]

was finally able to get a vendor in and found that

1. the device with the issue had the wrong mac-address mapped to the gateway in its arp table. it was not tied to the mac address of the standby interface of my core switch, it was instead mapped to the mac-address of the VM server.

2. another device is not able to have a gateway configured so this device had to be resolved differently.

learned something new.

[Richard Burts]

Thanks for posting back to the forum and letting us know that you have solved the problem. I am glad to see that the problem was on the device at 141, as I had suggested it might be. It is a good lesson that sometimes the problem is not with our network devices and their configuration but may be in the devices that connect to our networks.

HTH

Rick