Cannot ping server from local switch but servers can ping it. Intermittent problem.

[GCOosthuizen]

Really strange issue we have, running scom for server management we found that we have heartbeat failures to our DR site. The really weird thing is that when you run a trace you get the local 6500 at DR which is the gateway for the server. From the 6500 itself I am unable to ping server which is directly connected yet from another server in the same range and also attached to the 6500 I am able to ping the server in question. This clears by itself within 5min but it is becoming annoying and could really cause problems if we needed to utilize the DR site for production.

I have checked CEF / ARP and CAM tables and confirm visibility of the server yet it just becomes unreachable by the 6506 and the rest of the network. I have attempted upgrading the software on the 6500 to try resolve the issue in case it was software related but taking it to s72033-adventerprisek9_wan-mz.122-33.SXI13.bin which is a safe harbor version of code but this did not resolve the issue if anything it seems to have made it worse. Has anyone else experienced a problem of this nature before?

[GCOosthuizen]

Additional Information:

I can also confirm the server is running at a gig full-duplex, there are no errors on the interface

6506#ping 10.4.137.10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.4.137.10, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

6506#ping 10.4.137.10 so vl 10
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.4.137.10, timeout is 2 seconds:
Packet sent with a source address of 10.4.137.1
.....
Success rate is 0 percent (0/5)

6506#show ip arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.4.137.10             0   5cf3.fc4e.f5bc  ARPA   Vlan10

6506#show mac add | inc 5cf3.fc4e.f5bc
*   10  5cf3.fc4e.f5bc   dynamic  Yes          0   Gi1/6

6506#show ip cef 10.4.137.10
10.4.137.10/32
  attached to Vlan10

6506#show ip route 10.4.137.10
Routing entry for 10.4.137.0/24
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Redistributing via eigrp 100
  Routing Descriptor Blocks:
  * directly connected, via Vlan10
      Route metric is 0, traffic share count is 1

[Rajtilak Sivaluri]

Hi

If i understand this right, ocassionally you are able to ping the server from the switch and sometimes not. Just for ensuring , did you observe any mac flaps during this time. I guess mac-flap notifications are not enabled by default. Please enable and and check.

Also suggest  you to check if all the network configurations are fine on the server. Also next time when you see the issue, please check if  you are able to ping with a souce t his would clarify if there is any issue on the server.

Warm regards

Raj

[GCOosthuizen]

Hi Raj,

The issue occurs for about 5min every 4 hours some days then 8 hours apart the next time it happens. Very sporadic.

I have liaised with the internal Server team and they are checking on their side again but every time I have asked them to check they have come back with the all fine.

Regards,

[Rajtilak Sivaluri]

HI GCO

When the local server is able to reach the issue reported server, this might be issue when reaching different segment. Suggest you to take the routes of  server during the time of issue. which may give a clue, if you still do not find any issue/variation in routes. You probably have to go for a snoop on the server interface or switch interface

[lthangli]

Hi GCOosthuizen, 

Indeed this is strange issue.

Can you please paste output of show run int vlan 10 and show run int gig1/6?

[GCOosthuizen]

As requested:

interface GigabitEthernet1/6
 description ## IS-DC Value Host - Server Infrastructure ##
 switchport
 switchport access vlan 10
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 999
 switchport mode access
 switchport nonegotiate
 logging event link-status
 logging event trunk-status
 load-interval 30
 carrier-delay msec 0
 spanning-tree portfast edge
 spanning-tree bpdufilter enable
 spanning-tree bpduguard enable
 spanning-tree link-type point-to-point
 spanning-tree guard root
 

interface Vlan10
 description ## IS LAN Infrastructure - Layer 3 Management ##
 ip address 10.4.137.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip wccp 61 redirect in
 no ip mroute-cache
 logging event link-status
 load-interval 30
 carrier-delay msec 0
 arp timeout 200
end