Ping/Connection issues Layer 2 swiched network

[gcunningham4]

Hello,

This is an issue we seem to have and I am hoping someone may have seen it before. It is a very random issue, I cant replicate it, it doesn't happen often, maybe 4-6 times per month.

Network background: Layer 2 switched network (VSS). 2x core switches, with roughly 60 distribution/access switches hanging off them. Switches are mostly 3750V2s (Access) with 4500s providing distribution and access.

Issue: Computer is connected to switch port. Computer mac address comes across switch and all the way back to Cores (can trace and verify with sho Mac add command. Arp table, (Sho IP arp), DHCP (AD), and computer (sho ip) verifies that computer is pulling a DHCP address. However, computer does not have network connectivity (cant access internet or internal network resources). Cant ping computer from another computer, the switch it is connected to, or the core switches.  

Troubleshooting attempted. Replace cables (at device and switch), default port and rebuild, move to different port on switch, clear mac add dynamic, move to different VLAN (Pulls IP but same issues persist), clear IP ARP on Core.

So far the only fix we have found is to move the device to a completely different switch, somewhere else in the building. When we do this, it works as designed. Then we move it back to the original switch/Port it was plugged into and everything works as designed.

Any suggestions of where else to look would be greatly appreciated. Yes we have a fix, but would be nice if there was something I can do without moving the PC to a different location.

Thanks!!!

[Reza Sharifi]

Hi,

Does this happen across multiple access switches or just a single one?

What version of IOS is running on the access switches?

Have you tried upgrading the IOS on a few of them and test?

HTH

[gcunningham4]

This happens across multiple accesses switches, but never all at the same time. I will say that I am not positive if this is relegated to the 3750s or the 4500s or if it is both. I will definitely have to monitor that. The most recent was on a 3750 stack.

IOS:

C3750V2   12.2(55)SE4

4500s (cat4500e-UNIVERSALK9-M), Version 03.04.04.S. Unfortunately IOS Version is dictated to us. Do to the random nature of the event, even if I did upgrade the IOS on one switch for test purposes, it wouldn't be much a scale for success or failure.  

[cofee]

Issue: Computer is connected to switch port. Computer mac address comes across switch and all the way back to Cores (can trace and verify with sho Mac add command. Arp table, (Sho IP arp), DHCP (AD), and computer (sho ip) verifies that computer is pulling a DHCP address. However, computer does not have network connectivity (cant access internet or internal network resources). Cant ping computer from another computer, the switch it is connected to, or the core switches.  

Based on your explanation it appears host has a valid IP address and a default gateway but unable to reach anything inside the network or outside.

I have following questions/suggestions:

A) When it tries to ping its default gateway what happens? can you try to sniff the traffic on the local machine with wireshark?

B) When it tries to ping its gateway which I believe is the core switch (VSS stack) can you create an access list with specific host (local machine) and default gatway (SVI on the core) and debug it so you can make sure that icmp packet is even reaching the core switch.

c) Also when the ping fails can the host ping its own ip address?

d) could this issue have to do something with spanning tree or a duplicate address in the network?

[gcunningham4]

1. A) When it tries to ping its default gateway what happens? can you try to sniff the traffic on the local machine with wireshark?

We haven't tried. Will look into doing that next time we have issues. Should be able to see if were getting some type of retransmission or blocked errors.

1. B) When it tries to ping its gateway which I believe is the core switch (VSS stack) can you create an access list with specific host (local machine) and default gatway (SVI on the core) and debug it so you can make sure that icmp packet is even reaching the core switch.

Don't want to ACL the VLAN, however, Wire shark on the core should give me same results.

c) Also when the ping fails can the host ping its own ip address?

Yes. I was wondering if maybe something was going on with the IP stack. Might try deleting and restarting the stack. I have tried disabling and enabling the NIC card.

d) could this issue have to do something with spanning tree or a duplicate address in the network?

I have verified with DHCP (AD) and IP Arp on the core (no duplicates). I have done a Cursory sho spanning tree vlan, everything seemed to be in order. However, I will definitely dig a little deeper, with the spanning Tree.

Thank you for the suggestions..