3750 stack issues

[Jon Marshall]

This is following on from - 

https://community.cisco.com/t5/switching/windows-2016-connectivity-issue/m-p/4477088#M511617

Ran some SPAN tests on the 3750 stack and it looks like it may be an issue with the switches but the behaviour  is still not consistent. 

storage1 is the working storage subnet 

storage2 is the one with issues 

Test 1

=====

Spanned gi1/0/3 to gi2/0/32 on the 3750 stack. Gi1/0/3 is a port in the storage1 subnet and ran a filter to capture ICMP traffic. Then pinged to hypervisor connected to gi1/0/3 from other hypervisors and traffic shows in capture. 

Test 2 

=====

Spanned gi2/0/3 to gi2/0/32. Gi2/0/3 is in the storage2 subnet and ran same filter to capture ICMP. 

Did a ping from another hypervisor where the ping fails and as expected there was no traffic captured. 

Test 3

===== 

Using same session as Test 2 then ran a ping from the storage server to hypervisor connected to gi2/0/3.  

The ping is successful but no traffic captured. 

It is looking to me like there is an issue with the switches but not sure what. The hypervisor ports we are having issues with pass some traffic to some destinations in the same subnet but not to others but wireshark captures none of the traffic whether it works or not. 

Any ideas most welcome.

Also is there a non intrusive way to test the hardware on the 3750 stack ? 

Jon

[Reza Sharifi]

Hi Jon,

Does show interface for storage-1(1/0/3) vs storage-2(2/0/3) show anything unusual?

Since 1/0/3 woks but not 2/0/3, is there a way to connect storage-2 to a different switch port on switch-1 (1/0/x) and test?

Do the interfaces on the switch stack and the storage have the same MTU setup?

Thanks,

Reza

[Jon Marshall]

Hi Reza 

I ran a show interface on both and there is nothing obvious that is different and the MTU is the same on the hypervisor NICs and the switches across both stacks. 

I want to move gi2/0/3 to another port on switch 2 first and if that doesn't work then switch 1 but I can't really at the moment as we can't afford any more downtime and the risk is always there when moving NICs. 

As this has been working for over a year before we started seeing problems and based on the symptoms I am seeing with packet captures etc. my best guess at the moment is failing hardware but still not confident that is the issue. 

The really weird test result was that even with a successful ping (test 3) the packet capture showed nothing which does point at something going on with switch 2 but there is nothing in the logs and some of the other ports in the storage 2 vlan/subnet are working fine. 

The IOS  12.2(55) SE10 is in use across many of our 3750s and we have not seen this issue before although most are not in stacks. 

The storage 2 vlan is only in use on switch 2 but again that should not be an issue as far as I can see. 

Jon

[paul driver]

Hello Jon
I wonder if you tried clearing the ip stack of the teamed nics to see if it made any difference? I remember having an issue some time back on a wk2012 server regarding intermittent connection drops and disappearing arp pertaining to a pair of teamed nics.

I evntually decided to clear down the ip stack and winsock api of the server nics bindings reapplied the addressing and it solved the issue, I am not saying this would be the root cause but could be worth a shot

The below netsh commands will remove obviously reset you nics, remove and the ip addressing,  a reload is require for winsock as well.

netsh interface ip reset
netsh winsock reset < reload is required

[Jon Marshall]

Hi Paul

As I mentioned to Georg in the other thread the storage NICs are not actually teamed but I have seen the exact same behaviour you describe on teamed NICs on this cluster a few weeks ago. 

I have not got extensive experience with switch stacks as such but I am assuming there is nothing extra you need to do in terms of configuration after you create the stack ie. all configuration applies to all members of the stack ? 

Once we rebuild this cluster onto newer hardware which was planned anyway but has been moved forward I am going to be able to do some proper testing on this cluster to try and work out exactly what is happening. 

The new cluster storage will be on separate 10Gbps switches but the front end side will still be on a 3750 stack so just trying to rule out a general issue with 3750s and switch stacks, IOS version etc. 

Jon