Odd 3750X/N7K behavior

[Chris McDaniel]

Group - thank you in advance for your help on this issue.

Our setup -

1 N7k using 10G uplinks (po interface) to a stack of 3750X-48(2)

2 Exchange servers gig attached to the second 3750X switch

N7k is running 5.1.3

3750X - 12.2.58.se2

Issue -

Periodically and at odd hours we lose connectivity to the exchange cluster at this facility.  However the servers are reachable from inside the facility they are questionable from outside the facility.

behavior we have noticed -

During the event a packet capture was taken on the exchange server and some interesting packets showed up.  Specificlly packets destined for MACs that are not connected to the exchange server port.  Also these are not broadcast or multi-cast packets.  The MAC addresses are actually connected to 10G ports direclty attached to the N7K via HP flex fabric blade centers.

Questions -

How can this happen? Yes both 10G ports in the 7K and the access ports of the 3750X are in the same VLAN and sharing the same IP space.  The exchange servers are connected to gigabit switchports in the 3750Xs using the switchport access vlan X command - no trunking or extra commands.

Its important to note its not just 1 or 2 packets, but an entire conversation between two devices showing up on a port that is not directly connected to either device.  The two MAC we see are attached to the 7K.

The issue only last for a few minutes, but is noticable by users and admins alike.  It clears just as misteriously as it begins.

Help is greatly appreciated!

[Bilal Nawaz]

Hello Chris, this is quite interesting, sounds related (but not exactly) to a problem I am seeing in the environment i work in - specifically with Exchange (2010).

I do not think its strange that you see other traffic on a packet capture - IMHO this is normal, its like a sniffer and will pick out anything and everything on the same wire! (same vlan) Although probably worth taking a look at the packet capture to investigate further.

We use a load-balancer that serves the Exchange connections - is there something similar in your environment?

Is your Exchange on dedicated hardware or is it VMs?

Our problem has been with VM on HP ESX hosts. Every now and then the Exchange CAS VM's seems to drop connections for whatever reason, when clients try to connect back via the load balancer, it smashes the CAS, and it spikes the resources on the servers and takes a few mins to recover. We've temporarily changed to dedicated HP servers with 1Gb nics to 3750X which trunks to the 6509's and it has been perfectly fine.

Case has been with Microsoft and is now with VMWare.

When you mean the servers are reachable within the facility but not out of the facility, does this mean within the datacenter or site it works, but out of this the problem happens? What tests have been carried out for this?

What kind of performance do you see on your Exchange servers during this time. Is there anything happening in the network at that time, like sudden traffic load, or poor performance from network infrastructure - any drops or latency in links, any errors on the interfaces that could potentially be in the path of traffic flow? etc...

Just one point I'd like to mention - 12.2(55)SE7 is recommended IOS for 3750X.

See what it did for me, here: https://supportforums.cisco.com/message/3962833#3962833

There are many factors here which could be the cause for the problem.

Hope this helps.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

[Chris McDaniel]

Hi Balil - Your issue sounds similar but like you said - "not 100% the same".  What I find odd is that the packet capture is being performed on the Exchange server, not the switchport.  Therefore I believe we should only see traffic to/from that server.  Infortunately we see traffic from servers that are not connected to that switchport.

In the packet capture I noticed that during the minute or two time from where the servers are not available remotely, there is a sizable traffic load increase on the switchport with traffic not destined for the exchange servers. This is TCP based traffic between different servers. I am excluding any broadcast traffic as that will always be in the capture.

Our hardware is dedicated HP servers 1G connected to the 3750X stack 10G uplinked to our N7K.  I'll look at the support forum link and see of that gives me any other suggestions.

Thanks!