6513 ARP issue with Exchange cluster

[j.boyson]

The problem we are having is with ARP. We have an Exchange cluster. There are two Exchange servers in the cluster. Each Exchange server has two NICs. The NICs are teamed and use one IP address. The Exchange cluster has its own IP. Each Exchange server has a virtual IP for the virtual server configuration. When we run both Exchange servers we are getting conflicting IP messages. We added a static arp entry on one of the 6500s for on of the Exchange servers. The static entry seemed to fix the problem for the short term. After performing more testing we discovered other issues regarding this configuration.

[rjackson]

Which IP addresses are you getting the conflict on, the cluster or the individual server address? What kind of error is it, dup ip?

Have you configured the 6513 port pairs to each server as a channel?

[j.boyson]

We did get a dup ip message. The conflict is coming from the virtual server addresses. The actual NIC addresses do not have problems. The ports on the 6513 are not configured as a channel. I'm not sure if the teaming software on the server will support the ether channel on the NICs. I have not worked with Excahnge clusters.

[rjackson]

Thats what teaming is, a channel.

If I'm following this right it sounds like the servers are sending packets from both nics and not using a virtual mac address. The router thinks its seeing two hosts with the same ip but different macs. That is if the conflict happens with a single server and two nics.

If the problem is happening only when both servers are up then it sounds like the cluster is not working correctly.

One more question for clarification. A server has two nics and one ip address. It should only use one mac address too. The cluster has an address that should only be used by the active server. What is the virtual server? Does it use another address? Whoshares it?

[t.baranski]

You're getting the duplicate IP errors on the servers, right? Generally when clustering/HA is being used this indicates that the servers can't talk to each other for some reason, so they both think they're active.

Not that this helps you all that much. I just want to make sure we're on the same page.

[j.boyson]

I went on site this morning to work on this. We had 3 hours to perform the work. 6 - 9 am EDT. Around 8:45 we did not have the problem resloved. We tried to turn everthing back up to working condition. One of the Exchange stores in the cluster would not mount. We ended up having to restore 150gig of data to get the server back up. We finished at about 3:30 this afternoon. I tried spanning the vlan to put a sniffer on the CAT. As soon as I enabled the spanning, all the nodes in the VLAN dropped off the network. I was on the phone with TAC during this time. The disruption interupted the dtatbases on the Exchange server. The databases are stored on a SAN. There were log files missing that were causing the store not to mount.

So now it's 5:00 and I'm getting ready to recover a blue screening CCM Subscriber. Ouch!

The dup IP messages were on the server. The problems were only occuring when both servers were on line. As a temporary solution I addded static arp entries for the virtual server IPs in the MSFC. I'm not sure how this will affect the failover process though.

Microsoft recommend we change the cluster network infrastructure. It is currently configured with two NICs per server as a team. They are suggesting we use a seperate NIC for the 'hearbeat process'. Good suggestion! I did not configure any of the Exchange Clustering. The current config worked very well with the HP Procurves we replaced.

TAC seems to think that the 6500 is doing exactly what it is supposed to. MS is saying that they've heard of L3 clustering problems like this on the 6500.

I'm not quite sure why the module dropped all the nodes off the network when I tried to sniff. The ode on the 6500 was upgraded to support the new module before the switch was delivered and installed. The modules is a WS-X6148-GE-TX. TAC mentioned something about a problem with the module sending small packets across the ASIC.