vPC domain id's and sys

[Ronni Feldt]

Hi,

We have a setup with two pairs of Nexus 7000 in the following topology:

NX7K-1A ---------- NX7K-1B

     |                         |

     |                         |

     |                         |

NX7K-2A ---------- NX7K-2B

NX7K-1A|B is one pair, and NX7K2-A|B is the second pair.

Unfortunately we didn't configure the two pairs with a unique vPC domain id, which have turned out to cause some problems especially when enabling the peer-switch feature and MSTP.

In short we are using MSTP with all VLAN's assigned to MIST1. It works in a loop-free topology, where we disable (shutdown) one of the interfaces between the two Nexus pairs, but when enabled a loop is formed. When in a loop-free topology (might also be the case when not in a loop-free topology), the two Nexus pairs, each using peer-switch, sees themselves as root, even though they have different priorities (4095 / 8192 respectively) and the one and only active interface between the two Nexus pairs, is in ALTN/BLK in MIST0 (CST), which basically cuts MIST0 in two.

The documentation is pretty vague on what specific kind of problems two identical vPC domain id's leads to, other than some port-channel issues.

The vPC system-mac is created using 00:23:04:ee:be:<domain-id> which properly is the main cause of the problem, when having two vPC pairs with the same domain id in the same L2 domain. The documentation states that you could configure a static vPC system-mac, and since the domain id is local to the Nexus, would this solve the problem? Or are there other things derived from the vPC domain id?

[alanjbrown]

Ronni,

We have the same physical topology as yourself but don't use MSTP. We also had the same domain id used across sites. The only way forward was to change the domain ids. This caused a massive hit on our network which should have caused a 5min outage but everthing is work now.

Why don't you change the domain ids?

thks,

Alan

[Ruhann Du Plessis]

What you have created is two switches/VPCs (2x pairx of N7Ks) both speaking on the same L2 network using the same identifier, the VPC system MAC.

The domain ID is used to derive this VPC MAC. The VPC MAC address is what is used in STP and other layer2 protocols, to trick switches into believing a pair of VPC switches is one logical switch. This is the reason why it is strongly recommended to use a +unique Domain ID in a contigious Layer2 network.

Manually setting the VPC system MAC should solve your problem. But I would highly recommend you rather schedule 5 minutes of down time and re-configure one of the N7K pairs. And avoid any further undiscovered problems.

HTH

[maheshvisave]

Hi Ruhann........Nice post......Could you please also tell me if the VPC MAC is used for STP and other L2 protocols......is HSRP will be compulsion for data traffic redundancy?........Could you please explain in brief....

Its very old post but still if you/someone can help on data traffic pattern.

Appreciated... 

[Ronni Feldt]

Hi,

Thank you for the answers.

I have already been recommended to change the domain id of one NX7K pair, but because of the weak documentation, I was wondering if you could solve the problem simply by configureing a new vPC system-mac, and if anyone had done that.

It's right that it might solve the problem, but I then will experience new problems - so the safe way is to change the domain id.

[Bamjive06]

Your topology is brave mate. Wachu using it for?

Based on Cisco docs like everyone would agree - I'd   recommend you use a single unique domain ID across. Even though I have   never seen such topology? Maybe let me ask you this question: Do you   have NX5Ks pair hanging off each pair of NX7Ks?  If so I'd stil marvel   at this topology - u'v given me something to think about.

Anyhow. Yes, i look foward to you response esp. that u asked this question 6 months ago. It must be working hey.

How many VDCs per pair? Btw guys, when is L2 network not contiguous?

[Colby Beam]

The domain IDs really need to be different. Each pair of 7ks would be in a single vPC domain. I have seen issues where the vPC will not form between the two pairs, or traffic is forwarded incorrectly if they are the same.

"The vPC peer devices use the vPC domain ID that you configure to automatically assign a unique vPC system MAC address. Each vPC domain has a unique MAC address that is used as a unique identifier for the specific vPC-related operations, although the devices use the vPC system MAC addresses only for link-scope operations, such as LACP. We recommend that you create each vPC domain within the contiguous Layer 2 network with a unique domain ID. You can also configure a specific MAC address for the vPC domain, rather than having the Cisco NX-OS software assign the address."

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/interfaces/configuration/guide/if_vPC.html#wp1803808

[Lucien Avramov]

Best practice would be to use different domain IDs for each vPC pair,

however this can work with same domain ID.

[Bamjive06]

Thanks all.

Side-question: how do i change the email address in this forum? Have since left the company and the forum uses that apparently - i donno how this happened. I tried changing it from main cisco profile - in vain!