Nexus FEX and Static pinning

[fsebera]

Hi guys!!

With reference to Cisco FEX and Static Pinning 1, 2, 3 or 4, it appears the static pinning rules apply equally to both the Nexus 5000 and 7000 series switches - Is this correct?

5000<---->FEX

7000<---->FEX

Also, If you do not configure the optional static pinning # parameter on the 5000 or 7000 series switches, the 5000/7000 will perform this pinning selection dynamically. If you start off with a single uplink without configuring the static pinning option all data will obviously flow on the single uplink. BUT, If you then add a second uplink, the 5000/7000 will automatically reset the FEX ports (causing an outage) with the end result being:

The first half of the FEX switch ports will communicate on uplink 1 and the second half of the FEX switch ports to uplink 2

Is this correct?

Thank you

Frank

[Kelvin Willacey]

Q1: I am not 100% sure but I believe it is the same for both platforms and I haven't seen any difference.

Q2: As far as I know it's not dynamic, I believe the default is one so all the FEX ports will use a single uplink, unless it's a port channel. If you add a second uplink then you would need to change the max pinning links to two to utilize the second uplink at which point half the ports will use uplink 1 and the other half will use uplink two. I do believe this is disruptive and there is a command that can be used to help with it.

This is a great document that you should check out:

http://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-5000-series-switches/C07-572829-01_Design_N5K_N2K_vPC_DG.pdf

[fsebera]

Yes I just confirmed the default setting for Nexus 7010 Sup-1 NX-OS 6.2 is when you create a new FEX configuration NX-OS automatically configures   pinning max-links 1   under the FEX # configuration. So you have a choice to change the pinning number - if desired or needed - assuming you have additional uplinks.

We have multiple FEX interfaces, some are configured as a port-channel and some are not but both have pinning max-links 1 configured - here is the config for port-channelled FEX

Config:

fex 116
  pinning max-links 1
  type N2248T
!
interface E8/3 and e8/4
  switchport
  switchport mode fex-fabric
  fex associate 116
  channel-group 116
  no shut

interface port-channel 116
  switchport
  switchport mode fex-fabric
  fex associate 116

I guess the question is does the port-channel override the pinning max-links 1 command or does the pinning max-links 1 have no effect on the port-channel?

Thank you again

Frank

Thanks for the doc, I already have it bound in my binder along with this one too for N7k.

http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

[Kelvin Willacey]

Yes and no.Yes from the perspective that only one uplink is in use, in your case the port channel, so the pinning-max links sends all traffic over the single uplink. No from the perspective that the portchannel hashing algorithm is what dictates what physical links the traffic will be sent over.

[fsebera]

So I understand from your explanation as:

if I have a single port-channel (2 or 4 physical links) this port-channel link is seen as a single logical link between the 5000 or 7000 and FEX so pinning max-links 1 would apply. The port-channel logic would determine which physical link to forward traffic.

If I have 2 port-channels (2 physical links per port-channel) between 5000 or 7000 and FEX pinning max-links 1 would still use only a single port-channel link and the second port-channel would be unused. Changing  the pinning max-links 1 to pinning max-links 2 would then use both port-channel links (I.E. all 4 physical links).

Is this correct?

Thank you

Frank

[Kelvin Willacey]

Yes I believe that is what would happen. The first port group will use the first uplink and the other port group will use the second uplink. The document does state that re-configuring the pinning is a disruptive process.

[fsebera]

I also wonder if using 2 port-channel up-links (each with 2 physical links) and pinning max-links 2 would also set the FEX 48 port switch to

ports 1-24  to port-channel up-link 1 and

ports 25-48 to port-channel up-link 2

Sorry for asking for the continuous conformation but I don't want to be the guy that brings the data center down :(

Thank you

Frank 

[Kelvin Willacey]

Are you saying that you want two port channels from one FEX to the same parent switch? I don't think that's a supported topology. I can't recall seeing it in the design guide, although it does seem that it could work.

It seems your only options would be to add all four ports to a single port channel or utilize max-links 4. Are you doing this for over subscription reasons? What type of 2K do you have and is the parent a 5K or 7K?

[fsebera]

Ok, after reading many different Cisco docs and finally getting a hold of the Cisco Press Data Center Virtualization Fundamentals by Gustavo Santana - he clearly states on PAGE 303 the FEX connection only supports a single port-channel of 1 to 4 ports and thus pinning max-links 1 is the only option.

Thank you guys for hanging in there with me on this one!!

Frank

[Reza Sharifi]

Hi,

I don't think the pinning command has any effect on the Portchannel, because I can assign multiple physical interfaces to the same Portchannle and the pinning command still show the max is 1.

HTH