Servers with dual NICs connected to separate FEXs

[Christopher Bell]

Our current topology is a single N7K with two 48port 10Gb F2 linecards and FEX 2224TP in the server racks.  In a few of the racks that house important servers, we have placed dual FEX 2224's.  Each FEX has 2 10Gb links back to the N7k (each link on a separate LC).  I've tried to create a port-channel for the important servers in such a way that each NIC on the server is connected to a separate FEX - but its failing (simple drawing attached).  I've read multiple posts saying this is possible, and others that say its not.  I've also submitted a TAC case and have been told it's impossible to port-channel interfaces on separate FEXs connected to a single N7k - but I find this an impossible limitation and want to verify it's actually right.  Has anybody done this?  Is this really a limitation?

[Bradford Chatterjee]

It is as you say. If you dual-home your FEXs using VPC then you cannot use VPC to connect your server to two dual-homed FEXs. A supported solution would be to single-home your FEXs - each FEX connected via port channel to one 7k - and then dual-home your server to the single-homed FEXs.

[Christopher Bell]

We are not using vPC for connecting the FEXs to the N7k.  We are using Port-Channels.  Each FEX has two 10Gb uplinks to the single N7k - each link on a separate line card: 

interface port-channel6

  description FEX 101

  switchport mode fex-fabric

  fex associate 101

interface Ethernet2/6

  description Member PO 6 FEX 101

  switchport mode fex-fabric

  fex associate 101

  channel-group 6

  no shutdown

interface Ethernet9/6

  description Member PO 6 FEX 101

  switchport mode fex-fabric

  fex associate 101

  channel-group 6

  no shutdown

There is no vPC configuration, the way it's configure is the way I understand to be standard for attaching FEXs to a Nexus 7010.

[Bradford Chatterjee]

That should be supported, then; that is the way that Cisco is recommending we design our new DC.

[Christopher Bell]

My thoughts exactly.  I should be able to connect a single server to two FEXs for redundancy, but neverthless - when I try it fails.  The TAC engineer is saying it's b/c I don't have dual N7ks - each of the FEXs connected to separate N7ks rather than separate line cards on the same N7k.

[Christopher Bell]

Some more food for thought:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10110/data_sheet_c78-507093.html

“The Cisco Nexus 2000 Series can be used in conjunction with a Cisco Nexus parent switch in two main design scenarios (shown in Figure 3):

• Cisco Nexus 2000 Series Fabric Extenders single-connected to one upstream Cisco Nexus 5000 or 7000 Series Switch: In this deployment scenario, access-layer redundancy is achieved through redundant server connections to two upstream distributed modular systems, using vPC (Cisco Nexus 5000 Series) or server NIC teaming to two Cisco Nexus 2000 Series Fabric Extenders.”

I’m still having trouble believing this can’t be done…  I've found no documentation stating that it can't be done and what I have read (as implied by the above) it should be supported.

[Chad Peterson · Accepted solution]

Hi Christopher,

I know from the 7k side you cannot create a Port-channel that spans two local FEXs.  The documentation you highlight should suggest that you can use server NIC teaming to two Cisco Nexus 2000s in an active/passive setup for their NICs.

Only way to span a port-channel from a host to two different FEX is to have each FEX connected to a different N7K and create a vpc to the host.

Chad

[Christopher Bell]

Chad - you are correct.  Confirmed by my SE also.  It looks liek this may change later in an upcoming software release however.