N2K dual homing to N5K final sentence

[Massimo Baschieri]

Hi all,

I've read a lot of posts on this community and outside regarding the practice of dual homing nexus 2k to a couple of nexus 5k.

Comments vary from scared to enthusiast, so I'm somewhat usettled about what do to in a forthcoming project I'll have to put in place in a few weeks.

Can someone with hands on experience tell a definitive word about the topic?

Does it worth the effort?

Are there any tradeoffs or best practices?

Least but most important, does it work?

I'm however interested also in comments about the EvPC feature.

In my scenario I've got few vlans, all of them vpc, iscsi, n5k's as core with L3 module, about 20% of orphan hosts.

Models are 5596T and 2224TF.

Thanks,

Massimo.

[Leo Laohoo]

You'll also see alot of documentations about dual-homing the Nexus 2K up to two Nexus 5K.  Please avoid that temptation.  Instead, the best "well proven" practice is to dual/multiple-home your uplink to a single Nexus 5K and dual home your servers to two Nexus 2K.

[Massimo Baschieri]

Hi Leolaohoo,

thanks for your point of view, I can see that it's a quite pervasive opinion, but I'm interested in understanding why I have to avoid dual homing.

It simply doesn't work well?

It's about troubleshooting?

Or it's about error prone configuration?

[Leo Laohoo]

It simply doesn't work well?

When the VPC of the 5500 breaks, you're N2K downstream will go NUTS!

[Massimo Baschieri]

When the VPC of the 5500 breaks, you're N2K downstream will go NUTS!

Scary point !!!

Do you have experienced that in latest releases also, such as 6.x?

[Steve Fuller · Accepted solution]

Hi,

My take on this is as follows:

When the VPC of the 5500 breaks, you're N2K downstream will go NUTS!

Sorry, but I have to disagree with this statement. When the vPC peer-link breaks the behavior of the Nexus 5K and FEX is exactly the same as for any “device” connected via a port-channel, and that is that the vPC secondary will shut down its vPC member ports. The primary vPC peer continues to operate and you still have a working topology.

The following are my two dual-attached FEX via Po101 and Po102, and the result when I shut the vPC peer link. You can see this below in that the device 5548-1, which is vPC secondary, shows the FEX as off-line when the peer-link is down. The primary vPC peer, 5548-2 still has the FEX on-line and traffic still gets to/from servers connected to my FEX.

ocs5548-1# sh vpc

Legend:

                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 1

Peer status                     : peer link is down

vPC keep-alive status           : peer is alive

Configuration consistency status: success

Per-vlan consistency status     : success

Type-2 consistency status       : success

vPC role                        : secondary

Number of vPCs configured       : 2

Peer Gateway                    : Enabled

Peer gateway excluded VLANs     : -

Dual-active excluded VLANs      : -

Graceful Consistency Check      : Enabled

[..]

ocs5548-1# sh fex

  FEX         FEX           FEX                       FEX

Number    Description      State            Model            Serial

------------------------------------------------------------------------

101        FEX0101               Offline    N2K-C2232PP-10GE   SSI155001QZ

102        FEX0102               Offline    N2K-C2232PP-10GE   SSI15460AT7

ocs5548-2# sh fex

  FEX         FEX           FEX                       FEX

Number    Description      State            Model            Serial

------------------------------------------------------------------------

101        FEX0101                Online    N2K-C2232PP-10GE   SSI155001QZ

102        FEX0102                Online    N2K-C2232PP-10GE   SSI15460AT7

So what about my experience with dual-attached FEX? We have a large deployment of both single and dual attached FEX globally and have seen three significant issues with the dual attached FEX:

1. We replaced a Nexus 5K with a new device (so blank config) and we found that when the new switch booted we saw all interfaces on all FEX bounce. This meant we momentarily lost connectivity to all our servers, even those that are dual NIC'd. Neither we nor Cisco have been able to recreate.

2. There's a bug (CSCub46846) in the code we run whereby if a FEX has its speed hard coded, when one of the Nexus 5K is rebooted, those ports that have hard-coded speed will flap. This is due to the N5K that's being booted sending an auto-speed to the FEX interfaces before figuring it's actually supposed to be hard set.

3. We had an issue with MTS buffer leak that meant the vPC peer link went down, and the only fix was a reboot. That shouldn't have been a problem only the wrong switch was inadvertently booted. This meant when it came back up it still couldn't establish the peer link, and the switch also now couldn't bring up the vPC to the FEX. We obviously still needed to boot the switch with the MTS buffer leak, which was also now the only switch controlling the FEX, which meant all our FEX interfaces went down. This was avoidable, but I think it goes to show what the added complexity of dual attached FEX can do.

So would I run dual attached FEX? If you're using Nexus 5K as the upstream switch, and you have a lot of single attached servers/end system, then I think it's of benefit. If all your servers are dual NIC and connected to two FEX, then I think I would probably leave the server NIC teaming to sort out the problem of a FEX or Nexus 5K failure.

I'm sure given time Cisco will fix the issues we've seen, but at this point in time I think there's a number of issues still waiting to jump up and bite you.

Regards

[Leo Laohoo]

Thanks Dave.

That's a good update you've posted. 

[Massimo Baschieri]

I'm sure given time Cisco will fix the issues we've seen, but at this point in time I think there's a number of issues still waiting to jump up and bite you.

Nice post, did you take into account the latest 6.x release in your consideration?

[Steve Fuller]

Thanks for the rating etc.

@loelaohoo

Thanks Dave

It's Steve, but hey I've been called a lot worse in my time

@Massimo

Nice post, did you take into account the latest 6.x release in your consideration?

The issues I've had have been with 5.1. As I said, Cisco normally fix things and may have already done so in 6.0, but I've no experience so can't make any real statement as to the good, bad or otherwise.

Don't get me wrong, I think dual-homed FEX are a neat idea, but I just wonder if we're putting too many layers of resilience into the mix. As most servers that would go into data centre today have dual NIC connected to resilient FEX, do we also need to make the FEX resilient?

Regards

[Leo Laohoo]

It's Steve, but hey I've been called a lot worse in my time

Mea culpa.  Don't know what hapened, Steve.

[Steve Fuller]

No worries. I've been crying into my pillow for some time, but think I'm over it now

[Massimo Baschieri]

The issues I've had have been with 5.1. As I said, Cisco normally fix things and may have already done so in 6.0, but I've no experience so can't make any real statement as to the good, bad or otherwise.
Don't get me wrong, I think dual-homed FEX are a neat idea, but I just wonder if we're putting too many layers of resilience into the mix. As most servers that would go into data centre today have dual NIC connected to resilient FEX, do we also need to make the FEX resilient?

Thanks Steve and Loelohoo, now things are much more clear to me.

I'm curious to know your thoughts about day by day configuration, is the constraint to have the configurations of both n5k perfectly aligned a real pain?

And what about troubleshooting?

[Steve Fuller]

Hi Massimo,

I'm curious to know your thoughts about day by day configuration, is the constraint to have the configurations of both n5k perfectly aligned a real pain?

I wouldn't say it's a real pain, but it's different:

• For each FEX host interface you now have to configure both upstream Nexus 5Ks so double the configuration. That said, even with straight-through FEX you're probably visiting both switches anyway if you've a dual NIC server so not a significant increase in configuration effort.
  
• If your network operations teams have been used to only dealing with straight-through FEX, then they'll probably get tripped up a few times when a port doesn't come up.... as they've not yet configured it on the second switch.
  

And what about troubleshooting?

Again not a pain, but different. As there's effectively a vPC between the FEX and the upstream Nexus you need to consider the fact that traffic from a single server NIC can now go via either of the N5K.

The other thing that caught me, albeit only in our lab, was that as thre is now a vPC between the FEX and N5K you can't run routing protocols on servers attached to the FEX. We use Anycast IP on our DNS servers and then use OSPF to advertise the Anycast address to the network. When I moved our lab DNS server to the dual-homed FEX it immediately broke the routing.

Regards

[Steve Fuller]

Hi,

And to just add a little more to the final sentance, the other factor you may want to consider in relation to straight-through or dual homed FEX is the number of FEX supported by the Nexus 5K.

Basically if you dual-home FEX to a pair of Nexus 55500 switches then the maximum supported is 24. If you single attach the FEX, that same pair of Nexus 5500 switches can support 48. You may not want that many from an operational perspective, or even have enough physical ports on the Nexus 5500, but something to consider.

The maximum number of supported FEX is discussed in the forum post Max FEX with Nexus 5000 VPC Pair.

Regards

[Massimo Baschieri]

Basically if you dual-home FEX to a pair of Nexus 55500 switches then the maximum supported is 24. If you single attach the FEX, that same pair of Nexus 5500 switches can support 48. You may not want that many from an operational perspective, or even have enough physical ports on the Nexus 5500, but something to consider.

That was clear to me, but I agree that this discussion would not be the "final sentence" without a mention about that

Thanks to your precious and fruitful help.