Ask the Expert: Scaling Data Center Networks with Cisco FabricPath

[ciscomoderator]

With Hatim Badr and Iqbal Syed

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about the Cisco FabricPath with Cisco technical support experts Hatim Badr and Iqbal Syed. Cisco FabricPath is a Cisco NX-OS Software innovation combining the plug-and-play simplicity of Ethernet with the reliability and scalability of Layer 3 routing. Cisco FabricPath uses many of the best characteristics of traditional Layer 2 and Layer 3 technologies, combining them into a new control-plane and data-plane implementation that combines the immediately operational "plug-and-play" deployment model of a bridged spanning-tree environment with the stability, re-convergence characteristics, and ability to use multiple parallel paths typical of a Layer 3 routed environment. The result is a scalable, flexible, and highly available Ethernet fabric suitable for even the most demanding data center environments. Using FabricPath, you can build highly scalable Layer 2 multipath networks without the Spanning Tree Protocol. Such networks are particularly suitable for large virtualization deployments, private clouds, and high-performance computing (HPC) environments.

This event will focus on technical support questions related to the benefits of Cisco FabricPath over STP or VPC based architectures, design options with FabricPath, migration to FabricPath from STP/VPC based networks and FabricPath design and implementation best practices.

Hatim Badr is a Solutions Architect for Cisco Advanced Services in Toronto, where he supports Cisco customers across Canada as a specialist in Data Center architecture, design, and optimization projects. He has more than 12 years of experience in the networking industry. He holds CCIE (#14847) in Routing & Switching, CCDP and Cisco Data Center certifications.

Iqbal Syed is a Technical Marketing Engineer for the Cisco Nexus 7000 Series of switches. He is responsible for product road-mapping and marketing the Nexus 7000 line of products with a focus on L2 technologies such as VPC & Cisco FabricPath and also helps customers with DC design and training. He also focuses on SP customers worldwide and helps promote N7K business within different SP segments. Syed has been with Cisco for more than 10 years, which includes experience in Cisco Advanced Services and the Cisco Technical Assistance Center. His experience ranges from reactive technical support to proactive engineering, design, and optimization. He holds CCIE (#24192) in Routing & Switching, CCDP, Cisco Data Center, and TOGAF (v9) certifications.

Remember to use the rating system to let Hatim and Iqbal know if you have received an adequate response.  

They might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community Unified Computing discussion forum shortly after the event. This event lasts through Dec 7, 2012.. Visit this support forum often to view responses to your questions and the questions of other Cisco Support Community members.

[Jepoy Diaz]

Hatim,

Is the FabricPath only works if all your layer 2 gear are Nexus flatform? Meaning Can I connect 2960-S to 5ks using FabricPath?

Thanks,

[habadr]

Hi Jepoy,

A device that does not support Cisco FabricPath can be attached redundantly to two separate Cisco FabricPath switches with enhanced virtual PortChannel (vPC+) technology as shown in figure below. Just like vPC (Virtual Portchannel) , vPC+ relies on PortChannel technology to provide multipathing and redundancy without resorting to Spanning Tree Protocol.

For more details, please refer to  "Scale Data Centers with Cisco FabricPath" white paper at

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-605488.html

Thanks

Hatim Badr

[Sarah Staker]

Hatim,

thanks for your answer. Really good information. I have another question though.

What benefits does Fabricpath offer in comparison to STP? Can I still run VPC along with Fabricpath?

Sarah

[habadr]

Hi Sara,

let me start from the last question. Yes you can run vPC however it is called vPC+. VPC+ is FP variant of VPC ...so it is pretty much similar to VPC but with no SPT running as fail safe role since Fabric Path has its own loop prevention mechanism. This is a way for hosts connecting to FP network to connect in an active-active fashion.

So Fabricpath (vPC+) has the same major benefit of  vPC of mitigating the limitations of Spanning Tree Protocol allowing   a single Ethernet device to connect simultaneously to two discrete Cisco Nexus chassis while treating these parallel connections as a single logical PortChannel interface. The result is active-active forwarding paths and the removal of Spanning Tree Protocol blocked links, delivering an effective way to use two parallel paths in the typical Layer 2 topologies

Back to first question

Here are typical L2 limitations 

• Local STP problems have network-wide impact, troubleshooting is difficult
• STP provides limited bandwidth (no load balancing)
• STP convergence is disruptive
• Tree topologies introduce sub-optimal paths
• MAC address tables don’t scale
• Flooding impacts the whole network

With Cisco FabricPath network, you can create a flexible Ethernet fabric that eliminates the constraints of Spanning Tree Protocol mentioned above. At the control plane, Cisco FabricPath uses a Shortest-Path First (SPF) routing protocol to determine reachability and selects the best path or paths to any given destination in the Cisco FabricPath domain. In addition, the Cisco FabricPath data plane introduces capabilities that help ensure that the network remains stable, and it provides scalable, hardware-based learning and forwarding capabilities not bound by software or CPU capacity.

I'm adding previous Iqbal's summary for more details 

- Scalability: Cisco Fabricpath allows you to scale to much larger extent compared to a STP or VPC based design mainly because FP is not using STP at all , so because you are not using STP you dont have to worry about blocking vs forwarding ports as well as things like STP logical ports which is huge consideration for scale when designing any networks using STP.

- Growth : Even if you have two spines ( like in VPC designs ) - its still a good idea to consider FP solution because FP will allow you to easily grow in future without making significant changes in the design or worry about STP loops.

- Bandwidth  the bandwidth increase with VPC compared to a STP design is typically double however with FP you get huge improvement over bandwidth availability as mentioned above.

- MAC address scale : the typical STP networks we are used ( same with VPC ) has drawbacks when it comes to scaling of mac addresses essentially because we are used to concept that every device will learn every mac address in a transit network. FP offers conversational learning which is a new and innovative way of handling mac address learning .With Convesational learning - the devices which are either connected to source or destination hosts will learn the mac address , all the other devices in transit will not learn that mac address thereby providing major scalability benefits to mac address scale.

Thx

Hatim Badr

[chouvantran]

Hi Hatim,

Interested in finding out best method to migrate existing CE using vPC to new FabricPath Spine/Leaf/BorderLeaf architecture (N6k).  We have built the FabricPath spine/leaf in parallel, but would like to know high level steps on the actual cutover with minimal downtime.  Existing topology is N7k (Core) / N7k (Agg) / N5k (Access) + 6200 FI for UCS. We will be connecting the N7k (Agg) to the Border Leaf device and all other Access devices to remainng Leafs.  Thx

[gauravncp]

Hi Hatim & Iqbal,

We have a loop prevention mechanism in vPC - Packets traversed via PL will not be forwarded to VPC member port. Only exception is forwarding will take place if one of the switches member port/ports are down.

Let me explain you my scenario/question. We are running FP throughout our topology. At one point, connections going to Firewall are bundled in vPC. Our peer link is on FP. So if a packet pushed from one Peer device to other, over the peer link (running in FP) & destined to Firewall, do the loop prevention rule is still valid & traffic will not be forwarded to vPC member port despite of the fact PL is on Fabric path.

Note: Question may arise why the Nexus Switch will use the Peer Link for the traffic destined to Firewall when it has the direct path to Firewall. Its because traffic coming from southbound has a next hop of peer switch & we have not configured the peer-gateway command. we are using L3 routing over Layer 2 vpc

In short my question is - Can you please confirm if the below rule applies to both vpc & vpc+ 

"Packets traversed via PL will not be forwarded to VPC member port"

Thanks, 

Gaurav Jain

[dgarnham40]

Hi Guys,

I have a question about FabricPath implementation with 5Ks as aggregation/edge devices and 6500 (VSS) as Core devices. Currently we run vPC on Nexus and southbound to FEX. Northbound is classic Ethernet to the VSS.

I appreciate that the spine/leaf topology with FabricPath requires nexus 7K/5K but could the VSS be used as a non-FabricPath spine? Although FP will only be operating between 5Ks and Soutbound to FEXs?

Is this hybrid topology worthwhile? or is there no real benefit since the 'spine' core switches are not capable of FP?

thank

David