03-28-2011 09:03 AM - edited 03-01-2019 05:53 AM
Ozden Karakok is a Technical Leader at Global Technical Center in Europe. Ozden has been with Cisco for twelve years. She specializes in Storage Area and Data Center Networks. She worked as an escalation engineer in the Cisco Storage Technical Assistance Center before joining to Customer Advanced Engineering Team. She is currently supporting Data Center and Unified Computing Solutions through product testing and early field trials. She holds CCIE certification #6331 in Routing and Switching, SNA/IP and Storage Networking. She is also VMware Certified Professional.
This document contains the answers provided for the questions asked during the live "Ask the Expert" Webcast session on the Topic - Fibre Channel over Ethernet (FCoE)
The following experts were helping Ozden to answer few of the questions asked during the session: Ph.D. J Michel Metz and Nina Rezai-Mia. Both are data center technical experts at Cisco.
The Complete Recording of this live Webcast can be accessed here.
The presentation provided during the live event is attached to this document. Scroll down to the bottom to be able to download it.
Yes, it is supported in NX-OS 5.0. Nexus 5000 platform supports N port identifier virtualization (NPIV) and N-Port Virtualization(NPV) since NX-OS 4.0 version. Nexus 5K supports two modes. By default, Cisco Nexus 5000 and 5500 Series switches operate in fabric mode, in this mode switch provides standard FC switching capability and features. Second mode is NPV, in NPV mode Nexus 5K switches relays all the traffic from server-side ports oto the core switch.(NPIV enabled switch). The core switch provides F port functionality and all FC switching capabilities.
Yes, FCoE between Data Centers is possible. VE Ports allow you to build multi-hop FCoE fabrics by interconnecting multiple FCF, VE port functions as a Fibre Channel E-port running on top of a lossless Ethernet fabric. The maximum distance between two Cisco Nexus 5000 Series switches is 3000m for FCoE (lossless Ethernet) traffic with NX-OS 5.0(2)N1.1. It can be enabled using the pause no-drop buffer-size buffer-size pause-thresholdxoff-size resume-threshold xon-size command in QOS configuration. Here are the maximum distances:
Nexus 55xx to Nexus 55xx - 3km
Nexus 50x0 to Nexus 50x0 - 3km
Nexus 50x0 to Nexus 55xx - 3km
Nexus 55xx to Nexus 2232 - 300m
Nexus 50x0 to Nexus 2232 - 300m
We support Virtual LAN on top of the Fibre channel Over ethernetwork. So trunking means we carry different Virtual LAN altogether in one ISL. And the same on FCOE we also support Virtual SANs and can carry multiple virtual SANs on one isl. In turns of multi-hop FCOE we will see that we could carry multiple virtual LAN and virtual SANs together.
Yes. In latest UCS 1.4 release, you can directly connect FCoE targets to the Fabric Interconnect ports. Also on UCS system we have FIP cabable (Gen2) CNAs and Pre-FIP (Gen1) CNAs running on the blades that utilizes FCoE between blades and Fabric Interconnect. If you would like to extend Fabric Interconnect connectivity with VE ports and connecting them to Nexus 5K switches in terms of running multi-hop FCoE, we don't support this on the UCS software yet.
After the 10Gig model the discussion are in 12Gig and 16Gig. If you are asking Cisco will be supporting 16gig FC, yes we will be supporting it and we will have products that will be supporting 16gig FC.
The limit is 7 actually. You can have upto 7 FCoE hops that are tested and supported by Cisco. If there is any requirement to suppoprt more than 7 hops, that can be tested by Cisco. Please contact your Cisco Account Manager if you require more than 7 hops.
Yes, if you are asking you can use different COS values for PFC. PFC uses the IEEE 802.1p CoS values in the IEEE 802.1Q VLAN tag to differentiate up to eight CoSs that can be subject to flow control independently. So on Nexus 5K,by default we have 8 lanes , 8 qos-groups. Qos 6 and Qos 7 are reserved for control plane trafic and the rest 6 qos-groups can be assigned to drop-class or no-drop-class. By default FCoE is assigned COS 3.
Yes if you want to carry multiple data VLANs on ESXi , we definitely need to configure trunking. Also another reason if you want to use FCoE, Fiber Channel over Ethernet Initialization Protocol (FIP) uses the native VLAN and therefore all FCoE links should be trunked to carry the FCoE VLAN as well as the native VLAN.
CEE is the Converged Enhanced Ethernet. The CEE authors group was an informal group of companies that submitted the initial proposals of PFC , ETS and DCBX to the IEEE DCB working group.DCE is a Cisco term that describes the architectural collection of ethernet extensions based on open standards. DCE includes support for the ethernet extensions found in EEE DCB.
There is an option in the Nexus 5000 called storage service license which is activated when it is shipped from Cisco. When ordered as a seperate line card item an activation key and the procedure for activating the code is delivered in an envelope. Because this is always present and only activation is needed and any Nexus 5k will automatically initiate the 120 days temporary license for a storage protcol service from the first use of FC or FCoE command.
No, there is no Ip layer in FCoE. FC-BB-5 contains FCIP that is "IP-routable". FCoE is FC-routable. FCoE switches may forward FC frames across different Ethernet Clouds, FCoE switches may forward FC frames over the Internet using FCIP.
Yes. You need to enable jumbo frames because the FC payload is 2,112 bytes and enabling jumbo frames will keep FCoE traffic from being fragmented. On Nexus 5K, we have a pre-defined Class of Service (CoS) to be FCoE, which is by default has the value 3. Within this CoS, the MTU size it set to 2158.
You can refer the book “I/O Consolidation in the Data Center” by Silvano Gai, Claudio DeSanti
You can also visit the following resources:
Fibre Channel over Ethernet (FCoE)
IEEE 802.1 Data Center Bridging – White Papers
http://www.fcoe.com & FC-BB-5 Standard http://www.fcoe.com/09-056v5.pdf
Nexus 5000 Troubleshooting Guide
Yes you can use multiple classes of service for FCoE. We have 8 lanes, 8 QOS-groups. 6 and 7 are reserved for control plane traffic that is used by the system. That leaves us 6 qos-group to use. One of the 6 qos-groups is pre-defined as drop class and it is class-default(drop class) and that leaves us 5 qos-groups. In the remaining 5 qos-groups, only 4 can be configured as "no-drop" and one of these could be FCoE.
Yes we have the interoperability matrix which we published on CCO Cisco.com.
We try to certify, different Storage, different CNAs, different O/S and different NX-OS versions. Also most of the OSM vendors publish their interoperability matrixes in their own web sites.
A World Wide Name (WWN) is a unique identifier which identifies a particular Fibre Channel target, Each WWN is an 8 byte number derived from an IEEE OUI and vendor-supplied information.
We have two flavours of Nexus 5K switches.They are Nexus 55xx and Nexus 5010/5020. On Nexus 5010/5020 we support Gen1 and Gen2 CNAs.
On Nexus 55xx and Nexus 2232PP we only support Gen2 CNAs such as QLE814x/QLE815x, Emulex One Connect...
You will need a lossless ethernet network that will support FCoE, to build this network minimum requirement is to have a Fibre Channel Forwarder (FCF switch) , host/server that has CNA installed and a FC or FCoE target that is providing the access to the block storage for the server.
FC and iSCSI share the same SCSI stack (to some extent), there are some differences in the protocol stacks. FC is generaly used for tier-1 apps so will FCoE by extension. iSCSI however is also seeing attraction in that area.
The iSCSI protocol can be implemented in networks that are subject to packet loss, and iSCSI can run over 1 Gigabit Ethernet (1GbE). FCoE requires 10 Gigabit Ethernet (10GbE) and a lossless network with infrastructure components that properly implement pause frame requests and per-priority pause flow control (PFC) based on separate traffic classes that map to different priorities. The idea behind PFC is that during periods of congestion, high-priority traffic is allowed to continue while lower-priority traffic is paused.
Your 10GbE switches will need DCB support for FCoE, including a range of enhancements for classes of service, congestion control, and management. FCoE also requires Jumbo Frames because the FC payload is 2,112 bytes and cannot be broken up; iSCSI does not require Jumbo Frames.
iSCSI has a TCP overhead contributing to both the size of frame and also operational inefficiencies that come with TCP retransmissions. There maybe benefits to iSCSI in an ethernet loss-less environment as these retransmissions are eliminated.
Cisco currently has multi-hop FCoE in the Nexus 5000 series of switches. Additionally, the Nexus 7000 and MDS 9500 series Director switches have announced multihop capability; watch for further information very soon.
I hope I understand your question properly. The DCB standards are built on what CEE defined originally. Cisco switches abide by both standards.
DCB is an absolute requirement for FCoE. The links *need* to be loss-less before it can even be considered for FC traffic as storage applications are generally intolerant of loss. Regarding O/S, every organisation is different and a value is chosen based on their tolerances. We have seen 4/1 over subscription to up to 15/1. When designing the storage network the link O/S should be considered as well as the internal switch hardware O/S however the latter is generally lower so not a bottleneck but needs to be considered. This O/S is platform specific and changes in accordance to the HW.
I assume the question is if the process of de-encap/re-encap FC-to-FCoE causes a performance hit, and that is not the case since packets are parsed and modified as they are transmitted to xbar on their way to egress regardless of protocol type.That all happens at speeds much faster then any input device can forward frames in,or receive frames out.
The QoS setting (negotiated) sets up the adjacent neighbors so that they can have a loss-less ethernet conversation. The packets are tagged with a CoS which indicates the Virtual Lane they ough to be in. In your case where I/O rates are increasing due to VM adoption, if the host is tagging these packets as loss-less with the correct CoS marking, then the frames are transmitted with those capabilities and the switch treats them as such. However you only ever have the 10G link offered. This QoS setting doesn't affect the increasing I/O rates.
Yes, we need to configure "switchport trunk" since Fiber Channel over Ethernet Initialization Protocol (FIP) uses the native VLAN and therefore all FCoE links should be trunked to carry the FCoE VLAN as well as the native VLAN.
29. Can you explain what you meant by multihop extending beyond the access?
Up until now, you could connect FCoE from the server (using CNA) to an access switch. With Multihop FCoE as defined by FC-BB-5 working group of T11 we can extend FCoE beyound the access switch (via an ISL) to other switches and perhaps into the SAN core. You can now build Ethernet based SAN rather than a FC only SAN.
At this point in time, no. Currently ethernet traffic and FCoE is handled on their own respective VLANs, which means that you can apply FabricPath to the Ethernet VLAN. FCoE uses FSPF for its forwarding mechanisms.
It is 802.1q , please have a look at the frame formats for FCoE and FIP in the FC-BB-5 standard page 126.
http://www.t11.org/ftp/t11/pub/fc/bb-5/09-056v5.pdf
At this moment, there is no interoperability with the third party Blade switches.
You can zone on the N5K. It's per VSAN. Zoning is distributed to all the switches connected in that VSAN. Zoning is a security feature which limits the ports which are participating in the SAN. If the Nexus 5K switch is running in NPV mode, the zoning must be distibuted from the core NPIV enabled switch.
That's correct VE_ports can be configured between two switches using a single port-channel or multiple individidual links. To keep the SAN A and SAN B isolation, we have to carry FCoE and LAN traffic with different port-channels.
There are many flavors of Nexus 2Ks (FEX). Only Nexus 2232PP supports FCoE on 10GE host ports with NX-OS 4.2(1)N1 and higher releases.
As of now we dont have any certification for FCoE with Cisco or any other vendor. These certifications will be launched very soon.
Live Webcast Video
https://supportforums.cisco.com/videos/2023
Unified Fabric
http://www.cisco.com/en/US/netsol/ns945/index.html
Fibre Channel over Ethernet (FCoE)
http://www.cisco.com/go/fcoe
IEEE 802.1 Data Center Bridging – White Papers
http://www.cisco.com/en/US/netsol/ns783/index.html
http://www.fcoe.com
FC-BB-5 Standard
http://www.fcoe.com/09-056v5.pdf
“I/O Consolidation in the Data Center” by Silvano Gai, Claudio DeSanti
http://www.ciscopress.com/bookstore/product.asp?isbn=158705888X
Nexus 5000 Troubleshooting Guide
http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/N5K_Troubleshooting_Guide.html
With buffer to buffer credit increase for Nexus 5500 from 64 to 240, will maximum distance increase from 3km to any higher number?
Which exactly?
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: