Introduction
Fibre Channel Over Ethernet (FCoE)
1. Does NX-0S 5.0 run NPIV?
Yes, it is supported in NX-OS 5.0. Nexus 5000 platform supports N port identifier virtualization (NPIV) and N-Port Virtualization(NPV) since NX-OS 4.0 version. Nexus 5K supports two modes. By default, Cisco Nexus 5000 and 5500 Series switches operate in fabric mode, in this mode switch provides standard FC switching capability and features. Second mode is NPV, in NPV mode Nexus 5K switches relays all the traffic from server-side ports oto the core switch.(NPIV enabled switch). The core switch provides F port functionality and all FC switching capabilities.
2. Is FCoE between data centers possible?
Yes, FCoE between Data Centers is possible. VE Ports allow you to build multi-hop FCoE fabrics by interconnecting multiple FCF, VE port functions as a Fibre Channel E-port running on top of a lossless Ethernet fabric. The maximum distance between two Cisco Nexus 5000 Series switches is 3000m for FCoE (lossless Ethernet) traffic with NX-OS 5.0(2)N1.1. It can be enabled using the pause no-drop buffer-size buffer-size pause-thresholdxoff-size resume-threshold xon-size command in QOS configuration. Here are the maximum distances:
Nexus 55xx to Nexus 55xx - 3km
Nexus 50x0 to Nexus 50x0 - 3km
Nexus 50x0 to Nexus 55xx - 3km
Nexus 55xx to Nexus 2232 - 300m
Nexus 50x0 to Nexus 2232 - 300m
3. Can you explain the vlan trunking channels a little bit more?
We support Virtual LAN on top of the Fibre channel Over ethernetwork. So trunking means we carry different Virtual LAN altogether in one ISL. And the same on FCOE we also support Virtual SANs and can carry multiple virtual SANs on one isl. In turns of multi-hop FCOE we will see that we could carry multiple virtual LAN and virtual SANs together.
4. Can FCoE be used with UCS?
Yes. In latest UCS 1.4 release, you can directly connect FCoE targets to the Fabric Interconnect ports. Also on UCS system we have FIP cabable (Gen2) CNAs and Pre-FIP (Gen1) CNAs running on the blades that utilizes FCoE between blades and Fabric Interconnect. If you would like to extend Fabric Interconnect connectivity with VE ports and connecting them to Nexus 5K switches in terms of running multi-hop FCoE, we don't support this on the UCS software yet.
5. What is the next data rate after 10Gig for FC? is it 16Gig? When will it be supported?
After the 10Gig model the discussion are in 12Gig and 16Gig. If you are asking Cisco will be supporting 16gig FC, yes we will be supporting it and we will have products that will be supporting 16gig FC.
6. Is there a limit on the number of hops that can be made in a multihop environment/design?
The limit is 7 actually. You can have upto 7 FCoE hops that are tested and supported by Cisco. If there is any requirement to suppoprt more than 7 hops, that can be tested by Cisco. Please contact your Cisco Account Manager if you require more than 7 hops.
7. Can we configure multiple COS values for PFC ?
Yes, if you are asking you can use different COS values for PFC. PFC uses the IEEE 802.1p CoS values in the IEEE 802.1Q VLAN tag to differentiate up to eight CoSs that can be subject to flow control independently. So on Nexus 5K,by default we have 8 lanes , 8 qos-groups. Qos 6 and Qos 7 are reserved for control plane trafic and the rest 6 qos-groups can be assigned to drop-class or no-drop-class. By default FCoE is assigned COS 3.
8. What if we want to trunk multiple data vlans to a service thats running ESXi for example do we still need to configure trunking on switch ports?
Yes if you want to carry multiple data VLANs on ESXi , we definitely need to configure trunking. Also another reason if you want to use FCoE, Fiber Channel over Ethernet Initialization Protocol (FIP) uses the native VLAN and therefore all FCoE links should be trunked to carry the FCoE VLAN as well as the native VLAN.
9. What is the difference between CEE and a DCE?
CEE is the Converged Enhanced Ethernet. The CEE authors group was an informal group of companies that submitted the initial proposals of PFC , ETS and DCBX to the IEEE DCB working group.DCE is a Cisco term that describes the architectural collection of ethernet extensions based on open standards. DCE includes support for the ethernet extensions found in EEE DCB.
10. How is the FCOE license activated on the Nexus 5000 or 5500 series?
There is an option in the Nexus 5000 called storage service license which is activated when it is shipped from Cisco. When ordered as a seperate line card item an activation key and the procedure for activating the code is delivered in an envelope. Because this is always present and only activation is needed and any Nexus 5k will automatically initiate the 120 days temporary license for a storage protcol service from the first use of FC or FCoE command.
11. Is FCOE IP Routable?
No, there is no Ip layer in FCoE. FC-BB-5 contains FCIP that is "IP-routable". FCoE is FC-routable. FCoE switches may forward FC frames across different Ethernet Clouds, FCoE switches may forward FC frames over the Internet using FCIP.
12. Do I need to enable Jumbo frames for FCOE?
Yes. You need to enable jumbo frames because the FC payload is 2,112 bytes and enabling jumbo frames will keep FCoE traffic from being fragmented. On Nexus 5K, we have a pre-defined Class of Service (CoS) to be FCoE, which is by default has the value 3. Within this CoS, the MTU size it set to 2158.
13. Can you advise on any resources for FCOE where I can read more about them?
You can refer the book “I/O Consolidation in the Data Center” by Silvano Gai, Claudio DeSanti
You can also visit the following resources:
Fibre Channel over Ethernet (FCoE)
IEEE 802.1 Data Center Bridging – White Papers
http://www.fcoe.com & FC-BB-5 Standard http://www.fcoe.com/09-056v5.pdf
Nexus 5000 Troubleshooting Guide
14. Can I use multiple classes of service for FCoE?
Yes you can use multiple classes of service for FCoE. We have 8 lanes, 8 QOS-groups. 6 and 7 are reserved for control plane traffic that is used by the system. That leaves us 6 qos-group to use. One of the 6 qos-groups is pre-defined as drop class and it is class-default(drop class) and that leaves us 5 qos-groups. In the remaining 5 qos-groups, only 4 can be configured as "no-drop" and one of these could be FCoE.
15. Did Cisco produce interoperability testing results with various vendors (SAN, CNA adapters) ?
Yes we have the interoperability matrix which we published on CCO Cisco.com.
We try to certify, different Storage, different CNAs, different O/S and different NX-OS versions. Also most of the OSM vendors publish their interoperability matrixes in their own web sites.
16. Can the concept and purpose of Worldwide name can be briefy explained?
A World Wide Name (WWN) is a unique identifier which identifies a particular Fibre Channel target, Each WWN is an 8 byte number derived from an IEEE OUI and vendor-supplied information.
- Original format: addresses are assigned to manufacturers by the IEEE standards committee, and are built into the device at build time, similar to an Ethernet MAC Address. The first 2 bytes are either hex 10:00 or 2x:xx (where the x's are vendor-specified) followed by the 3-byte vendor identifier and 3 bytes for a vendor-specified serial number
17.Which CNA types are supported on Nexus 5000 and Nexus 2232PP?
We have two flavours of Nexus 5K switches.They are Nexus 55xx and Nexus 5010/5020. On Nexus 5010/5020 we support Gen1 and Gen2 CNAs.
On Nexus 55xx and Nexus 2232PP we only support Gen2 CNAs such as QLE814x/QLE815x, Emulex One Connect...
18. What are the requirements of FCOE?
You will need a lossless ethernet network that will support FCoE, to build this network minimum requirement is to have a Fibre Channel Forwarder (FCF switch) , host/server that has CNA installed and a FC or FCoE target that is providing the access to the block storage for the server.
19. Can you explain a little background about FC and ISCSI, and the differences between FCoE and ISCSI? I've never had the need for FC, so it's a different world with "credits" for me.
FC and iSCSI share the same SCSI stack (to some extent), there are some differences in the protocol stacks. FC is generaly used for tier-1 apps so will FCoE by extension. iSCSI however is also seeing attraction in that area.
20. I would like to know the difference nowdays between ISCSI and FCoE as I have never had the need for FC in the old days.
The iSCSI protocol can be implemented in networks that are subject to packet loss, and iSCSI can run over 1 Gigabit Ethernet (1GbE). FCoE requires 10 Gigabit Ethernet (10GbE) and a lossless network with infrastructure components that properly implement pause frame requests and per-priority pause flow control (PFC) based on separate traffic classes that map to different priorities. The idea behind PFC is that during periods of congestion, high-priority traffic is allowed to continue while lower-priority traffic is paused.
Your 10GbE switches will need DCB support for FCoE, including a range of enhancements for classes of service, congestion control, and management. FCoE also requires Jumbo Frames because the FC payload is 2,112 bytes and cannot be broken up; iSCSI does not require Jumbo Frames.
21. Do you know the difference for the ethernet header between ISCSI and FCoE ? Could this be an advantage for one of the protocols?
iSCSI has a TCP overhead contributing to both the size of frame and also operational inefficiencies that come with TCP retransmissions. There maybe benefits to iSCSI in an ethernet loss-less environment as these retransmissions are eliminated.
22. Does FCoE (in general) support by all SAN vendors (for their own proprietary modes), or do their gears need to run in interop mode?
It's important to remember that FCoE *is* FC. It uses the exact same protocol services as FC. Once a FCoE switch decapsulates the FC from the Ethernet Frame, it is FC, and is handled by SAN vendors as a FC frame. See the UNH Plugfest information at http://fibrechannel.org for interoperability tests.
23. Does Cisco have any multi-hop FCoE products?
Cisco currently has multi-hop FCoE in the Nexus 5000 series of switches. Additionally, the Nexus 7000 and MDS 9500 series Director switches have announced multihop capability; watch for further information very soon.
24. Is CEE still active ?
I hope I understand your question properly. The DCB standards are built on what CEE defined originally. Cisco switches abide by both standards.
25. What is the typical uplink oversubscription on Fiber Channel switches ? If the oversubscription cannot be avoided, does it mean FCoE must be deployed with DCB together to avoid FC packets being dropped?
DCB is an absolute requirement for FCoE. The links *need* to be loss-less before it can even be considered for FC traffic as storage applications are generally intolerant of loss. Regarding O/S, every organisation is different and a value is chosen based on their tolerances. We have seen 4/1 over subscription to up to 15/1. When designing the storage network the link O/S should be considered as well as the internal switch hardware O/S however the latter is generally lower so not a bottleneck but needs to be considered. This O/S is platform specific and changes in accordance to the HW.
26. What is the aggregate encap/decap rate of FCF inside the switch, like Nexus 5K?
I assume the question is if the process of de-encap/re-encap FC-to-FCoE causes a performance hit, and that is not the case since packets are parsed and modified as they are transmitted to xbar on their way to egress regardless of protocol type.That all happens at speeds much faster then any input device can forward frames in,or receive frames out.
27. Does the default QoS setting imply I/O consolidation would apply only to servers running FCoE or below? how does it relate to the trend of server consolidation, where I/O rate will get higher and higher as more VMs are created?
The QoS setting (negotiated) sets up the adjacent neighbors so that they can have a loss-less ethernet conversation. The packets are tagged with a CoS which indicates the Virtual Lane they ough to be in. In your case where I/O rates are increasing due to VM adoption, if the host is tagging these packets as loss-less with the correct CoS marking, then the frames are transmitted with those capabilities and the switch treats them as such. However you only ever have the 10G link offered. This QoS setting doesn't affect the increasing I/O rates.
28. Based on slide #51. "Trunking is not required on the host driver - all FCoE frames are tagged by the CNA." what about if we want to trunk multiple data VLANs to the servers running ESXi for example? do we still need to configure trunking on the switch ports?
Yes, we need to configure "switchport trunk" since Fiber Channel over Ethernet Initialization Protocol (FIP) uses the native VLAN and therefore all FCoE links should be trunked to carry the FCoE VLAN as well as the native VLAN.
29. Can you explain what you meant by multihop extending beyond the access?
Up until now, you could connect FCoE from the server (using CNA) to an access switch. With Multihop FCoE as defined by FC-BB-5 working group of T11 we can extend FCoE beyound the access switch (via an ISL) to other switches and perhaps into the SAN core. You can now build Ethernet based SAN rather than a FC only SAN.
30. Is it possible carry FCoE traffic across fabricpath cloud?
At this point in time, no. Currently ethernet traffic and FCoE is handled on their own respective VLANs, which means that you can apply FabricPath to the Ethernet VLAN. FCoE uses FSPF for its forwarding mechanisms.
31. Do I understand correctly ISL, so DOT1Q trunks are a no go?
It is 802.1q , please have a look at the frame formats for FCoE and FIP in the FC-BB-5 standard page 126.
http://www.t11.org/ftp/t11/pub/fc/bb-5/09-056v5.pdf
32. How would you connect a 3rd party blade switch (FCoE enabled) to Cisco Nexus infrastructure? Do i need to do multi-hop FCoE design?
At this moment, there is no interoperability with the third party Blade switches.
33. Can you elaborate a little on how zoning is implemented in the N5K please? I understand that zoning cannot be configured directly on the N5K but has to be downloaded from a SAN switch, is this undertanding correct?
You can zone on the N5K. It's per VSAN. Zoning is distributed to all the switches connected in that VSAN. Zoning is a security feature which limits the ports which are participating in the SAN. If the Nexus 5K switch is running in NPV mode, the zoning must be distibuted from the core NPIV enabled switch.
34. Referring to slide 54, if I understand correctly, you can still have more than one link aggregated into single logical link for both FCoE and LAN... just not through the same logical link?
That's correct VE_ports can be configured between two switches using a single port-channel or multiple individidual links. To keep the SAN A and SAN B isolation, we have to carry FCoE and LAN traffic with different port-channels.
35. Is Nexus 2K supporting FCoE?
There are many flavors of Nexus 2Ks (FEX). Only Nexus 2232PP supports FCoE on 10GE host ports with NX-OS 4.2(1)N1 and higher releases.
36. Is there is any training or course certification leading to FCoE Knowledge ?
As of now we dont have any certification for FCoE with Cisco or any other vendor. These certifications will be launched very soon.
testing the issue reported by Tomas De Leon
