Introduction
Vinayak Sudame is a technical lead with the Data Center Switching Support Team in Cisco's Technical Services in RTP, North Carolina. His current responsibilities include but are not limited to troubleshooting technical support problems and escalations in the areas of Cisco Nexus 5000, Cisco Nexus 2000, and FCoE. Vinayak is also involved in developing internal and external technical Cisco content: for example, the Cisco Nexus 5000 troubleshooting guide (cisco.com), Cisco Nexus 5000 portal (partners), and so on. This involves cross-team collaboration and working with multiple different teams within Cisco. Vinayak has also contributed to training account teams and partners in customer assurance engineering (CAE) bootcamp dealing with Cisco Nexus 5000 technologies. In the past, Vinayak's responsibilities included supporting the MDS platform (Fibre Channel technologies) and work with EMC support on escalated MDS cases. He was the subject matter expert for Santap Technologies before moving to Cisco Nexus 5000 support. Vinayak holds a master's degree in electrical engineering with specialization in networking from Wichita State University, Kansas. He also holds Cisco certification CCIE (no. 20672) in routing and switching.
The following experts were helping Vinayak to answer few of the questions asked during the session: Carlos Lopez and Carlo Schmidt. Carlos and Carlo are Data Center experts and have vast knowledge in this topics..
You can download the slides of the presentation in PDF format here. The related Ask The Expert sessions is available here. The Complete Recording of this live Webcast can be accessed here.
Error Statuses
Q: With the "sfp invalid" error on the Nexus 5000 Series switches (N5Ks), is the port actually down?
A: Yes, if the small form-factor pluggable (SFP) validation fails, the port is down. This typically happens when you use a 1G SFP on a port that is configured for 10G. "sfp invalid" is the scenario where the port is down.
Q: Does the "sfpInvalid" status mean that the Cisco Fabric Extender (FEX) cannot be inserted in that port?
A: A FEX can only be used on a port that is configured for "switchport mode fex." A FEX transceiver is used for the connection between N5Ks and Nexus 2000 Series switches (N2Ks). So if you use these, only use them in order to connect to your FEX. Some vendor SFPs can create issues if they are not from Cisco.
Q: What do statuses other than "connected," "disabled," "not connected," and "sfpInvalid" represent?
A: You might also see an "err-disabled" status. There are not many statuses. Sometimes, if an interface goes down because of congestion, older switches throw Packet Over SONET (POS) errors. There are error detection methods now for link recovery. Most of the time, it ends up with an end device issue.
Q: If you have some "sfpInvalid" ports connected to a Cisco 6500, and the ports on the 6500 are up/up and Spanning Tree Protocol (STP) ran on those ports, then why would the 6500s still see the ports as up/up?
A: This depends on the configuration on the 6500. Auto Negotiation (AN) should not complete when a 1G SFP is inserted into a port configured for 10G.
Best Practices
Q: What is the best practice in order to deploy a single-homed device that is connected to a N5K, for example? Should you use a separate VLAN for non-Virtual Port Channel (non-vPC) peer link VLANs?
A: Yes. It is recommended that you run the VLAN only between the two N5Ks. It should be a separate VLAN, and the VLAN should be carried over a separate link. It could be through a management port or a physical 1G or 10G port.
Q: Should you use a dedicated link between two switches of the same domain in order to carry the single-homed devices VLANs, as opposed to the use of the VPC peer link in order to transport these VLANs ?
A: The peer link can be used in order to carry this traffic, and there is no need for a separate link unless you peer routing protocols. It is encouraged to trunk all VLANs over the peer link if you have routed VLANs.
Q: Can you use a separate interface for the VLANs connected to single-homed devices?
A: You can use a vPC VLAN. In most configurations, the Layer 3 (L3) gateway is connected via a vPC. Due to this, it is required that you use a vPC VLAN.
Caveats and Bugs
Q: Is it possible to subscribe to automatic bug reports for a product instead of manually searching for them?
A: Besides the Bug Search Tool, you can use the notification tool, http://www.cisco.com/cisco/support/notifications.html.The notification tool might be able to help you.
Q: Is there a location where you can find the latest up-to-date information in regards to Nexus caveats and limitations - something like Virtual Device Context (VDC) compatibility for M with F2 line cards or the incompatibilities depending on the parent switch for N2K?
A: The release notes and the configuration guide are the best places to start. If a particular issue appears frequently, Cisco often releases "Quick Configuration guides" for these specific problems. This is not done for everything, though. N6K and N2K do not have M2 or F2 cards.
Q: Sometimes you have to browse and read through a lot of documentation, some of which is not updated, in order to find information online. It takes too long to read the release notes. Can you make a summarized version of the most common caveats for the Nexus series?
A: The Configuration Limits for the Cisco Nexus Operating System (NX-OS) has the limitations. The Release Notes have caveats.
Q: In what version was the adjacent route fixed?
A: Cisco bug ID CSCui74708 Nexus 6000: Loses Static route in hardware after reload. This issue is fixed in Version 6.0(2)N2(2) and later.
Q: In what version was Cisco bug ID CSCuc84658 fixed?
A: Cisco bug ID CSCuc84658 – Nexus 5000: Incorrect Adjacency for Next Hop. This was fixed in Version 6.0(2)N1(1), Version 5.2(1)N1(3), and later versions.
Q: What is the solution for the Storage Area Network (SAN) port channel?
A: Cisco bug ID CSCta97375 GLDN:SABRE:Sabre oui is not recognized as cisco oui 547FEE. Cisco bug ID CSCtr01652 Add New OUIs to liboui's list of recognized Cisco OUIs. Cisco bug ID CSCty04686 Add MAC OUI "002a6a", "8c604f", "00defb" for 5k/UCS-FI.
Miscellaneous
Q: How do you download the software for the Data Center Network Manager (DCNM) after you get a license for it?
A: You can download the software from the Cisco.com download software page. The DCNM download is under Cisco.com in the support section.
Q: Are Nexus ports autosensing? Does this mean 1/10 G automatically?
A: No, the Nexus 3000 Series switches (N3Ks) and N5Ks must be manually configured for 1G if a 1G SFP is used. T ports (copper) are autosensing. The 10G speed is autosensing, but the 1G speed must be manually configured.
Q: Is the solarwinds issue only specific to dual-homed N2Ks, or is it also possible with the single-homed N2K?
A: This issue should not be limited to dual-homed FEXs, because it occurs on single-homed FEXs as well. The issue is possible with single-homed or dual-homed.
Q: If you use Extended Range (ER) or Long Range (LR) transceivers, can you support Fibre Channel over Ethernet (FCoE) over extended ranges, such as seven miles in the distance?
A: It is not yet possible.
Q: Is it advisable to run NX-OS Version 6.0(2)N2(2) on a 5596UP?
A: Yes, you can.
Q: How much heavy-loaded ESX do you recommend to route over a single FCoE link?
A: Cisco supports a 10G environment, so you should be able to go anywhere close to the line rate. During vMotion, it could boil down to how many simultaneous vMotions can be done. It is basically limited as per the resources on hosts and not from the network perspective.
Q: What is the Ethernet Ethertype for FCoE Initialization Protocol (FIP)?
A: It is 0x8906.
Q: FCoE is encapsulated over Ethernet with which Ethertype?
A: The ethertype is 0x8914.
Q: What speed is required to run FCoE?
A: 1G is the required speed.
Q: Is the Etherlink N_Port ID Virtualization (NPIV) feature disruptive or nondisruptive?
A: The NPIV feature is nondisruptive, but the Node Port Virtualization (NPV) feature is disruptive.
Q: Does NPV require the NPIV core switch?
A: Yes, it does require an NPIV core switch.
Q: Is local switching supported for the NPV switch?
A: No, switching is primarily done on the upstream NPIV switch. There is no local switching.
Related Information
- Cisco Nexus 5000 Series Switches
- Cisco Nexus 2000 Series Fabric Extenders
- Datacenter Switches
- Fibre Channel over Ethernet
- Bug Search Tool
- Nexus 5000, 6000 and Fibre Channel over Ethernet Important Caveats and Best Practices - Video from live webcast
- Ask the Expert Event
- Cisco Nexus 5000, 6000 and Fibre Channel over Ethernet Important Caveats and Best Practices - Slides from Live Webcast
