This session provides an opportunity to learn and ask questions about Cisco Nexus 3000 Series Switches hardware architecture and their role in Data Center / Enterprise switching design, configuration and troubleshooting.
Cisco Nexus 3000 Series Switches extend Cisco Unified Fabric to the data center. Highly programmable, these low-latency switches offer simplified management, enhanced network visibility, advanced monitoring features, and wire-speed Layer 2 and 3 switching for data center top-of-rack (ToR) deployments. These highly programmable, cost-effective, deliver flexible port densities and low power consumption for data centers.
Ask questions from Monday, September 21 to Friday, October 2nd, 2015
Vikash Kumar is a Customer support engineer in High-Touch Technical Services (HTTS) team supporting Data center Products. His areas of expertise include Cisco Nexus 9000, Nexus 7000/7700 , Nexus 5000, Nexus 3000, Nexus 2000, UCS, and MDS SAN Switches. Kumar has over 6 years of industry experience working with large enterprise and Service Provider networks. He has delivered several internal and external trainings on Datacenter technologies and created external documents on Datacenter products and technologies. He holds Bachelors in Information Technologies and PGDM in Cyber security from IMT Ghaziabad, India and these Cisco certifications: CCNA, CCNP (Routing and Swtiching), CCNP (security) and he is triple CCIE (#27857) (in R&S, Security and Datacenter). He is pursuing for CCDE
** Ratings Encourage Participation! **
Please be sure to rate the Answers to Questions
We are receiving following UDLD empty echo error message on one of the Nexus 3000 switch connected to core Nexus 7000.
%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet1/48 is down (Error disabled. Reason:UDLD empty echo)
Can you kindly explain what it means and how to stop receiving it?
When this error is detected, the cause could at either side of the peer port, let say we are having topology as following:
- Echo Packet from A to B has “My Switch-ID A, My Port-ID e x/y”, But when N7k sends the echo-reply back, it is expected to have “My Switch-ID of B, My Port-ID e w/z” AND “Your Switch-ID A, Your Port-ID e x/y”
- If for some reason packets from N3k did not received by N7k, it will not have the switch A(N3k) info. So when B (N7k) sends the echo-reply back, the echo-reply packet will have only “My Switch-ID of B , My Port-ID e w/z”.
After receiving such “Empty-echo” packets continuously, switch A will declare a “Empty-echo” error is detected.
What is wrong here?: The cause could be:
- Packet did not sent out from e w/z of switch-B,
- Packet arrived at e x/y of switch-A did not get to UDLD (CPU).
First look at Switch-B(N7k):
-- "show interface e w/z counters” will indicate if there is any packet sent at port (do a clear counter first will make it easier to ser any increase on packet count).
-- "show system internal pktmgr interface e w/z” will show if there is any packet passing thru the pktmgr to UDLD from port e w/z
Note:- To further debug the packet flow between Sup (pktmgr) and Module, elam on N7k can be used to capture the UDLD packet at Sup and module level
-- "show udld internal event-history msgs | grep –A 3 –B 3 L2_RX_DATA” will show if there is any packet arrived at UDLD from pktmgr. If above steps show no packets coming at port e w/z, then the issue may be at N3k switch-A side
Next look at Switch-A: (N3k):
Repeat the same steps as with switch-B, with following commands:
-- "show interface e x/y counters”
-- "show system internal pktmgr interface e x/y”
-- "debug logfile udld-log” to log into a file with “debug udld trace”, from the log file check if there is any msg as follow to show the packet is sent to the port:
udld_send_update(3610): Dst Port: Ethernet x/y [0xaabbcddee]
-- These steps will pin point where the UDLD packet is lost, i.e., did it get to the pktmgr? If it got to the pktmgr, did it get to the module? Did the module sent it out?
Should you be having any further query in this regards, please feel free to revert back.
we are planning to upgrade our nexus N3K-C3064PQ-10GX switches to 5.0(3)U5(1a), however in the compatibility check it is showing upgrade is disruptive?There is any possibility go with non disruptive?
Cisco N3548 provides normal port latency is 250ns
Can you explain how much latency of UCS Mini FI6324 is? I mean the ASIC chipset on FI6324.
If UCS mini is using the same ASIC with N3548, that will be great for HFT or Algo trading
UCS Mini has Malibu ASIC which is based on the Monticello-CR (MTC-CR) ASIC same as on Nexus 3548/3524. And since it is the same Monticello-CR chip with just less ports hence latency is also same (~250ns) like N3500.
What are the advantages of Nexus3000 over N5500/5600 and in which scenario/Layer should a customer prefer putting N3K in his DataCenter. Also N3k works in which mode i.e Cut-through or Store and Forward ?
Thanks & Regards,
Both N3k and N5k series switch are designed for low latency, top-of-rack(TOR) / Access layer deployment. And both switch equipped with Cut-through switching mechanism. However following are few key difference in both generations of the switch, on which base you can choose the best switch for your design requirement.
Nexus 3K has comprehensive Layer-2 and Layer-3 functionality along with rich programmability feature-set for traditional L2/L3 deployments , lower per port power consumption with improved latency of 480ns and increased buffering and table sizes(Multicast,Unicast).
Where N5k have feature/functionality, which are not available in the Nexus 3000 series switches, such as:
The Nexus 5000 offers the following unique features: FEX support, Unified Port, Unified Fabric - FCoE, FabricPath, Adapter FEX, VM FEX. Further, the Nexus 5000 offers a flexible port configuration from 32 to 96 ports in 16 port increments.
Nexus 3000/3500 for competitive data center switching opportunities that require a smaller form factor 10GE purpose-built, line-rate switch, but do not need advanced features such as FEX, FCoE and FabricPath, Dynamic Fabric Automation (DFA).
Apart from the cost factor, following are some hardware and software features and capacity difference:
|L2 Switching Capacity||960Gbs||176Gbps||1.92Tbps|
|L3 Switching Capacity||960Gbs line rate||48x1GE + 4x10GE line rate||16x10GE line rate|
|L2 Throughput||720 Mpps||131 Mpps||1428 Mpps|
|L3 Throughput||720 Mpps||131 Mpps||240 Mpps|
|Power (Typical/Max)||152W/265W||143W/267W||390W/600W (N5548)|
|Route Table||24K||Up to 16K||8K|
|Multicast Routes||8K||Up to 8K||Up to 4K|
|MAC Unicast Table||64K||128K||25K|
|MAC Multicast Table||8K||8K||4K|
|Buffer||18MB Shared||9MB Shared||640KB/port|
|Logical VLAN ports||9000||9000||48K (H+)|
|Ether Channel||24 ports||16 ports||16 ports|
Hope this is helpful and answers your query, let me know if you need any further information in this regards.
I hope you will help me for this issue.
I am trying to upgrade a nexus switch but it is throwing this error.
#install all system bootflash:n3500-uk22.214.171.124.A3.0.674.bin kickstart bootflash:n3500-uk9-kickstart.6.0.2.A3.0.674.bin
Installer is forced disruptive
Pre-upgrade check failed. Return code 0x40930062 (free space in the filesystem is below threshold).
i am suspecting that your switch /var/tmp directory is not having enough space, which is causing upgrade failed.
please use following command and check the status of "/var" directory.
HTTS-DCN-N3k# show system internal flash
Mount-on 1K-blocks Used Available Use% Filesystem
/ 89243 67554 21689 76 /dev/root
/proc 0 0 0 0 proc
/post 2048 4 2044 1 none
/var 89243 67554 21689 76 none <<<<<
/sys 0 0 0 0 none
/debugfs 0 0 0 0 nodev
/isan 1536000 593280 942720 39 none
/nxos/tmp 40960 36 40924 1 none
/var/tmp 409600 399100 10500 93 none
/var/sysmgr 921600 60772 860828 7 none
/var/sysmgr/ftp 307200 72 307128 1 none
/var/sysmgr/ftp/cores 102400 0 102400 0 none
/dev/shm 409600 257412 152188 63 none
/volatile 102400 0 102400 0 none
/debug 20480 0 20480 0 none
/dev/mqueue 0 0 0 0 none
As you can see from the logs above "/var/tmp" folder is almost full. To troubleshoot and fix the issue further we need to check the files under "/var/tmp" directory and need to delete the files, larger in size, as following:
HTTS-DCN-N3k# show system internal dir /var/tmp
HTTS-DCN-N3k# filesys delete /var/tmp/vdc.log
Please Wait.File is being deleted.
Successfully deleted the file.
If you fails to delete unnecessary files from /var/tmp directory, please reache out to cisco TAC so that CSE can load debug plugin file on the switch and can help you deleting unnecessary files from linux kernel shell.
For further information please refer following software defect:
CSCuh69073 Pre-upgrade check failed. Return code 0x40930062 (free space in the file
CSCuo00926 SNMP get operations causing /var/tmp/vdc.log to grow too large
should you be having any further query in this regards, please feel free to revert back.
Hi SBT team,
Both N3k and N9k use same Trident 2 ASIC and best for low latency data-center environment. Where N3k is designed for top-of-rack(TOR) implementation and best match for 10G / 40G line rate switching requirement. At the same time N9k is designed for leaf or spine (distribution and core) layer implementation and offers more port density, redundancy, high availability, more throughput and less power consumption, alone with rich features like VXLAN, ACI , FEX, single binary image, patch upgrade etc.
If your requirement is to provide high speed with ultra low latency to the traffic in DMZ, Nexus 3100 or 3500 cut-through switches is the good choice. which offers 960 Gbs L2/L3 line-rate speed with less power consumption.