Ask the Expert: Cisco Nexus 3000 Series Switches

ciscomoderator · ‎09-21-2015

This session provides an opportunity to learn and ask questions about Cisco Nexus 3000 Series Switches hardware architecture and their role in Data Center / Enterprise switching design, configuration and troubleshooting.

Cisco Nexus 3000 Series Switches extend Cisco Unified Fabric to the data center. Highly programmable, these low-latency switches offer simplified management, enhanced network visibility, advanced monitoring features, and wire-speed Layer 2 and 3 switching for data center top-of-rack (ToR) deployments. These highly programmable, cost-effective, deliver flexible port densities and low power consumption for data centers.

Ask questions from Monday, September 21 to Friday, October 2nd, 2015

Featured Experts

Vikash Kumar is a Customer support engineer in High-Touch Technical Services (HTTS) team supporting Data center Products. His areas of expertise include Cisco Nexus 9000, Nexus 7000/7700 , Nexus 5000, Nexus 3000, Nexus 2000, UCS, and MDS SAN Switches. Kumar has over 6 years of industry experience working with large enterprise and Service Provider networks. He has delivered several internal and external trainings on Datacenter technologies and created external documents on Datacenter products and technologies. He holds Bachelors in Information Technologies and PGDM in Cyber security from IMT Ghaziabad, India and these Cisco certifications: CCNA, CCNP (Routing and Swtiching), CCNP (security) and he is triple CCIE (#27857) (in R&S, Security and Datacenter). He is pursuing for CCDE

Find other https://supportforums.cisco.com/expert-corner/events.

** Ratings Encourage Participation! **
Please be sure to rate the Answers to Questions

Sarah Staker · ‎09-22-2015

Hello Vikash,

We are receiving following UDLD empty echo error message on one of the Nexus 3000 switch connected to core Nexus 7000.
%ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet1/48 is down (Error disabled. Reason:UDLD empty echo)

Can you kindly explain what it means and how to stop receiving it?

Thank you

Sarah

Vikash Kumar · ‎09-22-2015

Hi Sarah,

When this error is detected, the cause could at either side of the peer port, let say we are having topology as following:

Switch-A(N3k)—Ethx/y————————————————————Ethw/z—Switch-B(N7k)

Empty-Echo condition:
- Echo Packet from A to B has “My Switch-ID A, My Port-ID e x/y”, But when N7k sends the echo-reply back, it is expected to have “My Switch-ID of B, My Port-ID e w/z” AND “Your Switch-ID A, Your Port-ID e x/y”
- If for some reason packets from N3k did not received by N7k, it will not have the switch A(N3k) info. So when B (N7k) sends the echo-reply back, the echo-reply packet will have only “My Switch-ID of B , My Port-ID e w/z”.
After receiving such “Empty-echo” packets continuously, switch A will declare a “Empty-echo” error is detected.

What is wrong here?: The cause could be:

- Packet did not sent out from e w/z of switch-B,
- Packet arrived at e x/y of switch-A did not get to UDLD (CPU).

Troubleshooting Steps:

First look at Switch-B(N7k):

-- "show interface e w/z counters” will indicate if there is any packet sent at port (do a clear counter first will make it easier to ser any increase on packet count).
-- "show system internal pktmgr interface e w/z” will show if there is any packet passing thru the pktmgr to UDLD from port e w/z

Note:- To further debug the packet flow between Sup (pktmgr) and Module, elam on N7k can be used to capture the UDLD packet at Sup and module level

-- "show udld internal event-history msgs | grep –A 3 –B 3 L2_RX_DATA” will show if there is any packet arrived at UDLD from pktmgr. If above steps show no packets coming at port e w/z, then the issue may be at N3k switch-A side

Next look at Switch-A: (N3k):

Repeat the same steps as with switch-B, with following commands:
-- "show interface e x/y counters”
-- "show system internal pktmgr interface e x/y”
-- "debug logfile udld-log” to log into a file with “debug udld trace”, from the log file check if there is any msg as follow to show the packet is sent to the port:

udld_send_update(3610): Dst Port: Ethernet x/y [0xaabbcddee]

-- These steps will pin point where the UDLD packet is lost, i.e., did it get to the pktmgr? If it got to the pktmgr, did it get to the module? Did the module sent it out?

Should you be having any further query in this regards, please feel free to revert back.

Regards,
Vikash

vedprakash0802 · ‎09-24-2015

we are planning to upgrade our nexus N3K-C3064PQ-10GX switches to 5.0(3)U5(1a), however in the compatibility check it is showing upgrade is disruptive?There is any possibility go with non disruptive?

Vikash Kumar · ‎09-25-2015

Hi Vedprakash,

non-disruptive upgrade (ISSU) is not supported on 5.0(3)U5(1a).

regards,
Vikash

james-yang · ‎09-28-2015

Hi Vikash,

Cisco N3548 provides normal port latency is 250ns

Can you explain how much latency of UCS Mini FI6324 is? I mean the ASIC chipset on FI6324.

If UCS mini is using the same ASIC with N3548, that will be great for HFT or Algo trading

Thanks

James

Vikash Kumar · ‎09-29-2015

Hi James,

UCS Mini has Malibu ASIC which is based on the Monticello-CR (MTC-CR) ASIC same as on Nexus 3548/3524. And since it is the same Monticello-CR chip with just less ports hence latency is also same (~250ns) like N3500.

regards,
Vikash kumar

Vikash Kumar · ‎09-25-2015

Hi Vedprakash,

non-disruptive upgrade (ISSU) is not supported on 5.0(3)U5(1a).

regards,
Vikash

Sandeep Maurya · ‎09-25-2015

Hi Vikash,

What are the advantages of Nexus3000 over N5500/5600 and in which scenario/Layer should a customer prefer putting N3K in his DataCenter. Also N3k works in which mode i.e Cut-through or Store and Forward ?

Thanks & Regards,

Sandeep Maurya.

Vikash Kumar · ‎09-25-2015

Hi Sandeep,

Both N3k and N5k series switch are designed for low latency, top-of-rack(TOR) / Access layer deployment. And both switch equipped with Cut-through switching mechanism. However following are few key difference in both generations of the switch, on which base you can choose the best switch for your design requirement.

Nexus 3K has comprehensive Layer-2 and Layer-3 functionality along with rich programmability feature-set for traditional L2/L3 deployments , lower per port power consumption with improved latency of 480ns and increased buffering and table sizes(Multicast,Unicast).

Where N5k have feature/functionality, which are not available in the Nexus 3000 series switches, such as:

The Nexus 5000 offers the following unique features: FEX support, Unified Port, Unified Fabric - FCoE, FabricPath, Adapter FEX, VM FEX. Further, the Nexus 5000 offers a flexible port configuration from 32 to 96 ports in 16 port increments.

Positioning:

Nexus 3000/3500 for competitive data center switching opportunities that require a smaller form factor 10GE purpose-built, line-rate switch, but do not need advanced features such as FEX, FCoE and FabricPath, Dynamic Fabric Automation (DFA).

Quick Comparison:
Apart from the cost factor, following are some hardware and software features and capacity difference:

	N3548	N3048	N5548/N5596
L2 Switching Capacity	960Gbs	176Gbps	1.92Tbps
L3 Switching Capacity	960Gbs line rate	48x1GE + 4x10GE line rate	16x10GE line rate
L2 Throughput	720 Mpps	131 Mpps	1428 Mpps
L3 Throughput	720 Mpps	131 Mpps	240 Mpps
Power (Typical/Max)	152W/265W	143W/267W	390W/600W (N5548)
Power (Typical/Max)	152W/265W	143W/267W	660W/882W (N5596)
Route Table	24K	Up to 16K	8K
Multicast Routes	8K	Up to 8K	Up to 4K
MAC Unicast Table	64K	128K	25K
MAC Multicast Table	8K	8K	4K
SVI	4K	1K	4K
Buffer	18MB Shared	9MB Shared	640KB/port
Logical VLAN ports	9000	9000	48K (H+)
Ether Channel	24 ports	16 ports	16 ports
SPAN Session	4	4	4
ACL Table	4K	2K	4K
Switching Mode	Cut-through	Cut-Through	Cut-Through
100Mb support	No	Yes	No

Hope this is helpful and answers your query, let me know if you need any further information in this regards.

Thanks !

Regards,
Vikash Kumar

keshav.kumar · ‎09-25-2015

Hi Vikash,

I hope you will help me for this issue.

I am trying to upgrade a nexus switch but it is throwing this error.

#install all system bootflash:n3500-uk9.6.0.2.A3.0.674.bin kickstart bootflash:n3500-uk9-kickstart.6.0.2.A3.0.674.bin
Installer is forced disruptive
Pre-upgrade check failed. Return code 0x40930062 (free space in the filesystem is below threshold).

Vikash Kumar · ‎09-25-2015

Hi Keshav,

i am suspecting that your switch /var/tmp directory is not having enough space, which is causing upgrade failed.

please use following command and check the status of "/var" directory.

example:-

HTTS-DCN-N3k# show system internal flash
Mount-on 1K-blocks Used Available Use% Filesystem

/                             89243     67554       21689     76   /dev/root
/proc                             0         0           0      0   proc
/post                          2048         4        2044      1   none
/var                          89243     67554       21689     76   none <<<<<
/sys                              0         0           0      0   none
/debugfs                          0         0           0      0   nodev
/isan                       1536000    593280      942720     39   none
/nxos/tmp                     40960        36       40924      1   none
/var/tmp                     409600     399100      10500      93   none
/var/sysmgr                  921600     60772      860828      7   none
/var/sysmgr/ftp              307200        72      307128      1   none
/var/sysmgr/ftp/cores        102400         0      102400      0   none
/dev/shm                     409600    257412      152188     63   none
/volatile                    102400         0      102400      0   none
/debug                        20480         0       20480      0   none
/dev/mqueue                       0         0           0      0   none

As you can see from the logs above "/var/tmp" folder is almost full. To troubleshoot and fix the issue further we need to check the files under "/var/tmp" directory and need to delete the files, larger in size, as following:

HTTS-DCN-N3k# show system internal dir /var/tmp
                                                                ./         800
                                                               ../        1024
                                                      etherfilter          784
                                           util_cli_history_admin            0
                                                   bootloader_ver            0
                                                     libvirtd.log          116
                                                 m2rib_ascii.3480          374
                                            _clis_parse_tree_.log         2706
                                                 sdwrap.378.378.5      6291456
                                                 sdwrap.378.378.4       524288
                                                 sdwrap.378.378.3      1048576
                                                 sdwrap.378.378.2      1048576
                                                 sdwrap.378.378.1      1048576
                                                 sdwrap.378.115.0        65536
                                                   sdwrap.378.0.0       524288
                                                  pim_restart.log            0
                                                        stp.log.1         4948
                                                           ntpd_1            0
                                                      cfs_mac.log          166
                                                     ntp_client_1            0
                                                 igmp_restart.log            0
                                                         fwm1.out            0
                                                     ufdm_log.txt          109
                                                          vdc.log        269399
                                                  security_stderr           77
                                                  security_stdout            0
                                               security_debug.log          647

                                                              vsh/          40

HTTS-DCN-N3k# filesys delete /var/tmp/vdc.log
Please Wait.File is being deleted.
Successfully deleted the file.

If you fails to delete unnecessary files from /var/tmp directory, please reache out to cisco TAC so that CSE can load debug plugin file on the switch and can help you deleting unnecessary files from linux kernel shell.

For further information please refer following software defect:

CSCuh69073 Pre-upgrade check failed. Return code 0x40930062 (free space in the file
CSCuo00926 SNMP get operations causing /var/tmp/vdc.log to grow too large

should you be having any further query in this regards, please feel free to revert back.

thanks !

regards,
Vikash kumar

susim · ‎09-28-2015

Hi vikash

How would you compare n3k and n9k .

Is it wrong choice for DMZ where only web traffic passing mainly ?

Thanks

Vikash Kumar · ‎09-28-2015

Hi SBT team,

Both N3k and N9k use same Trident 2 ASIC and best for low latency data-center environment. Where N3k is designed for top-of-rack(TOR) implementation and best match for 10G / 40G line rate switching requirement. At the same time N9k is designed for leaf or spine (distribution and core) layer implementation and offers more port density, redundancy, high availability, more throughput and less power consumption, alone with rich features like VXLAN, ACI , FEX, single binary image, patch upgrade etc.

If your requirement is to provide high speed with ultra low latency to the traffic in DMZ, Nexus 3100 or 3500 cut-through switches is the good choice. which offers 960 Gbs L2/L3 line-rate speed with less power consumption.

regards,
Vikash kumar

kp-tkr2014 · ‎09-30-2015

Hi ,

Can we build double sided vpc topology using n3k ?

Thanks