Ask the Expert: Cisco Unified Computing System (UCS) Blades

ciscomoderator · ‎11-10-2015

This session provides an opportunity to learn and ask questions about Cisco Unified Computing System in general and the blades in more specific. Sivakumar will cover questions on design, configuration and troubleshooting.

The Cisco Unified Computing System™ is a next-generation data center platform that unites compute, network, storage access, and virtualization into a cohesive system designed to reduce total cost of ownership (TCO) and increase business agility. The system is an integrated, scalable, multichassis platform in which all resources participate in a unified management domain. Managed as a single system whether it has one server or hundreds of servers with thousands of virtual machines, the Cisco Unified Computing System decouples scale from complexity.

Ask questions from Monday, November 30 to Friday December 11, 2015

Featured Experts

Sivakumar Sukumar is an experienced Customer support engineer in High-Touch Technical Services (HTTS) team supporting Data center Products. His areas of expertise include Cisco UCS, UCS central, Cisco Nexus 9000, Nexus 5000, Nexus 3000, Nexus 2000, MDS SAN switches, Cisco Application Control Engine (ACE), Cisco Wide Area Application Services (WAAS), Cisco Content Switching Module, Cisco Content Services Switches, and other content products. He has been with Cisco for more than 6 years, working with major customers to help resolve their issues related to content products. Prior to Cisco, he worked with IBM and other service providers managing their networks. Sivakumar holds a bachelors in Information Technologies from SSNCE Anna University Chennai India, and these Cisco certifications: CCIE (#44288) (in Datacenter), CCNA, CCNP, RHCE and VCP.

Sivakumar might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community

Find other https://supportforums.cisco.com/expert-corner/events.

** Ratings Encourage Participation! **
Please be sure to rate the Answers to Questions

kthned · ‎11-30-2015

Hi Sivakumar

Can you suggest some good start up documentations for UCS for some one coming from router/Switch/Nexus world and want to pursue for expert level certification in DC. Is it correct that one can install an emulator directly on his machine for learning ?

thanks in advance

//Umair

sivaksiv · ‎12-09-2015

Hi Syed,

Glad to help you here. Please find the below links which are useful for DC.

CCIE Data Center

CCIE Data Center Online Resources

CCIE Data Center Reading List

HTH

Siva

Sasikumar Gunasekaran · ‎12-03-2015

Hi Siva,

We are having UCS Chassis 5108 model in a production envirionment. Recently we had IO module failure and cisco RMA the part . Based on the error code cisco TAC engineer said do not have the required hardware fixes. So we need to RMA the part.

I am trying to understand is there a know bug with the Cisco IO Module 2104XP ?

sivaksiv · ‎12-03-2015

Hi Sasi,

Thanks for your question.

We have few bugs around HW failure on IOM's and replace sometimes to have new revised hardware version. I would be able to provide more information if you have the error code as this may be related to environmental or ASIC failure.

CSCuc94895 2104 IOM reboots randomly with reset status register 0x00000000

HTH

Siva

Walter Dey · ‎12-03-2015

Siva

Is this issue 2104 specific: or are there any similar problems with 2204/2208 ?

Does this also imply, that one chassis is a single point of failure, and therefore one chassis implementation should be avoided ?

Thanks

Walter.

sivaksiv · ‎12-04-2015

Hi Walter,

This is specific to 2104. When you have a chassis normally has 2 paths each via both IOM. If one goes down you have single path from that specific chassis to the outside network.

HTH

Siva

Walter Dey · ‎12-04-2015

Hi Siva

Yes I know of course; however I have seen cases where a whole chassis died, due to IOM failures, taking down both of them.

This was well known caveat ! can you please clarify.

Walter.

sivaksiv · ‎12-07-2015

Hi Walter,

There are couple of bugs which I know could have caused the problem. We have several fixes went in the later release where these issues got resolved.

HTH

Siva

Walter Dey · ‎12-07-2015

Hi Siva

Not related to IOM, but both FI failing.

Just had a case, where both FI (6248 FW-Release 2.2(3d), rebooted at the same time (after 300 days up time); reboot reason: fwm hap reset !
I think this must be a known defect ! and meanwhile fixed ! can you please confirm.
Thanks Walter.

sivaksiv · ‎12-09-2015

Hi Walter,

Could be the leap second bug looking at the version on UCS

Regards,
Siva

Sarah Staker · ‎12-03-2015

Hello Sivakumar,

I have some latency issues on some application running on UCS. How can I check UCS for any packet drops or errors on the interface? Is there a guide to check?

Thank you

Sarah

sivaksiv · ‎12-04-2015

Hi Sarah,

Thanks for your question.

In UCS B environment, many components are described as many different type of virtual

component such as vnic, vnmic,vif, vethernet, virtual circuit, border port, uplink port, server
port, virtual cable, physical cable. It is not surprising that there can be confusion about what
path packets are actually taking through the UCS infrastructure.

However, knowing the full data path through the UCS infrastructure is very critical to understand where to troubleshoot.

Users will often complain of high latency or dropped packets on their hosts/VM's. It can affect a single host or multiple hosts, depending on the traffic pattern.

You should also understand and collect following information and it will also be helpful if you want assistance from Cisco TAC:

If there are virtual systems involved in the tests on both sides, are these on UCS blades on both sites or just the one site?
A topology for the tests you ran. Include and & all devices in the path from source to destination. The difference in behavior is likely going to involve different devices and/or paths.
Gather a UCSM show tech and upload it to your case. Check your QoS settings.
Were the ESX VIC enic drivers on your UCS ESX hosts ever updated as per cisco recommendations?

Interoperability matrix

Have you looked for drops throughout the path from UCS source vNIC to egress UCS for any drops or Pauses? The commands would help you get there.

The first step is to identify the affected blades.

The following SAM CLI command provides complete path info for server NIC / vHBA

show service-profile circuit server X/Y

There are other commands which does not completely trace the path.

connect nxos
show pinning border-interfaces
show pinning server-interfaces

For example, in a Virtualised environment, we may see ping packet loss to VM's and also the Host vmk/management interfaces.

We can then connect to the IOM in the chassis housing the affected blades:

UCS-6296-A# connect iom 1
Attaching to FEX 1 ...
fex-1#

From the IOM, there are several useful commands.

1) Identify backplane (HIF) and Network Interfaces (NIF) experiencing packet loss:

(Woodside for Gen2 IOM, Redwood for Gen1 IOM)

fex-1# show platform software woodside loss
fex-1# show platform software redwood loss

This will show any/all interfaces that have loss or pause frames seen. From there, we wcan drill down to specific interfaces.

Troubleshooting guide

HTH

Siva

vsri · ‎12-04-2015

Hello SivaKumar,

What are the work loads we can use for stressing UCS Blades. Could you provide some

Thanks

Srini

sivaksiv · ‎12-04-2015

Hi Srini,

Thanks for your question.

We have cisco diagnosctic tool was designed as a multipurpose tool for troubleshooting, stress testing, and burning in new equipment before entering

Cisco UCS Blade Server Diagnostics tool for Cisco UCS Blade Servers enables you to verify the health of the hardware components on your servers. The diagnostics tool provides a variety of tests to exercise and stress the various hardware subsystems on the Cisco UCS Blade Servers, such as memory and CPU. You can use the tool to run a sanity check on the state of your Cisco UCS Blade Servers after you fix or replace a hardware component. You can also use this tool to run comprehensive burn-in tests before you deploy a new Cisco UCS Blade Server in your production environment.

UCS diagnostic tool

There are other multiple 3rd party tools available too aid in performance testing, benchmarking and help with isolation of possible network, fabric, I/O performance issues.

IOMeter is a common tool that I use, which is a free open source tool to test disk performance. Moreover, you can measure the I/O transfer rate and the average I/O response time. IOMeter can be used with Windows 2003 and 2008.

Another tool, IORate developed by EMC, which also helps measure performance.

One more tool - IPerf

Iperf was developed by NLANR/DAST as a modern alternative for measuring maximum TCP and UDP bandwidth performance. Iperf allows the tuning of various parameters and UDP characteristics. Iperf reports bandwidth, delay jitter, datagram loss.

As always we recommend try booting the blades up with a required OS and have it run test loads
before migrating production workloads as secondary test.

HTH

Siva