Re: ASK THE EXPERTS - Cisco Unified Computing Systems

ciscomoderator · ‎04-08-2011

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn how to configure and troubleshoot UCS in various environments with Cisco expert Stephen McCabe. Stephen is a customer support engineer in Cisco’s Advanced Services group providing High Touch Technical Support. As part of the Data Center Networking team, he supports customers on the Cisco Unified Computing System and Cisco Nexus platforms. In addition, he serves as an escalations engineer for Cisco's Application Networking Services team supporting server/application load balancing, WAN acceleration, web caching proxy, and video and streaming media. Stephen holds a bachelor's degree in computer engineering from Ohio University. He also holds CCIE #23917 certification in Security and is a Red Hat Certified Engineer, and is VMWare Certified Professional #71728.

Remember to use the rating system to let Stephen know if you have received an adequate response.

Stephen might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Unified Computing discussion forum shortly after the event. This event lasts through April 22, 2011. Visit this forum often to view responses to your questions and the questions of other community members.

stmccabe · ‎04-11-2011

To kick the conversation off regarding UCS troubleshooting, I'm posting a "how-to" that I've found very useful when working with customers on UCS B-Series servers. There is an interest and benefit in understanding how to trace a specific host/guest os's path through the UCS system.

Here is a high-level method that I use via the UCS CLI that a customer can use to identify the path of the packet given either a MAC address from a host and/or, VM behind a host, or the location of the server, that is, chassis number and slot number..

I'm going to use an example here where I have the mac address of a VMWare host with a mac address of 0050.56a6.076a

1) SSH to the UCSM IP address

2) Assuming that you do not know which fabric is receiving frames from the given MAC address, check both Fabric interconnects for the mac address.


6100-A-B# connect nxos a
6100-A-A(nxos)# show mac-address address 0050.56a6.076a
VLAN      MAC Address       Type    Age       Port                          
---------+-----------------+-------+---------+------------------------------
503       0050.56a6.076a    dynamic 10        veth928  

#Found mac address in Fabric interconnect A. It should not be visible on Fabric interconnect B.  If it is then the customer is doing 
per flow/packet load balancing at the host level, which is not allowed on UCS B-Series

6100-A-A(nxos)# exit


#Checking fabric interconnect B for the mac address. MAC address is not present here, which is what we expect to see.

6100-A-B# connect nxos b
6100-A-B(nxos)# show mac-address address 0050.56a6.076a
Total MAC Addresses: 0 

3) Now identify blade location. In this example, I logged back Fabric Interconnect A.


 UCS-6100-A-A(nxos)# show int veth928
 vethernet928 is up
   Bound Interface is Ethernet1/1/1 
 Hardware: VEthernet
 Encapsulation ARPA
 Port mode is trunk
 Last link flapped 1week(s) 5day(s)
 Last clearing of "show interface" counters never
 1 interface resets

Interpreting Ethernet1/1/1 you get Chassis 1 and slot 1. Ignore the remote entity field.

Ethernet	1 / \|	1 / \|	1
	chassis no.	remote entity	slot no

Now that you have the chassis and slot number, you can begin the data path troubleshooting.

Given a server location (chassis/slot no)

If given a server location, it is good to verify that the information is correct. Log into the affected server and get the mac address.
Using the steps above, identify that the mac address is associated with the blade mentioned.

Summary Diagram

The diagram below summarizes the commands used to figure out the path of the packet through the UCS using this example.

Fabric Interconnect

There are 2 ports on the Fabric interconnect to identify.

Network Port: This connects to the upstream switch.
Server Port: This connects to the IOM

Identifying Network Port

The network port can be obtained by checking the pinning for the vethernet associated to the blade

6100-A-A(nxos)# show interface | include veth|1\/1\/1
vethernet928 is up
   Bound Interface is Ethernet1/1/1
vethernet1048 is up
vethernet1052 is down (vicEnNotRcvd)
vethernet1053 is down (vicEnNotRcvd)
vethernet9122 is up
   Bound Interface is Ethernet1/1/1 
vethernet9241 is up
vethernet9246 is down (vicEnNotRcvd)
   Bound interface is vethernet9122
   Bound interface is vethernet9241
   Bound interface is vethernet9246
Ethernet1/1/1 is up
6100-A-A(nxos)# show pinning interface veth928 

---------------+-----------------+------------------------+-----------------
SIF Interface   Sticky            Pinned Border Interface  Pinned Duration  
---------------+-----------------+------------------------+-----------------
veth928         No                Eth1/8                   5d 39:13:43

The Fabric interconnect network facing port is Eth1/8

Identifying Server Port

find the associated Server port to this blade issue the following command. The port number used in the command, can be found in the troubleshooting used earlier on this guide.

6100-A-A(nxos)# show platform software fex info satport ethernet 1/1/1
 Interface-Name  ifindex  State Fabric-if  Pri-fabric Expl-Pinned
        Eth1/1/1 0x1f000000   Up    Eth1/1    Eth1/4    Eth1/1
 Port Phy Up. Port dn req: Not pending
 SDB entry: ifindex(1f000000) fabric if(1a000000)
    Dev: 0 Nif3 Hif7 (Nif:0x20000000 Hif:0x1f000000)

The Fabric interconnect port connected to Chassis 1/Blade 1 is Eth1/1.

IOM

IOM has Fabric interconnect facing ports, 4 in total, and 8 blade server facing ports (internal).

Identify Fabric interconnect Facing Port and NIF number

Using the example, identify the IOM uplink port that connects to Fabric Interconnect port Eth1/1

6100-A-A(nxos)# show interface fex-fabric 
     Fabric      Fabric       Fex                FEX           
Fex  Port      Port State    Uplink    Model         Serial    
---------------------------------------------------------------
 1    Eth1/1        Active     1         N20-C6508  QCI132800P2
 1    Eth1/2        Active     2         N20-C6508  QCI132800P2
 1    Eth1/3        Active     3         N20-C6508  QCI132800P2
 1    Eth1/4        Active     4         N20-C6508  QCI132800P2

The IOM's Fabric interconnect facing interface is 1. I noticed that this physical position doesn't correspond the NIF (network interface) number shown in the next command. It is NIF3.

Identify Host facing Port

Using the same command, used earlier to find the fabric interconnect server port, the IOM NIF (network interface) and IOM HIF(host interface) numbers associated with the blade can be identified. The HIF points to the server, the NIF points to the fabric interconnect.

6100-A-A(nxos)# show platform software fex info satport ethernet 1/1/1
 Interface-Name  ifindex  State Fabric-if  Pri-fabric Expl-Pinned
        Eth1/1/1 0x1f000000   Up    Eth1/1    Eth1/4    Eth1/1
 Port Phy Up. Port dn req: Not pending
 SDB entry: ifindex(1f000000) fabric if(1a000000)
    Dev: 0 Nif3 Hif7 (Nif:0x20000000 Hif:0x1f000000)

The fabric interconnect facing NIF is NIF3
The server facing HIF is HIF7

For a diagrammatic representation of this ports from the IOM asic (redwood) perspective, log into the IOM and run the following command

6100-A-B# connect local-mgmt a
6100-A-A(local-mgmt)# connect iom 1
fex-1# show platform software redwood sts
Board Status Overview:
legend:
       ' '= no-connect
       X  = Failed
       -  = Disabled
       :  = Dn
       |  = Up
       $  = SFP+ present
       v  = Blade Present
------------------------------
       +---+----+----+----+
 SFP:  |[$]| [$]| [$]| [$]|
       +---+----+----+----+
         |    |    |    |
       +-+----+----+----+-+
       | 0    1    2    3 |
       | I    I    I    I |
       | N    N    N    N |
       |                  |
       |      ASIC 0      |
       |                  |
       | H H H H H H H H  |
       | I I I I I I I I  |
       | 0 1 2 3 4 5 6 7  |
       +-+-+-+-+-+-+-+-+--+
         - - - - | : | |
        +-+-+-+-+-+-+-+-+
        |-|-|-|-|v|v|v|v|
        +-+-+-+-+-+-+-+-+
Blade:    8 7 6 5 4 3 2 1

mario55555 · ‎04-11-2011

Hallo, once again!
Can You explain?

Question about IPM 4.0:
1) I executed device colletor, but my new collector's status is always "Configuring" (not "Running"). I update page, and nothing... Why not "Running"?

2) When i created colletor, i set time range - "Monitored/real-time". Status was "Running". But creating IPM report, my collector doesn't appear neither in Operation task, neither in User defined group. After clicked OK, IPM requires collector setting. What's wrong?

stmccabe · ‎04-11-2011

Hello Timur,

I believe you are posting on the wrong discussion thread. This discuss is for Cisco UCS I believe the thread you were looking for is as follows:

https://supportforums.cisco.com/thread/2075426 - which has closed, but you can pick up the converstion on the general LMS list.

BR.

davidcheung · ‎04-12-2011

HI Steve,

We are troubleshooting a SAN performance issue.

The first step is to isolate where is the issue.

Our SAN network topology is

Blade (Bare metal windows 20008R2 server)->IOM->Fabric Interconnect->MDS Switch->EMC Storage

we think it shouldn't be bandwidth issue as we have 6x4G FC channle from each fabric to EMC storage.

We find it is hard to get the latency/jitter information for SAN traffic from Blade to Fabric Interconnect in UCS manager.

(There is no QOS policy in our environment as SAN traffic is given higher priority than Ethernet traffic.)

Is there any good tool can help us?

Thank you very much!

stmccabe · ‎04-12-2011

Hello David,

There are multiple tools out there to aid in performance testing, benchmarking and help with isolation of possible network, fabric, I/O performance issues.

IOMeter is a common tool that I use, which is a free open source tool to test disk performance. Moreover, you can measure the I/O transfer rate and the average I/O response time. IOMeter can be used with Windows 2003 and 2008. You can download IOMeter and users/admin guides by visiting the following:

http://www.iometer.org/

User guide: http://iometer.cvs.sourceforge.net/viewvc/iometer/iometer/Docs/Iometer.pdf

Another tool, IORate developed by EMC, which also helps measure performance. You can view the IORate userguide here and determine if this is a tool that you can use in your testing:

ftp://ftp.emc.com/pub/elab/iorate/

I'll mention one more tool - IPerf

Iperf was developed by NLANR/DAST as a modern alternative for measuring maximum TCP and UDP bandwidth performance. Iperf allows the tuning of various parameters and UDP characteristics. Iperf reports bandwidth, delay jitter, datagram loss.

link to IPerf on sourceforge: http://iperf.sourceforge.net/, and JPerf is a front end graphical tool for IPerf.

Can I ask, what are you using as a benchmark and what indications are you seeing that leads you to believe there is a SAN performance issue? Have you traced the Windows 2K8 host's path through the UCS, MDS and to the SAN (HBA ---> through to the ---> LUN)? If performance continues to be an issue, looking at the entire path and see if you have input/output errors, etc would be next step.

Hope this helps.

davidcheung · ‎04-12-2011

Thanks. Steve,

More information about our issue.

Our server guy is testing the performance from Web (Virtual machine)->App(Virtual Machine)->DB (Bare Metal). The taffic is initiated from Web server.

They found the response time from Web and DB is inconsistent. (sometimes it takes more than 150ms to get response)

All servers (including virtual machine and bare metal server) are running on EMC storage. Nexus1000v is used as virtual switch for Web and APP servers.

When server guy build a disk on the DB RAM, they can see the performance is perfect and consistent. The only difference is the SAN link from DB to SAN Storage.

It is the reason why we think the issue is around SAN network.

Could you please provide some recommendation about the tools to monitor the SAN performance (like Jitter and Latency) from UCS Chassis IOM to Fabric Interconnect, and from Fabric Interconnect to Cisco MDS switch?

Thank you very much!

stmccabe · ‎04-13-2011

Hi David,

Leveraging a tool like IOMeter along with isolating the host's path through the fabric will be key in pin pointing the issue. A few questions/suggestions:

1) Are there any other blades within the same chassis and/or UCS system experiencing storage related performance issues?
2) Has the MDS and 6100's been examined for any interface or queue errors, e.g., checking if buffer credit transition from zero?

3) Which adapter/HBA is in use on the Blade (e.g., M71KR-E, M81KR)?
4) Is this a redundant fabric - e.g., Fab-A and Fab-B, active/active paths? If so, I would start by isolating the host's path through the fabric and test/baseline with a tool like IOMeter with the host isolated to on single path, e.g. accessing the SAN over Fab-A. A second test where host is accessing SAN over fabric B.

On the MDS you can setup FC analyzer and SPAN. The FC analyzer examines packets to and from the entities that the switch provides. The FC analyzer is able to debug frames that the switch is responsible for receiving or sending to a storage device. You will then use the SPAN function to examine the session flows.

Configuring FC analyzer and SPAN:

http://www.cisco.com/en/US/products/hw/ps4159/ps4358/products_configuration_example09186a008026eb55.shtml

Setup a SPAN on UCS/6100:

http://www.cisco.com/en/US/partner/docs/unified_computing/ucs/sw/cli/config/guide/1.4.1/CLI_Config_Guide_1_4_1_chapter42.html#concept_0B45AC9C7E7A463AA97B728C728429DB

tenaro.gusatu.novici · ‎04-12-2011

Hi Steve,

this is one really great topic, and your first post is even better!

Now, let me start with one basic question: What will happen if two VMs are running on the same blade (yep, B series, no local disks inside blade, instead SAN is used) and exchanging huge amount of data between them? How many times will traffic pass on the link between Fabric Interconnect 6100 and the blade? Is it once from the storage via 6100 into first VM and copied to the second VM meaning going back via same link into 6100 and then back to the storage? And if we are talking about couple video files (i.e. dozen GB) will each block pass once in each direction over the link connecting blade and fiber interconnect?

Regards,

Tenaro

stmccabe · ‎04-12-2011

Hi Tenaro,

Good question.

With each of the VM's disks residing on the SAN, we have two unique virtual disks for each device (VM-A and VM-B) to read/write to - So, by the nature of this setup the VM's traffic would require traversing the UCS fabric. With that said, there are techniques that can be implemented on the SAN allowing optimization of I/O, and overall re-writes for data shared and/or common among the two VM's. Centralized data deduplication is one such process, however there are considerations here as well. For example, with encrypted data, and some imaging and streaming video files (as in your example), tend to yield poor deduplication results because of the random nature of the content.

From the LAN side of things, two VMs running on the same blade if in the same VLAN would be locally switched if connected (pinned) to the same 6100. The following VoD provides a good overview of how VM switching occures within UCS: http://vimeo.com/12782457

BR.

simon.geary · ‎04-13-2011

Hi,

I'm working on a UCS design that will use NetApp storage directly connected to the Fabric Interconnects using the FCoE storage ports.

The NetApp will be configured as a MetroCluster, with the two controllers being located in different rooms within the same building, 90m apart.

The plan at the moment is to use the SFP-10G-LR long range SFP+ modules to directly connect the FIs to the storage controller that is 90m away, and use the regular Twinax cables to connect to the second controller in the same room.

So my question is, are there any distance, latency or other considerations that would prevent such a configuration? Is it supported\recommended to run FCoE over this distance with UCS?

Cheers,

Simon

stmccabe · ‎04-13-2011

Hello Simon,

300meters is distance limitation for FCOE on the UCS, so you will be well within the limit if using LR SFP+ gbics.

For reference, FI-IOM for FCoE is limited by interface buffers and not optical power. 

Typically for Multi-Mode Fibre (MMF) using SW (850nm laser)  the limit is the fiber cable (due to modal dispersion). 
So OM-3 gives better range than OM-2 fiber.

For Single-Mode Fibre (SMF) using LW (1310nm laser) the limit is optical power (fiber attenuation) and 
buffer depth for no-drop FC/FCoE protocols - this is where the 300Meters come into play.

Documentation referencing distance limitations for long range (LR) SFP+ / Specifically refer to table 4:

http://www.cisco.com/en/US/prod/collateral/ps10265/ps10276/data_sheet_c78-524724.html

BR.

simon.geary · ‎04-13-2011

Great info, thanks.

stanibarb · ‎04-18-2011

Hello,

Is it possible to tell me when is expected for UCS blade and rack-mount severs to support Infiniband?

I will be very glad if you can share with me a reference documentation about Infiniband over Ethernet in Cisco UCS.

Thank you very much in advance!

Best regard,

Stanley

stmccabe · ‎04-18-2011

Hello Stanley,

As of now I'm not aware of any directly integrated managed support for Inifiband on B or C series.. However, with that said on C-Series is currently orderable with Mellanox ConnectX RoCE NICs. If you had the Mellanox IB adaptor and a Cisco CNA adapator you could still manage the UCS via UCSM (one to a 2248 and one to a Fabric interconnect).

Hth.