04-08-2011 11:50 AM - edited 03-01-2019 09:53 AM
Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn how to configure and troubleshoot UCS in various environments with Cisco expert Stephen McCabe. Stephen is a customer support engineer in Cisco’s Advanced Services group providing High Touch Technical Support. As part of the Data Center Networking team, he supports customers on the Cisco Unified Computing System and Cisco Nexus platforms. In addition, he serves as an escalations engineer for Cisco's Application Networking Services team supporting server/application load balancing, WAN acceleration, web caching proxy, and video and streaming media. Stephen holds a bachelor's degree in computer engineering from Ohio University. He also holds CCIE #23917 certification in Security and is a Red Hat Certified Engineer, and is VMWare Certified Professional #71728.
Remember to use the rating system to let Stephen know if you have received an adequate response.
Stephen might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Unified Computing discussion forum shortly after the event. This event lasts through April 22, 2011. Visit this forum often to view responses to your questions and the questions of other community members.
04-11-2011 08:29 AM
To kick the conversation off regarding UCS troubleshooting, I'm posting a "how-to" that I've found very useful when working with customers on UCS B-Series servers. There is an interest and benefit in understanding how to trace a specific host/guest os's path through the UCS system.
Here is a high-level method that I use via the UCS CLI that a customer can use to identify the path of the packet given either a MAC address from a host and/or, VM behind a host, or the location of the server, that is, chassis number and slot number..
I'm going to use an example here where I have the mac address of a VMWare host with a mac address of 0050.56a6.076a
1) SSH to the UCSM IP address
2) Assuming that you do not know which fabric is receiving frames from the given MAC address, check both Fabric interconnects for the mac address.
6100-A-B# connect nxos a 6100-A-A(nxos)# show mac-address address 0050.56a6.076a VLAN MAC Address Type Age Port ---------+-----------------+-------+---------+------------------------------ 503 0050.56a6.076a dynamic 10 veth928
#Found mac address in Fabric interconnect A. It should not be visible on Fabric interconnect B. If it is then the customer is doing
per flow/packet load balancing at the host level, which is not allowed on UCS B-Series
6100-A-A(nxos)# exit
#Checking fabric interconnect B for the mac address. MAC address is not present here, which is what we expect to see.
6100-A-B# connect nxos b 6100-A-B(nxos)# show mac-address address 0050.56a6.076a Total MAC Addresses: 0
3) Now identify blade location. In this example, I logged back Fabric Interconnect A.
UCS-6100-A-A(nxos)# show int veth928
vethernet928 is up
Bound Interface is Ethernet1/1/1
Hardware: VEthernet
Encapsulation ARPA
Port mode is trunk
Last link flapped 1week(s) 5day(s)
Last clearing of "show interface" counters never
1 interface resets
Ethernet | 1 / | | 1 / | | 1 |
chassis no. | remote entity | slot no |
The diagram below summarizes the commands used to figure out the path of the packet through the UCS using this example.
There are 2 ports on the Fabric interconnect to identify.
6100-A-A(nxos)# show interface | include veth|1\/1\/1 vethernet928 is up Bound Interface is Ethernet1/1/1 vethernet1048 is up vethernet1052 is down (vicEnNotRcvd) vethernet1053 is down (vicEnNotRcvd) vethernet9122 is up Bound Interface is Ethernet1/1/1 vethernet9241 is up vethernet9246 is down (vicEnNotRcvd) Bound interface is vethernet9122 Bound interface is vethernet9241 Bound interface is vethernet9246 Ethernet1/1/1 is up 6100-A-A(nxos)# show pinning interface veth928 ---------------+-----------------+------------------------+----------------- SIF Interface Sticky Pinned Border Interface Pinned Duration ---------------+-----------------+------------------------+----------------- veth928 No Eth1/8 5d 39:13:43
6100-A-A(nxos)# show platform software fex info satport ethernet 1/1/1
Interface-Name ifindex State Fabric-if Pri-fabric Expl-Pinned
Eth1/1/1 0x1f000000 Up Eth1/1 Eth1/4 Eth1/1
Port Phy Up. Port dn req: Not pending
SDB entry: ifindex(1f000000) fabric if(1a000000)
Dev: 0 Nif3 Hif7 (Nif:0x20000000 Hif:0x1f000000)
IOM has Fabric interconnect facing ports, 4 in total, and 8 blade server facing ports (internal).
6100-A-A(nxos)# show interface fex-fabric
Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
---------------------------------------------------------------
1 Eth1/1 Active 1 N20-C6508 QCI132800P2
1 Eth1/2 Active 2 N20-C6508 QCI132800P2
1 Eth1/3 Active 3 N20-C6508 QCI132800P2
1 Eth1/4 Active 4 N20-C6508 QCI132800P2
6100-A-A(nxos)# show platform software fex info satport ethernet 1/1/1
Interface-Name ifindex State Fabric-if Pri-fabric Expl-Pinned
Eth1/1/1 0x1f000000 Up Eth1/1 Eth1/4 Eth1/1
Port Phy Up. Port dn req: Not pending
SDB entry: ifindex(1f000000) fabric if(1a000000)
Dev: 0 Nif3 Hif7 (Nif:0x20000000 Hif:0x1f000000)
6100-A-B# connect local-mgmt a 6100-A-A(local-mgmt)# connect iom 1 fex-1# show platform software redwood sts Board Status Overview: legend: ' '= no-connect X = Failed - = Disabled : = Dn | = Up $ = SFP+ present v = Blade Present ------------------------------ +---+----+----+----+ SFP: |[$]| [$]| [$]| [$]| +---+----+----+----+ | | | | +-+----+----+----+-+ | 0 1 2 3 | | I I I I | | N N N N | | | | ASIC 0 | | | | H H H H H H H H | | I I I I I I I I | | 0 1 2 3 4 5 6 7 | +-+-+-+-+-+-+-+-+--+ - - - - | : | | +-+-+-+-+-+-+-+-+ |-|-|-|-|v|v|v|v| +-+-+-+-+-+-+-+-+ Blade: 8 7 6 5 4 3 2 1
04-11-2011 11:59 AM
Hallo, once again!
Can You explain?
Question about IPM 4.0:
1) I executed device colletor, but my new collector's status is always "Configuring" (not "Running"). I update page, and nothing... Why not "Running"?
2) When i created colletor, i set time range - "Monitored/real-time". Status was "Running". But creating IPM report, my collector doesn't appear neither in Operation task, neither in User defined group. After clicked OK, IPM requires collector setting. What's wrong?
04-11-2011 12:14 PM
Hello Timur,
I believe you are posting on the wrong discussion thread. This discuss is for Cisco UCS I believe the thread you were looking for is as follows:
https://supportforums.cisco.com/thread/2075426 - which has closed, but you can pick up the converstion on the general LMS list.
BR.
04-12-2011 03:22 AM
HI Steve,
We are troubleshooting a SAN performance issue.
The first step is to isolate where is the issue.
Our SAN network topology is
Blade (Bare metal windows 20008R2 server)->IOM->Fabric Interconnect->MDS Switch->EMC Storage
we think it shouldn't be bandwidth issue as we have 6x4G FC channle from each fabric to EMC storage.
We find it is hard to get the latency/jitter information for SAN traffic from Blade to Fabric Interconnect in UCS manager.
(There is no QOS policy in our environment as SAN traffic is given higher priority than Ethernet traffic.)
Is there any good tool can help us?
Thank you very much!
04-12-2011 08:11 AM
Hello David,
There are multiple tools out there to aid in performance testing, benchmarking and help with isolation of possible network, fabric, I/O performance issues.
IOMeter is a common tool that I use, which is a free open source tool to test disk performance. Moreover, you can measure the I/O transfer rate and the average I/O response time. IOMeter can be used with Windows 2003 and 2008. You can download IOMeter and users/admin guides by visiting the following:
User guide: http://iometer.cvs.sourceforge.net/viewvc/iometer/iometer/Docs/Iometer.pdf
Another tool, IORate developed by EMC, which also helps measure performance. You can view the IORate userguide here and determine if this is a tool that you can use in your testing:
ftp://ftp.emc.com/pub/elab/iorate/
I'll mention one more tool - IPerf
Iperf was developed by NLANR/DAST as a modern alternative for measuring maximum TCP and UDP bandwidth performance. Iperf allows the tuning of various parameters and UDP characteristics. Iperf reports bandwidth, delay jitter, datagram loss.
link to IPerf on sourceforge: http://iperf.sourceforge.net/, and JPerf is a front end graphical tool for IPerf.
Can I ask, what are you using as a benchmark and what indications are you seeing that leads you to believe there is a SAN performance issue? Have you traced the Windows 2K8 host's path through the UCS, MDS and to the SAN (HBA ---> through to the ---> LUN)? If performance continues to be an issue, looking at the entire path and see if you have input/output errors, etc would be next step.
Hope this helps.
04-12-2011 07:17 PM
Thanks. Steve,
More information about our issue.
Our server guy is testing the performance from Web (Virtual machine)->App(Virtual Machine)->DB (Bare Metal). The taffic is initiated from Web server.
They found the response time from Web and DB is inconsistent. (sometimes it takes more than 150ms to get response)
All servers (including virtual machine and bare metal server) are running on EMC storage. Nexus1000v is used as virtual switch for Web and APP servers.
When server guy build a disk on the DB RAM, they can see the performance is perfect and consistent. The only difference is the SAN link from DB to SAN Storage.
It is the reason why we think the issue is around SAN network.
Could you please provide some recommendation about the tools to monitor the SAN performance (like Jitter and Latency) from UCS Chassis IOM to Fabric Interconnect, and from Fabric Interconnect to Cisco MDS switch?
Thank you very much!
04-13-2011 10:29 AM
Hi David,
Leveraging a tool like IOMeter along with isolating the host's path through the fabric will be key in pin pointing the issue. A few questions/suggestions:
1) Are there any other blades within the same chassis and/or UCS system experiencing storage related performance issues?
2) Has the MDS and 6100's been examined for any interface or queue errors, e.g., checking if buffer credit transition from zero?
3) Which adapter/HBA is in use on the Blade (e.g., M71KR-E, M81KR)?
4) Is this a redundant fabric - e.g., Fab-A and Fab-B, active/active paths? If so, I would start by isolating the host's path through the fabric and test/baseline with a tool like IOMeter with the host isolated to on single path, e.g. accessing the SAN over Fab-A. A second test where host is accessing SAN over fabric B.
On the MDS you can setup FC analyzer and SPAN. The FC analyzer examines packets to and from the entities that the switch provides. The FC analyzer is able to debug frames that the switch is responsible for receiving or sending to a storage device. You will then use the SPAN function to examine the session flows.
Configuring FC analyzer and SPAN:
Setup a SPAN on UCS/6100:
04-12-2011 12:42 PM
Hi Steve,
this is one really great topic, and your first post is even better!
Now, let me start with one basic question: What will happen if two VMs are running on the same blade (yep, B series, no local disks inside blade, instead SAN is used) and exchanging huge amount of data between them? How many times will traffic pass on the link between Fabric Interconnect 6100 and the blade? Is it once from the storage via 6100 into first VM and copied to the second VM meaning going back via same link into 6100 and then back to the storage? And if we are talking about couple video files (i.e. dozen GB) will each block pass once in each direction over the link connecting blade and fiber interconnect?
Regards,
Tenaro
04-12-2011 03:08 PM
Hi Tenaro,
Good question.
With each of the VM's disks residing on the SAN, we have two unique virtual disks for each device (VM-A and VM-B) to read/write to - So, by the nature of this setup the VM's traffic would require traversing the UCS fabric. With that said, there are techniques that can be implemented on the SAN allowing optimization of I/O, and overall re-writes for data shared and/or common among the two VM's. Centralized data deduplication is one such process, however there are considerations here as well. For example, with encrypted data, and some imaging and streaming video files (as in your example), tend to yield poor deduplication results because of the random nature of the content.
From the LAN side of things, two VMs running on the same blade if in the same VLAN would be locally switched if connected (pinned) to the same 6100. The following VoD provides a good overview of how VM switching occures within UCS: http://vimeo.com/12782457
BR.
04-13-2011 11:51 AM
Hi,
I'm working on a UCS design that will use NetApp storage directly connected to the Fabric Interconnects using the FCoE storage ports.
The NetApp will be configured as a MetroCluster, with the two controllers being located in different rooms within the same building, 90m apart.
The plan at the moment is to use the SFP-10G-LR long range SFP+ modules to directly connect the FIs to the storage controller that is 90m away, and use the regular Twinax cables to connect to the second controller in the same room.
So my question is, are there any distance, latency or other considerations that would prevent such a configuration? Is it supported\recommended to run FCoE over this distance with UCS?
Cheers,
Simon
04-13-2011 12:10 PM
Hello Simon,
300meters is distance limitation for FCOE on the UCS, so you will be well within the limit if using LR SFP+ gbics.
For reference, FI-IOM for FCoE is limited by interface buffers and not optical power.
Typically for Multi-Mode Fibre (MMF) using SW (850nm laser) the limit is the fiber cable (due to modal dispersion).
So OM-3 gives better range than OM-2 fiber.
For Single-Mode Fibre (SMF) using LW (1310nm laser) the limit is optical power (fiber attenuation) and
buffer depth for no-drop FC/FCoE protocols - this is where the 300Meters come into play.
Documentation referencing distance limitations for long range (LR) SFP+ / Specifically refer to table 4:
http://www.cisco.com/en/US/prod/collateral/ps10265/ps10276/data_sheet_c78-524724.html
BR.
04-13-2011 12:38 PM
Great info, thanks.
04-18-2011 12:13 AM
Hello,
Is it possible to tell me when is expected for UCS blade and rack-mount severs to support Infiniband?
I will be very glad if you can share with me a reference documentation about Infiniband over Ethernet in Cisco UCS.
Thank you very much in advance!
Best regard,
Stanley
04-18-2011 07:58 AM
Hello Stanley,
As of now I'm not aware of any directly integrated managed support for Inifiband on B or C series.. However, with that said on C-Series is currently orderable with Mellanox ConnectX RoCE NICs. If you had the Mellanox IB adaptor and a Cisco CNA adapator you could still manage the UCS via UCSM (one to a 2248 and one to a Fabric interconnect).
Hth.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide