09-04-2013 09:52 AM - edited 03-01-2019 11:14 AM
Hi All,
Having a performance issue with FCoE multihop from UCS.
The layout is as follows:
UCS 5108 Chassis --> 6248UP FI's --> Pair of N5k's (5548) --> EMC VNX 5300 2 port FCoE 10gig card x2
Everything is functional - i can fc ping from n5k fabrics to the EMC pwwn.
I can see the pathing in vmware on the vhbas all is up and active. The VMware mode is set to round robin and the EMC side ALUA 4.
I'm at a loss on how to troubleshoot this further.The SAN is connected to the fabric with LC fibre om3 + 10gb SR SFP's. UCS has a seperate pair of fcoe port channels from the 5k fabric down to the FI's (using twinax leads). Data is seperated on its OWN vpc. I essentially followed the cisco guide on how to do this:
http://www.cisco.com/en/US/products/ps9670/products_configuration_example09186a0080c13e92.shtml
The hosts are not boot from SAN for now, they are booted off USB flash storage on the blade.
The show flogi on the 5ks shows everything logged in and as i said before i can see the LUNs on the blades. When howerver i try to uplod an ISO file say 2.5gb in size to a datastore is taking roughly 2 and half hours.
I have the newest UC 2.1(2)a firmware installed. The N5ks have 6.x NX-OS. The blades have ESXi 5.1 with the newest enet and fnic drivers from cisco installed.
The default qos for n5k is configured.
Any ideas would be greatly appreciated!
-Nick
Solved! Go to Solution.
09-06-2013 01:20 AM
Hi Nick,
Interesting observation. We have a hsrp IP in catalyst 4500 as a router to the management network with ip redirects on for some reason. I disabled the ip redirects as well and currently I'm doing 30K iops with RR path dual fabric's enabled and no fnic errors on esxi's logs.
I'll have to do some additional testing to see weather this is a coincidence or a fix by enabling ip redirects again. But it does look promising.
As my i/o load is done with i/o generator (http://labs.vmware.com/flings/io-analyzer) management network should have no effect on non routed FCOE network... It newer even occurred to my mind to check Ethernet network side.
Thanks Nick for a good advice.
regards
Mikko
09-04-2013 01:06 PM
Hi Nick,
How many QoS drop classes do you have on the UCS and 5k side (http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCuh72875)?
What happens if you copy the ISO or VMDK from one datastore to another from the ESXi host cli? Is it still slow?
09-04-2013 07:49 PM
Hi Brian,
Ok, i uploaded a 300mb iso which talk around 8 mins. I then copied it from one datastore to the other with ESXi cli - it took less than 30 seconds to copy the file.
So something is def dropping / impacting the traffic to the SAN from UCS...
Whats the best way to check if qos is doing it?
09-05-2013 05:08 AM
Hi Nick,
Based on the results of copying from datastore to datastore which is purely FCoE, it seems that the storage path is ok and the problem may likely be somewhere on the ethernet side between the Vsphere Client and ESXi mgmt vmkernel. What kind of load balancing is configured on the vSwitch (for b-series blades it shouldn't be route based on ip hash)? Have you tried copying the file using Winscp to the esxi host to rule out an issue with the vsphere client?
09-05-2013 07:03 AM
what i just noticed then brian - is copying directly to the blades USB storage (esxi) for a VIB file - i got full bandwidth and thats from a VM sitting on another host via 2k --> 5k --> FI --> Blade
so that tells me it aint the ethernet link to the vmware kernel - it points to the blades vhbas / fcoe uplinks from FI.
04-19-2016 09:30 AM
Hi All,
Looking for your help for setup FCOE network between VNX,nexus 5k and FI 6248.
i have below query-
1. where exactly i should use FCOE,between FI and VNX or N5k and vnx.
2. Which connectivity option should i use Ethernet 10G link on N5K and FI or FC Ports.
04-19-2016 11:32 PM
see e.g.
http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/116248-configure-fcoe-00.html
http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-manager/116188-configure-fcoe-00.html
Questions:
- Has your VNX FCoE and native FC interfaces, or only FCoE ? (I saw your discussion in https://supportforums.cisco.com/discussion/12983641/fcoe-module-part-number-vnx-5400); I would be conservative and go with classical FC !
- instead of multihop FCoE, you could also do classical FC from FI to N5k; connect the VNX either with FCoE or FC to N5k. Do zoning on N5k !
- this would avoid multhop FCoE, which works, but is challenging.
04-20-2016 01:51 AM
04-20-2016 03:53 AM
Hi Kshitij
I would simply go with classical FC, not FCoE.
UCS FI supports classical FC, N5k as well.
Why do you need iSCSI ? iSCSI and FC are block based storage ? do you have clients with Ethernet interfaces (and iSCSI clients) ?
Walter.
04-20-2016 05:07 AM
now we both have been frustrated and finally they ask me where exactly i want to deply FCOE on my nexus/MDS/FI.
as i know we can configure FCOE only on 10 G Ethernet ports on MDS/FI/Nexus and we require active twinax cables for fcoe connectivity between MDS/NExus/FI to VNX.
04-20-2016 07:59 AM
If your OS uses native FC, we actually run FCoE between blade and FI.
Between FI and Northbound we can run either FCoE (to a N5k),
or
disaggregate the converged traffic, running FC over a native FC link to a MDS or a N5k FC port, AND Ethernet / IP to N5k Ethernet ports
Storage would be the connected with classical FC to MDS or N5k native FC port.
FC Cabling between VNX and N5k would fibre, 8G SFP's.
Summary; FCoE is only used in the access layer, first hop.
This is what most UCS customers requiring classical FC do today.
04-20-2016 08:26 AM
Thanks walter for your reply.
But still i am confused what ports should i use on FI to run FCOE between VNX and FI.
as per my understanding we can use only 10G ports on FI to connect VNX FCOE Module to run FCOE traffic,please do let me know if i can run FCOE on FC ports.
Please do let me know if my understanding is correct.so based on that i need to order 2 Ports 10G FCOE module for VNX and active twinax cables and other parts which i require for FCOE connectivity.
04-20-2016 09:59 AM
Clarification: is your VNX directly connected to UCS FI, or N5k ?
And if it is directly connected to FI, why ?
You cannot run FCoE on a native FC port !
FI 62xx have so called universal ports: they can be configured to be Ethernet (IP and/or FCoE) or classical FC.
Yes, cheapest connectivity for 10G Ethernet (and FCoE) is passive or active twinax cables, although limited in length.
09-05-2013 06:40 AM
Hi Brian,
Ok the vcenter dvs is set to route based on originating virtual port id - i read to leave it like this i knew that ip hash is not supported.
I just tried with the winscp client - same issue get around 500kb/s see image.
09-05-2013 01:58 AM
On the ucs side I have only 2 enabled classes: Default Best Effort and FCOE.
(Attached picture).
What command can be used to get the same info out of nexus5500 switches? I think that support did check this out but to be sure I could double check.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide