04-07-2015 03:41 PM - edited 03-01-2019 12:07 PM
Hi All
I deployed a new DC using UCS 5108 with 4 blades (only 2 worked now) also 2-FI and connected to uptream core SW
2-links between IOM and FI (chassis discovery policy by 1 link)
1- uplink per FI to core SW as in the image1
also I deployed ESXi6 and server 2012 R2 on both blade 5,6 for testing
when I tried to test connectivity and HA by the following methodology:
1- disable one IOM1-FIA link : every thing is good and no pachets dropped for ESXi, VMs, mgmt
2- disable 2 links IOM1-FIA: every thing good also oh! thats awesome !!
3- disable 2 links IOM1-FIA and 1link IOM2-FIB the ESXi in blade 5 lost connection but ESXi on blade 6 did not lose connection !! and I found that the backplane ports for IOM 2 for blade 5 was down but for blade 6 not down as in test1.2.3 images
I do not know why such different behavior
could you help me?
thanks in advance
Amr
Solved! Go to Solution.
04-08-2015 01:17 PM
For the port-channel, go to the Equipment tab > Policies > Global policies> and under 1-Link, change it from none to Port-channel.
If you dont have port-channel configured, it is totally expected that the blade will go down when it loses both paths. Remember that in UCSM, connectivity is based on pinning hardware here in there, in other words, your blade is pinned to a backplane port in the IOM and that backplane port is pinned to Fabric port and then eventually pinned to a uplink port.
Let me know if you have doubts.
-Kenny
04-09-2015 11:25 AM
Yes, if you change chassis connectivity from none to port channel you have to re acknowledge the chassis. That tells the FI to dump existing config and rescan the IOMs. It's ar chassis acknowledgement that the port channel is created. If the port channel doesn't exist and you enable it, that only affects new chassis. Ucs won't reset the IOM-FI links without being told to.
04-07-2015 05:09 PM
Firsr, I suggest doing two as a minimum connection policy. If you can't for some business reason so be it, but you really should require two links from each IOM to the FI.
Second, do you have Port Channel specified as the Link Grouping policy? Assuming not, what's happening is your vNICs are being pinned to the physical IOM uplink interface. Based on number of links, each blade will be pinned statically to a cable if you're not using port channel. So in your case, uplink 1 from IOM carries traffic for blade 1,3,5 and 7. Uplink 2 from IOM a carries 2,4,6 and 8. So when you unplug certain cables the vNICs riding that cable (which is determined by the blade location by default) will show down in ESXi. In ESXi you likely have one nic from A and one from B. When you unplug IOM cables on both in the same position you lose all paths upstream. I hope that makes sense... Again this is if you're NOT using port channel. I strongly recommend port channel. When you port channel from IOM to FI your blade VNICs get pinned to the port channel, not the specific uplink. That lets you take links down without worrying about a vNIC losing its pinned uplinks.
Heres a good article on this: http://keepingitclassless.net/2013/06/cisco-ucs-port-channeling/
04-08-2015 10:51 AM
Steve,
You have any experience having issues with the Chassis Discovery Policy (CDP) set to 1-Link? In my case, I always recommend using 1 link in the CDP, that way your IOM only needs at least one link to be fully functional to come up, if more are connected, you just need to reack the chassis to re-pin your server's interfaces to the excess IOM interfaces.
I agree with you on everything else.
Note: For the Port-channel to work between FI and IOMs, you need to have 2nd generation hardware at least... that means 62xx FIs anf 220x IOMs.
-Kenny
04-08-2015 01:15 PM
Kenperez,
I agree with you I used only 1 link for each IOM and set CDP to 1 link and every thing is good
for port-ch I've already 6248UP and FEX 2208XP ?
but when using one IOM with one link to FI some blades down?
where can i configure port-ch between IOM-FI?
I'm trying to config po-ch as in pic1 but I cannot
04-08-2015 01:17 PM
For the port-channel, go to the Equipment tab > Policies > Global policies> and under 1-Link, change it from none to Port-channel.
If you dont have port-channel configured, it is totally expected that the blade will go down when it loses both paths. Remember that in UCSM, connectivity is based on pinning hardware here in there, in other words, your blade is pinned to a backplane port in the IOM and that backplane port is pinned to Fabric port and then eventually pinned to a uplink port.
Let me know if you have doubts.
-Kenny
04-08-2015 03:08 PM
Thanks Kenny
I do that now but the internal LAN port-ch still can not accessible as in img1 attached
also I tried to test using only 1 link between IOM-FI ( I used 2 for each fabric in common topology) after CDP 1 link and port-ch activated (as in cdp image) but the same result Blade 5 ESXi is not accessible but blade 6 ESXi accessible as in test1,2
thanks
Amr
04-08-2015 08:24 PM
Did you re acknowledge the chassis after enabling port channel?
04-08-2015 08:28 PM
No, I think no need for re-ack
but any way I'll try
04-09-2015 11:25 AM
Yes, if you change chassis connectivity from none to port channel you have to re acknowledge the chassis. That tells the FI to dump existing config and rescan the IOMs. It's ar chassis acknowledgement that the port channel is created. If the port channel doesn't exist and you enable it, that only affects new chassis. Ucs won't reset the IOM-FI links without being told to.
04-08-2015 02:55 PM
It's a "preference", I suppose, that we have adopted at my company to require redundancy whenever possible. We do not want chassis with single links, period. We always want a redundant link to the chassis IOMs (yes there are two IOMs, but if that's the argument why connect two power supplies to the FI?), so we require a minimum of two links per chassis.
Its ts just a policy I enforce in our environments because there's so much riding over this -- I dont want a single cable being a point of failure.
04-08-2015 12:50 PM
Thanks Steven
I got it
but where can I adjust Link Grouping policy for IOM-FI links?
Amr
04-09-2015 05:47 AM
Amr,
Just to confirm, you saved the changed when you moved to "Port-channel" from "none", right?
Can you open an SSH session and paste the output of the following commands here:
#show service-profile circuit server x/y >>> x=chassis y=server (one for each server involved in this thread.
#connect nx a
#show interface fex-fabric
#show port-channel summary <<<< LACP port-channels will be the uplink port-channels, the others are the internal port-channels going to the IOM
#exit
#connect nx b <<<< repeat the commands above
-Kenny
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide