cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2487
Views
20
Helpful
11
Replies

UCS failure test

Amr Hafez
Level 1
Level 1

Hi All

I deployed a new DC using UCS 5108 with 4 blades (only 2 worked now) also 2-FI and connected to uptream core SW

2-links between IOM and FI (chassis discovery policy by 1 link)

1- uplink per FI to core SW as in the image1

also I deployed ESXi6 and server 2012 R2 on both blade 5,6 for testing

when I tried to test connectivity and HA by the following methodology:

1- disable one IOM1-FIA link : every thing is good and no pachets dropped for ESXi, VMs, mgmt 

2- disable 2 links IOM1-FIA: every thing good also oh! thats awesome !!

3- disable 2 links IOM1-FIA and 1link IOM2-FIB the ESXi in blade 5 lost connection but ESXi on blade 6 did not lose connection !! and I found that the backplane ports for IOM 2 for blade 5 was down but for blade 6 not down as in test1.2.3  images

I do not know why such different behavior

could you help me?

thanks in advance

Amr 

2 Accepted Solutions

Accepted Solutions

For the port-channel, go to the Equipment tab > Policies > Global policies> and under 1-Link, change it from none to Port-channel.

If you dont have port-channel configured, it is totally expected that the blade will go down when it loses both paths.  Remember that in UCSM, connectivity is based on pinning hardware here in there, in other words, your blade is pinned to a backplane port in the IOM and that backplane port is pinned to Fabric port and then eventually pinned to a uplink port.

 

Let me know if you have doubts.

 

-Kenny

View solution in original post

Yes, if you change chassis connectivity from none to port channel you have to re acknowledge the chassis.  That tells the FI to dump existing config and rescan the IOMs.  It's ar chassis acknowledgement that the port channel is created.  If the port channel doesn't exist and you enable it, that only affects new chassis. Ucs won't reset the IOM-FI links without being told to.

View solution in original post

11 Replies 11

ssumichrast
Level 1
Level 1

Firsr, I suggest doing two as a minimum connection policy.  If you can't for some business reason so be it, but you really should require two links from each IOM to the FI.

 

Second, do you have Port Channel specified as the Link Grouping policy? Assuming not, what's happening is your vNICs are being pinned to the physical IOM uplink interface. Based on number of links, each blade will be pinned statically to a cable if you're not using port channel.  So in your case, uplink 1 from IOM carries traffic for blade 1,3,5 and 7. Uplink 2 from IOM a carries 2,4,6 and 8.  So when you unplug certain cables the vNICs riding that cable (which is determined by the blade location by default) will show down in ESXi.  In ESXi you likely have one nic from A and one from B.  When you unplug IOM cables on both in the same position you lose all paths upstream.  I hope that makes sense...  Again this is if you're NOT using port channel.  I strongly recommend port channel.  When you port channel from IOM to FI your blade VNICs get pinned to the port channel, not the specific uplink.  That lets you take links down without worrying about a vNIC losing its pinned uplinks.

 

Heres a good article on this: http://keepingitclassless.net/2013/06/cisco-ucs-port-channeling/

 

Steve,

You have any experience having issues with the Chassis Discovery Policy (CDP) set to 1-Link?  In my case, I always recommend using 1 link in the CDP, that way your IOM only needs at least one link to be fully functional to come up, if more are connected, you just need to reack the chassis to re-pin your server's interfaces to the excess IOM interfaces.

I agree with you on everything else.

 

Note: For the Port-channel to work between FI and IOMs, you need to have 2nd generation hardware at least... that means 62xx FIs anf 220x IOMs.

 

-Kenny

 

Kenperez,

I agree with you I used only 1 link for each IOM and set CDP to 1 link and every thing is good

for port-ch I've already 6248UP and FEX 2208XP ?

but when using one IOM with one link to FI some blades down?

where can i configure port-ch between IOM-FI?

I'm trying to config po-ch as in pic1 but I cannot

For the port-channel, go to the Equipment tab > Policies > Global policies> and under 1-Link, change it from none to Port-channel.

If you dont have port-channel configured, it is totally expected that the blade will go down when it loses both paths.  Remember that in UCSM, connectivity is based on pinning hardware here in there, in other words, your blade is pinned to a backplane port in the IOM and that backplane port is pinned to Fabric port and then eventually pinned to a uplink port.

 

Let me know if you have doubts.

 

-Kenny

Thanks Kenny

I do that now but the internal LAN port-ch still can not accessible as in img1 attached

also I tried to test using  only 1 link between IOM-FI  ( I used 2 for each fabric in common topology) after CDP 1 link and port-ch  activated (as in cdp image) but the same result Blade 5 ESXi is not accessible but blade 6 ESXi accessible as in test1,2

thanks 

Amr

 

Did you re acknowledge the chassis after enabling port channel?

No, I think no need for re-ack

but any way I'll try 

 

Yes, if you change chassis connectivity from none to port channel you have to re acknowledge the chassis.  That tells the FI to dump existing config and rescan the IOMs.  It's ar chassis acknowledgement that the port channel is created.  If the port channel doesn't exist and you enable it, that only affects new chassis. Ucs won't reset the IOM-FI links without being told to.

It's a "preference", I suppose, that we have adopted at my company to require redundancy whenever possible.  We do not want chassis with single links, period.  We always want a redundant link to the chassis IOMs (yes there are two IOMs, but if that's the argument why connect two power supplies to the FI?), so we require a minimum of two links per chassis.  

 

Its ts just a policy I enforce in our environments because there's so much riding over this -- I dont want a single cable being a point of failure.

Thanks Steven

I got it 

but where can I adjust Link Grouping policy for IOM-FI links?

Amr

Amr,

Just to confirm, you saved the changed when you moved to "Port-channel" from "none", right?

Can you open an SSH session and paste the output of the following commands here:

 

#show service-profile circuit server x/y   >>> x=chassis y=server (one for each server involved in this thread.

#connect nx a

#show interface fex-fabric

#show port-channel summary <<<< LACP port-channels will be the uplink port-channels, the others are the internal port-channels going to the IOM

#exit

#connect nx b    <<<< repeat the commands above

 

 

-Kenny

 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: