Cisco UCS Uplinks to Brocade ICX or VDX Switch

[dorgillmstates]

We have two 10 Gb links going from our FIs to two core Brocade MLX switches. They work fine.

Now, we are trying to do a Layer 2 Disjoint for two 1 Gb links to a Brocade ICX 6610 DMZ switch. No matter what we do, we get an "ENM Loop Disabled" error. If we plug those 1 Gb connections into a Brocade VDX 6710 switch, there are no errors.

Can anyone give any insight on:

A - Can you even uplink to a Brocade ICX 6610?

B - Why would a VDX work but not an ICX?

I can't find any solid info on uplinking UCS Fabric Interconnects to Brocade Networking switches.. only fibre channel. Any help is appreciated!

[Robert Burns]

Are you trying to create a port channel (link aggregation) with the separate pairs of links?  LACP is the only channeling method we allow in UCS.

Do the ICX ports come up when you remove the other switch and Disjoint L2 configuration?

Robert

[dorgillmstates]

Robert,

Thank you for your prompt reply.

We disabled the 10 Gb uplinks to the MLX and only enabled the 1 Gb links to the ICX and still have the ENM Loop. The results were the same, even after removing the Disjoint.

We are did not create a LAG/channel as we are just trying to get the basic 1 Gb connections going at this point on the DMZ.

[Robert Burns]

Interesting.  We know UCS side works fine with the VCX, but not the ICX.

I'd guess either the Brocade is not configured correctly, or its not compatible (SW/HW). 

Can you paste:

1. The exact error message from UCSM when you see the ENM loop message.

2. The sw version of ICX and the port config

3. [SSH to UCSM]

-connect nxos

-show int x/y  (1G interface that's err disabling)

-show int trunk

-show int brief

Robert

[dorgillmstates]

Note: If we disable one of the 1 Gb ports, the DMZ connection works just fine. As soon as you bring both up, it loops.

1 - Error: Error Disabled on ENM Loop

2 - Ok, actually the DMZ switch is a FCX-24G. Software Version 7.0.01CT7F1

interface ethernet 1/1/3

no spanning-tree

!

interface ethernet 1/1/4

no spanning-tree

3 - See attached file

[Robert Burns]

Paste everything in a text file and attach it.  That'll keep the native formating and make it easier to read.

Robert

[Robert Burns]

From your putty Logs, I don't see any issues...

e1/17 and e1/21 are both up.  I also don't see any blocked VLANs.  Where's this "ENM Loop" you speak of?

Ethernet1/21 is up

  Hardware: 1000/10000 Ethernet, address: 547f.eeda.6f9c (bia 547f.eeda.6f9c)

  Description: U: Uplink

RE-SLCFI1-A(nxos)# sh int trunk

--------------------------------------------------------------------------------

Port          Native  Status        Port

              Vlan                  Channel

--------------------------------------------------------------------------------

Eth1/17       500     trunking      --

Eth1/21       501     trunking      --

--------------------------------------------------------------------------------

Port          Vlans Allowed on Trunk

--------------------------------------------------------------------------------

Eth1/17       1,5,500,999

Eth1/21       1,501,999

--------------------------------------------------------------------------------

Port          Vlans Err-disabled on Trunk

--------------------------------------------------------------------------------

Eth1/17       none

Eth1/21       none

--------------------------------------------------------------------------------

Port          STP Forwarding

--------------------------------------------------------------------------------

Eth1/17       1,5,500,999

Eth1/21       1,501,999

[dorgillmstates]

The connections come up, drop, loop error and then they come back up. Rinse, repeat.

I will pull the logs again with sh int trunk when the connection shows down again..

[Robert Burns]

Why are you disabling STP on the brocarde intefaces?  Can you remove that config.  Leave the default STP mode, which on brocade should be Fast Port Mode... (the equivelant to Cisco Port Fast).

Robert

[dorgillmstates]

Robert,

We tried STP in both modes. Same results.

[Robert Burns]

Can you get me the following output:

show platform software enm internal event-history interface ethernet 1/17

show platform software enm internal event-history interface ethernet 1/21

Again attach as text file.

Thanks,

Robert

[dorgillmstates]

Attached are both outputs as requested --

[Robert Burns]

Did you ensure the Brocade's interfaces are set as trunks, allowing the matching VLANs on the UCS side, and match the native VLANs on both sides?

Robert

[dorgillmstates]

Robert,

We tried tagging multiple VLANS to those Brocade ports -- This is the only equivalent setting I have found for these switches.

Also, yes, we have matching native VLANs. Still, the same results.

[dorgillmstates]

Just as a follow up, thank you for your help Robert. Here is the solution that we found with TAC's help to end this ENM Loop issue:

- The cause of this is due to the way the Foundry switch handles LLDP frames.  The IEEE 802.1D specification indicates that any frame with a destination MAC address of 01:80:c2:00:00:0e should be dropped and should never be forwarded.  The Foundry switch floods this frame. So the LLDP frame is received from one FI and is flooded so that the other FI receives a copy of the frame causing the FIs to believe they are connected to each other.  This is why the ports become ErrDisabled.

access-list 401 deny any 0180.c200.000e 0fff.ffff.ffff any access-list

401 permit any any any

interface e4/10

mac access-group 410 in

interface e4/11

mac access-group 410 in

Actual command:

lldp run

The workaround is to apply an inbound MAC access-list on the Foundry on any port which connects to a UCS FI, denying LLDP frames. Once the following configuration changes were made, the interfaces on the UCS and the Foundry switch moved to the Up state.