cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1149
Views
4
Helpful
8
Replies

Cannot create disjoint layer 2 VLANS - all traffic stops.

JLCrabtree
Level 1
Level 1

I have the fairly common situation that I need to setup a disjoint layer 2 network on our UCS (UCSM 2.1(3b)) I've been careful to follow the documentation as closely as I can, but as soon as I assign a VLAN to an uplink interface ALL traffic stops routing. The documentation (and the UCSM itself) I've read repeatedly states If no interfaces are implicitly assigned to a VLAN, then default behavior will be implemented & that VLAN will be allowed on all interfaces but what I'm seeing is that no VLANS are allowed anywhere once even a single VLAN is assigned to a specific interface or port channel.

Am I possibly overlooking something in my configuration that I need to do before trying do this? Is there, perhaps, and assumed configuration that I don't know about?

We're using a UCS with a 6248 Fabric Interconnect connected to a Nexus 5548 switch.

I was referencing the following guides when trying to do this.

https://supportforums.cisco.com/sites/default/files/legacy/5/5/3/59355-Disjoint%20L2%20Uplinks.pdf

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-0/b_UCSM_GUI_Configuration_Guide_2_0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_010101.html#concept_4E7BF4F5FA754183B5FE3298755800A1

8 Replies 8

Walter Dey
VIP Alumni
VIP Alumni

Which UCS version ?

Is FI in Ethernet End Host Mode ?

Is Vlan 1 on all uplinks ?

Please post the CLI output of the uplink trunk configuration.

The version information

Software
BIOS: version 3.6.0
loader: version N/A
kickstart: version 5.0(3)N2(2.11.3b)
system: version 5.0(3)N2(2.11.3b)
power-seq: Module 1: version v2.0
Module 3: version v2.0
uC: version v1.2.0.1
SFP uC: Module 1: v1.1.0.0
BIOS compile time: 05/09/2012
kickstart image file is: bootflash:/installables/switch/ucs-6100-k9-kickstart.5.0.3.N2.2.11.3b.bin
kickstart compile time: 12/19/2013 14:00:00 [12/19/2013 16:41:54]
system image file is: bootflash:/installables/switch/ucs-6100-k9-system.5.0.3.N2.2.11.3b.bin
system compile time: 12/19/2013 14:00:00 [12/19/2013 18:25:29]


Hardware
cisco UCS 6248 Series Fabric Interconnect ("O2 32X10GE/Modular Universal Platform Supervisor")
Intel(R) Xeon(R) CPU with 16622556 kB of memory.
Processor Board ID FOC17275VK9

Device name: NCT-DALS-DC-UCSPOD1-B
bootflash: 31266648 kB

Kernel uptime is 643 day(s), 0 hour(s), 37 minute(s), 34 second(s)

Last reset
Reason: Unknown
System version: 5.0(3)N2(2.11.3b)
Service:

plugin
Core Plugin, Ethernet Plugin, Fc Plugin, Virtualization Plugin

It looks like end-host mode is enabled, though I'll admit I may not know where else to look.

(nxos)# show platform software enm internal info global | grep -A 6 'Global Params'
Other Global Params:
end-host-mode: Enabled
fcoe-native-vlan (skip pinning for vlan): 4049
rc = success (code 0x0)

# show fc-uplink

FC Uplink:
Mode
----
End Host

It looks like VLAN 1 is on all physical interfaces, but it isn't on any if the Vethernet interfaces.

(nxos)# show vlan

VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Po217, Po218, Eth1/9, Eth1/10
Eth1/11, Eth1/12, Eth1/13
Eth1/14, Eth1/15, Eth1/16
Eth1/19, Eth1/20, Eth1/21
Eth1/22, Eth1/23, Eth1/24
Eth1/1/10, Eth1/1/12, Eth1/1/14
Eth1/1/16, Eth1/1/17, Eth1/1/18
Eth1/1/19, Eth1/1/20, Eth1/1/21
Eth1/1/22, Eth1/1/23, Eth1/1/24
Eth1/1/25, Eth1/1/26, Eth1/1/27
Eth1/1/28, Eth1/1/29, Eth1/1/30
Eth1/1/31, Eth1/1/32

These are the interfaces I want to send traffic to. Eth 1/9 should be separated from everything else.  Right now it is not physically connected because I haven't been up to the datacenter to do it. If I use the Uplink Manager in the UCSM GUI to assign VLANs to specific interfaces all non-management traffic stops.  (screenshot from the PDF guide)

(nxos)# show running-config interface Port-channel 217

!Command: show running-config interface port-channel217
!Time: Mon Mar 7 10:08:47 2016

version 5.0(3)N2(2.11.3b)

interface port-channel217
description U: Uplink
switchport mode trunk
pinning border
switchport trunk allowed vlan 1,252-258,500
speed 10000

(nxos)# show running-config interface Ethernet 1/19

!Command: show running-config interface Ethernet1/19
!Time: Mon Mar 7 10:08:55 2016

version 5.0(3)N2(2.11.3b)

interface Ethernet1/19
description U: Uplink
pinning border
switchport mode trunk
switchport trunk allowed vlan 1,252-258,500
speed 1000
no shutdown

Hi

my 2c

1) I would not use vlan 1 for any interfaces ! Please move any pc and member links to another vlan. 

2) you seem to use a rather old UCS release ? 2.1.3b ?

3) why are is Interface Ethernet 1/19 speed 1000 ? I would assume this should be 10'000 ?

4) port channel speed is 10'000 ? I would expect n x 10'000 ?

Walter.

1. We don't actually use VLAN 1

2. Our UCS probably hasn't been upgraded since initial deployment (see also: Uptime 643+ days)

3. It's an isolated dev/testing network for an outside entity.

4. Don't know, I didn't configure that.

5. After further investigation it appears I may need to create a new pair of vNICs dedicated to just this VLAN.

  From the guide: If a vNIC is defined to carry VLANs belonging to two separate disjoint Layer 2 upstream networks, pinning will fail, and a fault will be raised.

When I created a vNIC I ran into a PCIe re-enumeration warning when trying to add it to the actual server.  We have scheduled downtime for our blades later this week, I'll plan to add the new vNIC to one of the hosts then.

5) good point !

Nevertheless, please configure all productive interfaces / portchannels out of vlan 1.

vlan 1 is special, cannot be deleted, and by default all interfaces are in vlan 1.

Just stepped over https://www.youtube.com/watch?v=fqCkP1qihwg

and very useful Jeff's experience

http://jeffsaidso.com/2013/04/enm-source-pinning-failed-a-lesson-in-disjoint-layer-2/

I hope this is not your case ?

Qiese Dides
Cisco Employee
Cisco Employee

Hi JL,

Can you confirm on your upstream switch that the vlan is being allowed across the necessary ports?

Here is a very good guide:

http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/white_paper_c11-692008.html

Let me know the outcome when you review these things.

Regards,

Qiese Dides

It's definitely not an upstream problem.  When I explicitly assign VLANs to the Port Channel they're already using traffic stops.

I followed the newer versions of the same guide, and the problem is when I use the Uplink Manager to assign VLANs to interfaces. As soon as I click 'Apply' everything stops routing.

JLCrabtree
Level 1
Level 1

The solution was to create a new vNIC for my isolated VLAN and then add the vNIC to the server profiles.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: