03-04-2016 12:48 PM - edited 03-01-2019 12:37 PM
I have the fairly common situation that I need to setup a disjoint layer 2 network on our UCS (UCSM 2.1(3b)) I've been careful to follow the documentation as closely as I can, but as soon as I assign a VLAN to an uplink interface ALL traffic stops routing. The documentation (and the UCSM itself) I've read repeatedly states If no interfaces are implicitly assigned to a VLAN, then default behavior will be implemented & that VLAN will be allowed on all interfaces but what I'm seeing is that no VLANS are allowed anywhere once even a single VLAN is assigned to a specific interface or port channel.
Am I possibly overlooking something in my configuration that I need to do before trying do this? Is there, perhaps, and assumed configuration that I don't know about?
We're using a UCS with a 6248 Fabric Interconnect connected to a Nexus 5548 switch.
I was referencing the following guides when trying to do this.
https://supportforums.cisco.com/sites/default/files/legacy/5/5/3/59355-Disjoint%20L2%20Uplinks.pdf
03-04-2016 11:28 PM
Which UCS version ?
Is FI in Ethernet End Host Mode ?
Is Vlan 1 on all uplinks ?
Please post the CLI output of the uplink trunk configuration.
03-07-2016 08:27 AM
The version information
Software
BIOS: version 3.6.0
loader: version N/A
kickstart: version 5.0(3)N2(2.11.3b)
system: version 5.0(3)N2(2.11.3b)
power-seq: Module 1: version v2.0
Module 3: version v2.0
uC: version v1.2.0.1
SFP uC: Module 1: v1.1.0.0
BIOS compile time: 05/09/2012
kickstart image file is: bootflash:/installables/switch/ucs-6100-k9-kickstart.5.0.3.N2.2.11.3b.bin
kickstart compile time: 12/19/2013 14:00:00 [12/19/2013 16:41:54]
system image file is: bootflash:/installables/switch/ucs-6100-k9-system.5.0.3.N2.2.11.3b.bin
system compile time: 12/19/2013 14:00:00 [12/19/2013 18:25:29]
Hardware
cisco UCS 6248 Series Fabric Interconnect ("O2 32X10GE/Modular Universal Platform Supervisor")
Intel(R) Xeon(R) CPU with 16622556 kB of memory.
Processor Board ID FOC17275VK9
Device name: NCT-DALS-DC-UCSPOD1-B
bootflash: 31266648 kB
Kernel uptime is 643 day(s), 0 hour(s), 37 minute(s), 34 second(s)
Last reset
Reason: Unknown
System version: 5.0(3)N2(2.11.3b)
Service:
plugin
Core Plugin, Ethernet Plugin, Fc Plugin, Virtualization Plugin
It looks like end-host mode is enabled, though I'll admit I may not know where else to look.
(nxos)# show platform software enm internal info global | grep -A 6 'Global Params'
Other Global Params:
end-host-mode: Enabled
fcoe-native-vlan (skip pinning for vlan): 4049
rc = success (code 0x0)
# show fc-uplink
FC Uplink:
Mode
----
End Host
It looks like VLAN 1 is on all physical interfaces, but it isn't on any if the Vethernet interfaces.
(nxos)# show vlan
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Po217, Po218, Eth1/9, Eth1/10
Eth1/11, Eth1/12, Eth1/13
Eth1/14, Eth1/15, Eth1/16
Eth1/19, Eth1/20, Eth1/21
Eth1/22, Eth1/23, Eth1/24
Eth1/1/10, Eth1/1/12, Eth1/1/14
Eth1/1/16, Eth1/1/17, Eth1/1/18
Eth1/1/19, Eth1/1/20, Eth1/1/21
Eth1/1/22, Eth1/1/23, Eth1/1/24
Eth1/1/25, Eth1/1/26, Eth1/1/27
Eth1/1/28, Eth1/1/29, Eth1/1/30
Eth1/1/31, Eth1/1/32
These are the interfaces I want to send traffic to. Eth 1/9 should be separated from everything else. Right now it is not physically connected because I haven't been up to the datacenter to do it. If I use the Uplink Manager in the UCSM GUI to assign VLANs to specific interfaces all non-management traffic stops. (screenshot from the PDF guide)
(nxos)# show running-config interface Port-channel 217
!Command: show running-config interface port-channel217
!Time: Mon Mar 7 10:08:47 2016
version 5.0(3)N2(2.11.3b)
interface port-channel217
description U: Uplink
switchport mode trunk
pinning border
switchport trunk allowed vlan 1,252-258,500
speed 10000
(nxos)# show running-config interface Ethernet 1/19
!Command: show running-config interface Ethernet1/19
!Time: Mon Mar 7 10:08:55 2016
version 5.0(3)N2(2.11.3b)
interface Ethernet1/19
description U: Uplink
pinning border
switchport mode trunk
switchport trunk allowed vlan 1,252-258,500
speed 1000
no shutdown
03-07-2016 09:14 AM
Hi
my 2c
1) I would not use vlan 1 for any interfaces ! Please move any pc and member links to another vlan.
2) you seem to use a rather old UCS release ? 2.1.3b ?
3) why are is Interface Ethernet 1/19 speed 1000 ? I would assume this should be 10'000 ?
4) port channel speed is 10'000 ? I would expect n x 10'000 ?
Walter.
03-07-2016 12:48 PM
1. We don't actually use VLAN 1
2. Our UCS probably hasn't been upgraded since initial deployment (see also: Uptime 643+ days)
3. It's an isolated dev/testing network for an outside entity.
4. Don't know, I didn't configure that.
5. After further investigation it appears I may need to create a new pair of vNICs dedicated to just this VLAN.
From the guide: If a vNIC is defined to carry VLANs belonging to two separate disjoint Layer 2 upstream networks, pinning will fail, and a fault will be raised.
When I created a vNIC I ran into a PCIe re-enumeration warning when trying to add it to the actual server. We have scheduled downtime for our blades later this week, I'll plan to add the new vNIC to one of the hosts then.
03-08-2016 12:12 AM
5) good point !
Nevertheless, please configure all productive interfaces / portchannels out of vlan 1.
vlan 1 is special, cannot be deleted, and by default all interfaces are in vlan 1.
Just stepped over https://www.youtube.com/watch?v=fqCkP1qihwg
and very useful Jeff's experience
http://jeffsaidso.com/2013/04/enm-source-pinning-failed-a-lesson-in-disjoint-layer-2/
I hope this is not your case ?
03-05-2016 05:07 AM
Hi JL,
Can you confirm on your upstream switch that the vlan is being allowed across the necessary ports?
Here is a very good guide:
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/white_paper_c11-692008.html
Let me know the outcome when you review these things.
Regards,
Qiese Dides
03-07-2016 08:33 AM
It's definitely not an upstream problem. When I explicitly assign VLANs to the Port Channel they're already using traffic stops.
I followed the newer versions of the same guide, and the problem is when I use the Uplink Manager to assign VLANs to interfaces. As soon as I click 'Apply' everything stops routing.
03-08-2016 02:23 PM
The solution was to create a new vNIC for my isolated VLAN and then add the vNIC to the server profiles.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide