Nexus 3172TQ Trunking Private Vlans

[Bgedney]

I'm testing rolling out Private VLANs for VM Isolation in our VMWare Environment.  I've created a primary VLAN and an Isolated VLAN and associated the Isolated VLAN from the Primary VLAN on our Nexus 3172TQ.  

I then added them to the uplink trunks to the physical VMWare Hosts. I created the private VLANs in VMWare and created associated Distributed Port groups.  

When I take and add a VM to the port group to pull a DHCP lease, I get nothing- so I decided to grep 

show mac address-table

for the VM's mac- it doesn't show up.  I moved it to the primary private VLAN, then it shows up in mac address table of the switch.  (the vlans are created the same ways in VMWare- by defining them under private vlan configuration for the specific switch).

My question comes down to this- I was reading in the Layer 2 Switching Configuration Guide, and it discusses creating separate trunks for isolated and primary private vlans, but Its not clear if thats just an option, or is required.

If not, I'm not sure what to t-shoot next...  Any feedback on the separate trunk requirements/further troubleshooting suggestions?

Bill

[Peter Paluch]

Hi Bill,

Is there any chance you could post a sanitized configuration of your device here? The steps you have taken seem to be correct and do not point out any obvious problem, so we need to start from the ground up.

Best regards,
Peter

[Bgedney]

Peter-

Sure.  

The primary vlan is 1110 and the isolated vlan is 1111.

NEXUX-001# show run vlan 1110-1111

!Command: show running-config vlan 1110-1111
!Time: Thu Mar  1 14:48:18 2018

version 7.0(3)I2(2b)
vlan 1110
  name PLATFORM-DEV-P
  private-vlan primary
  private-vlan association 1111
vlan 1111
  name PLATFORM-DEV-I
  private-vlan isolated

The vlans are both on E1/21 (which is one of a few trunks to our vmware hosts).

NEXUX-001# show run inte e1/21

!Command: show running-config interface Ethernet1/21
!Time: Thu Mar  1 14:57:56 2018

version 7.0(3)I2(2b)

interface Ethernet1/21
  description SPAN VS1
  switchport mode trunk
  switchport trunk native vlan 999
  switchport trunk allowed vlan 1110-1111

I have 4 hosts in this VMWare infrastructure and I run CDP on the link.  Here is limited CDP information on the VDS from VMWare:

Cisco Discovery Protocol
	Version	2
	Timeout	60
	Time to live	146
	Samples	80477
	Device ID	NEXUX-001(XXXXXXXXXXX)
	IP address	XXX.XXX.XXX.XXX
	Port ID	Ethernet1/21
	Software version	Cisco Nexus Operating System (NX-OS) Software, Version 7.0(3)I2(2b)
	Hardware platform	N3K-C3172TQ-10GT
	IP prefix	0.0.0.0
	IP prefix length	0
	VLAN	999
	Full Duplex	Enabled
	MTU	1500
	System name	NEXUX-001
	System OId	1.3.6.1.4.1.9.12.3.1.3.1389
	Management address	XXX.XXX.XXX.XXX
	Location	

Peer device capability
	Router	Enabled
	Transparent bridge	Disabled
	Source route bridge	Disabled
	Network switch	Enabled
	Host	Disabled
	IGMP	Enabled
	Repeater	Disabled

 Here is a copy of the configuration details for the DVPG - PLATFORM-DEV-I (I for isolated):

General
	Name:	PLATFORM-DEV-I
	Port binding:	Static binding
	Port allocation:	Elastic
	Number of ports:	8
	Network resource pool:	(default)

Advanced
	Configure reset at disconnect:	Enabled

Override port policies
	Block ports:	Allowed
	Traffic shaping:	Disabled
	Vendor configuration:	Disabled
	VLAN:	Disabled
	Uplink teaming:	Disabled
	Security policy:	Disabled
	NetFlow:	Disabled
	Traffic filtering and marking:	Disabled

Security
	Promiscuous mode:	Reject
	MAC address changes:	Reject
	Forged transmits:	Reject

Ingress traffic shaping
	Status:	Disabled
	Average bandwidth:	--
	Peak bandwidth:	--
	Burst size:	--

Egress traffic shaping
	Status:	Disabled
	Average bandwidth:	--
	Peak bandwidth:	--
	Burst size:	--

VLAN
	Type:	Private VLAN
	Private VLAN ID:	Isolated (1110, 1111)

Teaming and failover
	Load balancing:	Route based on originating virtual port
	Network failure detection:	Link status only
	Notify switches:	Yes
	Failback:	Yes
	Active uplinks:	Link1
	Standby uplinks:	
	Unused uplinks:	

Monitoring
	NetFlow:	Disabled

Traffic filtering and marking
	Status:	Disabled

Miscellaneous
	Block all ports:	No

And here is the primary private vlan for reference - PLATFORM-DEV-P (P for promiscuous/Primary):

General
	Name:	PLATFORM-DEV-P
	Port binding:	Static binding
	Port allocation:	Elastic
	Number of ports:	8
	Network resource pool:	(default)

Advanced
	Configure reset at disconnect:	Enabled

Override port policies
	Block ports:	Allowed
	Traffic shaping:	Disabled
	Vendor configuration:	Disabled
	VLAN:	Disabled
	Uplink teaming:	Disabled
	Security policy:	Disabled
	NetFlow:	Disabled
	Traffic filtering and marking:	Disabled

Security
	Promiscuous mode:	Reject
	MAC address changes:	Reject
	Forged transmits:	Reject

Ingress traffic shaping
	Status:	Disabled
	Average bandwidth:	--
	Peak bandwidth:	--
	Burst size:	--

Egress traffic shaping
	Status:	Disabled
	Average bandwidth:	--
	Peak bandwidth:	--
	Burst size:	--

VLAN
	Type:	Private VLAN
	Private VLAN ID:	Promiscuous (1110, 1110)

Teaming and failover
	Load balancing:	Route based on originating virtual port
	Network failure detection:	Link status only
	Notify switches:	Yes
	Failback:	Yes
	Active uplinks:	Link 1
	Standby uplinks:	
	Unused uplinks:	

Monitoring
	NetFlow:	Disabled

Traffic filtering and marking
	Status:	Disabled

Miscellaneous
	Block all ports:	No

The VM I'm testing with has a mac address of XXXX.XXXX.645d (and now that I'm checking later, I'm noticing it does show up under the primary promiscuous VLAN [previously, I'd moved the VM from PLATFORM-DEV-I to PLATFORM-DEV-P to compete the install of the test OS and it was showing up in the primary promiscuous vlan when running a show mac address-table, but I'd assumed that the arp entry hadn't timed out.]) When I do put the VM into PLATFORM-DEV-P, I get a DHCP lease from the Firewall pair. 

So- as of right now, show mac address table | grep 645d gives me: 

NEXUX-001# show mac address-table | grep 645d
* 1110     XXXX.XXXX.645d   dynamic  0         F      F    Eth1/21

but the VM isn't able to talk to the DHCP server which is on my pair of Firepower 2130s- off of a pair of port channels (Po-13 and Po-14 with Po-10 being the Peer link).

NEXUX-001# show run int po13-14

!Command: show running-config interface port-channel13-14
!Time: Thu Mar  1 15:24:17 2018

version 7.0(3)I2(2b)

interface port-channel13
  description FW-01
  switchport mode trunk
  switchport trunk native vlan 999
  switchport trunk allowed vlan xxxx,1110-1111
  spanning-tree port type edge trunk
  vpc 13

interface port-channel14
  description FW-02
  switchport mode trunk
  switchport trunk native vlan 999
  switchport trunk allowed vlan xxxx,1110-1111
  spanning-tree port type edge trunk
  vpc 14

I get the feeling I'm not trunking correctly.

Thanks again for your help Peter.

Bill

[Peter Paluch]

Hi Bill,

Awesome, thank you very much!

On Nexus N3K/N9K, the MAC address of a host in a secondary VLAN (community or isolated) will still be shown in the show mac address-table as being learned on the primary VLAN. That makes the output quite confusing but sadly, it is what it is.

Either way, I believe that the problem in having your VM talk to the Firepower boxes on port-channels 13 and 14 is indeed in their trunk mode. Regular trunks do not rewrite the VLAN tags - whatever VLAN the frame was received in anywhere in the network, this VLAN's tag will be maintained on a regular trunk. In particular, if your VM is placed into VLAN 1111, its traffic will be tagged by 1111 when it leaves Po13/Po14 torward the Firepower. Unless the Firepower also understands Private VLANs and is able to understand that VLAN 1111 is associated with VLAN 1110, it will not be able to process this frame properly. That would explain your connectivity issues.

The solution would be to change Po13 and Po14 into promiscuous PVLAN trunk ports. A promiscuous PVLAN trunk port rewrites all secondary PVLAN tags on outgoing frames into the primary PVLAN. To the attached device, all frames will appear to be originating from the primary PVLAN. Regular VLANs will continue working on this trunk type normally.

This means that the Po13 and Po14 configuration would need to be changed as follows:

interface port-channel13
  description FW-01
  switchport mode private-vlan trunk promiscuous
  switchport trunk native vlan 999
  switchport trunk allowed vlan xxxx,999,1110-1111
  switchport private-vlan mapping trunk 1110 1111
  spanning-tree port type edge trunk
  vpc 13

interface port-channel14
  description FW-02
  switchport mode private-vlan trunk promiscuous
  switchport trunk native vlan 999
  switchport trunk allowed vlan xxxx,999,1110-1111
  switchport private-vlan mapping trunk 1110 1111
  spanning-tree port type edge trunk
  vpc 14

You can check a couple of more details here:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/layer2/7x/b_Cisco_Nexus_3000_Layer_2_Switching_Config_7x/b_Cisco_Nexus_3000_Layer_2_Switching_Config_7x_chapter_011.html#task_w45_mlt_qx

Since these are virtual port-channels, you will obviously need to configure both vPC peers identically to make this configuration work.

Can you give this a try? Please let me know!

Best regards,
Peter

[Bgedney]

Peter-

I'm having an issue configuring the trunk interface.  My only options under switchport mode private-vlan are host or promiscuous:

NEXUS-002(config-if)# switchport mode private-vlan ?
  host         Port mode pvlan host
  promiscuous  Port mode pvlan promiscuous

 And switchport mode private-vlan promiscuous is complete commmand:

NEXUS-002(config-if)# switchport mode private-vlan promiscuous ?
  <CR>  

 Not sure if there is a parallel option.

Thanks,

Bill

[Peter Paluch]

Hi Bill,

Let me see if I can quickly snatch a lab device identical to yours and check if the command is supported there. Sorry to say - but the particular limitations across the Nexus 3000 platform family are very varied and hard keep track of :-\

Will keep you posted.

Best regards,
Peter

[Peter Paluch]

Hi Bill,

Quickly testing a 3172TQ device running 7.0(3)I7(1), I can positively confirm that the switchport mode private-vlan trunk promiscuous was available. Unfortunately, that switch was used by another colleague, and I was not able to do any changes to it, so I could not test your particular NX-OS version and a "switch mode" setting I will describe in a moment.

You are running 7.0(3)I2(2b) which is quite outdated. However, I also checked the source code briefly, and this command is there already.

I therefore suspect that the unavailability of this command on your switch is likely due to a switch mode that presents you with a limited set of available commands on the 3172TQ - we call it amply the "n3k" mode. The N3100 series can also operate in the "n9k" mode in which the CLI set is equivalent to the Nexus 9000 series CLI set.

The particular switch mode can be changed in the configuration using the system switch-mode n9k command - unfortunately, changing this mode will require you to go through "write erase" and "reload" cycle before it becomes effective, and you will obviously need to restore the configuration afterward, so this is certainly something that can only be done in a maintenance window.

I am sorry for this setback - but it seems that this step will be inevitable.

Perhaps it would be advantageous to think about upgrading the NX-OS, too, if you need to go through the "write erase" and "reload" - the I2(2b) release is truly dated. The current recommended version is 7.0(3)I4(7) based on the following doc:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/recommended_release/b_Minimum_and_Recommended_Cisco_NX-OS_Releases_for_Cisco_Nexus_3000_Series_Switches.html

Best regards,
Peter

[Bgedney]

So, after a bit of time arranging a maintenance window to move our Nexus 3172TQ switches from N3K mode to N9K mode, I can confirm that it still does not support:

switchport mode private-vlan

and when I try to ignore the fact that there is no private-vlan under switchport mode and proceed to add:

 switchport private-vlan mapping trunk 1110 1111

I get an error message stating

ERROR: Port-Channel 13 : requested config not allowed on Port-Channels

Are there any other options? I tried this on the underlying member physical interfaces, but I can't even set a switchport mode.  

Thanks,

Bill

[Bgedney · Accepted solution]

So- let me clarify.  

Under the N9k mode, individual interfaces do have the option to enable 

switchport mode private-vlan trunk promiscuous

On N9k mode, port channels operate the same as n3k port channels do.

Under the default N3k mode, the only switchport mode options under a port channel are

• access
• dot1q-tunnel
• fex-fabric
• trunk

and an individual interface are, there is a switchport mode private-vlan but trunk is not an option.  Only 

• host
• promiscuous

are options.

When I get a maintenance window, I'm going to update the switches to the most current NX-OS.  If this changes after the update, I'll post an update. But until now, I'm going to mark this as the solution.

Bill