C9300 trunk not passing tagged traffic

[bdgarcia]

Hello, I have configured a trunk port on a C9300 stack as follows:

interface TenGigabitEthernet3/0/37
switchport trunk native vlan 91
switchport trunk allowed vlan 2-4,81,91
switchport mode trunk
no ip igmp snooping tcn flood

It will pass untagged traffic for vlan 91, but it will NOT pass any tagged traffic.   I have tested with 3 non-cisco switches and I have tested with my laptop with an adapter set to vlan 2.   I also tried with vlan 3 and vlan 4 with no luck.   I have been beating my head on this for over 8 hours and I am completely confused on what I am missing.

I have also tried this with a port-channel (LACP) with this single port and the other switches tested with a port-channel.   Both switches showed the port-channel (LACP) up on both ends and traffic from VLAN 91 passing.

Help!

Thank you

Bryan

[bdgarcia]

FYI te 3/0/37 is up:

NetworkCoreStack#sh interfaces te 3/0/37
TenGigabitEthernet3/0/37 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 3c13.cc27.5725 (bia 3c13.cc27.5725)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 100/1000/2.5G/5G/10GBaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:39, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2797
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3000 bits/sec, 3 packets/sec
5 minute output rate 41000 bits/sec, 36 packets/sec
620600 packets input, 83490639 bytes, 0 no buffer
Received 510933 broadcasts (499116 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 499116 multicast, 0 pause input
0 input packets with dribble condition detected
16755695 packets output, 2336925680 bytes, 0 underruns
Output 12940748 broadcasts (3689490 multicasts)
0 output errors, 0 collisions, 4 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Kasun Bandara]

@bdgarcia hi,

1. is it issue with untagged or tagged traffic? your title mentioned untagged and description says tagged. please help to clarify.

2. how you test tagged traffic? normally to test tagged traffic you need to configure access port which connected to your laptop and trunk need to configure between switches. 

[bdgarcia]

Hello Kasun,

Sorry about the title.  The issue is with tagged traffic.   I have tested with a netgear, a tp-link, and a ubiquiti switch and I have manually set on two different ethernet adapters using windows to use vlan 2, vlan 3, or vlan 4 by setting the vlan tag on the adapter.  You can set this in the driver properties.   I have used this testing method many times in the past.   You do not have to set an access port in this case.

[Kasun Bandara]

@bdgarcia hi is your non-cisco devices capable of passing tagged traffic? 802.1Q?

[bdgarcia]

Good morning, Kasun

The Netgear switch is this Smart Managed Pro Switches - MS510TXPP | NETGEAR

The TP-Link switch is this https://www.tp-link.com/us/business-networking/omada-sdn-switch/tl-sx3008f/

and the Ubiquiti is this https://store.ui.com/us/en/collections/unifi-switching-utility-hi-power-poe/products/switch-enterprise-8-poe

From what I can tell all 3 of these support 802.1q vlans.

I am away from my computer so I can't send you the the specifics of the config screens for the lan adapters from my PC and from the switches.

Thank you

Bryan

[bdgarcia]

Good evening Kasun

1st picture shows VLANs are enabled on the NIC

2nd picture show it is set to VLAN 2.   I have also tried with VLANs 3,4, and 81 with no success

If I disable the VLAN on the PC NIC it does get an IP address from VLAN 91 which is the native VLAN.   I h

[RAdamWilliams]

Oh this is a fun one! I think these images are your problem. Don't set the VLAN on the host device. If you set the VLANs on your host devices then your port connecting that host devices needs to be a trunk port. With the common rhetoric about how access ports and trunk ports operates this seems counter-intuitive so hopefully I can type this out well. 

When you put a port into access mode (or tag a port with other vendors) a switch adds the set tag onto any frame that goes into the port. Importantly for your issue though is that when a frame leaves that port, destined for whatever is plugged into it, the dot1q tag is removed. This means that the host machine will drop all of that traffic since the VLAN tags don't match (or in this case there isn't one). Trunk ports on the other hand only add the native VLAN tag to traffic that is not tagged and leave all tagged traffic unchanged. If this doesn't make sense I'll try and write a more thorough response!

[bdgarcia]

Hello RAdamWilliams,

I am sorry I do not understand your response.   It is my understanding that when I add a native vlan to a trunk that vlan is passed untagged, and I have repeatedly demonstrated that including this case.   When I set vlans as allowed.  In this case with switchport trunk allowed vlan 2-4,81,91 it is my understanding that traffic for vlans 2-4,81 and 91 will be tagged on the trunk and the both switches will expect and traffic placed on the trunk will be tagged for these vlans.   That is why I would expect when I set the ethernet port on my laptop to use vlan id, I would expect to get an address from the DHCP server for that VLAN.

If I am saying the same thing as you sorry for misunderstanding.   If we are saying different things, please help me understand where I am off track.   Specific references would be really helpful.

Thank you

Bryan

[RAdamWilliams]

Let me ask this then, are the PCs you have connected on access ports or trunk ports?

[RAdamWilliams]

Alright, using the very professional MS Paint diagram above let's talk about what the switch actually does when you configure it as an access port vs a trunk port. 

The port is labeled on this diagram because for the purposes of this discussion you can consider it the demarcation line between traffic being "in the switch" (or maybe a better description would be "inside the network") and "out of the switch" (or "outside the network"). 

Access mode: When traffic enters the network (host to switch) the port adds the VLAN tag to the frame for whatever VLAN is set. When traffic leaves the network (switch to host) the port removes the VLAN tag from the frame, making it untagged. Using the image above we can say that everything to the left of the port is always untagged when it comes from the switch, and everything that enters the switch is always tagged (by the switch) with whatever VLAN is configured on the port. Notice that the switch doesn't care about how the traffic looks when it comes in.

This is where I think your problem is. By setting the VLAN ID to 2 on your host network adapter you are telling your host to tag all traffic transiting that interface with VLAN 2. You're also implicitly telling your host to drop any traffic received on that interface that is not tagged with VLAN 2. You can leave "Priority and VLAN" set however you want (default is "Priority and VLAN Enabled") but any time you have a host connected to an access port your VLAN ID should be set to either 0 or "Not Present" because will never receive tagged frames from that port. 

Trunk Port: Trunk ports don't manipulate tags at all, mostly. The point of trunk ports is to preserve VLAN tags or frames that traverse it. There are two common commands that manipulate this behavior. Using "switchport trunk allowed vlan" causes the "port" to inspect the tags of transiting frames and drop any that have a tag not in the allowed list. "switchport trunk native vlan" causes the port to act like an access port for the vlan set in this command. 

This behavior is why I said that you could also fix your problem by having your host connected to a trunk port. Since you're setting your VLAN on the host itself you need the switch to not remove it on the way back to the host and that's exactly what trunk ports do (or don't do). I would say as a best practice though if you're not using a host with a vswitch or something similar don't change your VLAN settings on the host, ever. All of the tagging is the network equipment's responsibility.

[balaji.bandi]

 I have tested with a netgear, a tp-link, and a ubiquiti switch 

is this above switches are managed switches and as asked they understand tagging and untagging ?

[bdgarcia]

Yes

[alancelliott]

There's been some good suggestions so far. Alternatively do you have any config that would drop the traffic I.e. arp inspection. Can you provide output of "show int trunk" so we can see the forwarding status of the interface?

For the purpose of troubleshooting whether or not the switch interface is actually forwarding the tagged frames you could setup a span session with the "problem" port as the source, look at tx and rx packets, and see if frames are getting tagged correctly when traffic is egressing.

[bdgarcia]

Hello alancelliot,

When I do show int trunk I get this:

Port          Mode  Encapsulation Status     Native vlan
Te3/0/37  on       802.1q           trunking  91

Port          Vlans allowed on trunk
Te3/0/37  2-4,81,91

Port           Vlans in spanning tree forwarding state and not pruned
Te3/0/37   2-4,81,91

I will setup the span tomorrow.