OK. Now we are getting a little further in terms of troubleshooting. Not any resolution yet, but at least I know more information.
I finally got my USB-Ethernet adapter and set up port mirroring on my linux machine. NOW, I can see tagging going on and I have discovered that tagging IS INDEED being passed around, including from the USG.
In summary, the pings from the USG to the switch on its trunk do get passed with the 172 tag, but fail. I see no response from the client it is trying to ping, so it can be assumed that the packet goes no further than that trunk port as far as I can tell.
(key - 172.16.254.1 (USG), 172.16.254.25 (wireless client on VLAN 172), 172.16.254.10 (wired client on VLAN 172)
This is the traffic on the switch trunk port, with the filters of icmp && ip.addr==172.16.254.1 (or whatever IP address I am looking at) applied.
So you can see, the switch doesn't seem to direct the traffic properly after it's trunk.
However, when I ping from the client on VLAN 172 to the USG, traffic not only passes from the client, through the trunk port to the USG, IT COMES BACK, tagged appropriately.
From client to USG:
And back again (response from USG to client)
How does that even jive with the first capture screenshot above! Clearly traffic can get from the USG to the access ports on the switch through the trunk port, tagged at VLAN 172 (I have a theory, down below).
Here is something weird also. When I ping from the wireless VLAN 172 client to the wired VLAN 172 client, the only traffic I see on the switchport trunk is the response from the gateway back to the wireless client, but nothing else. (Note, the ping was from 172.16.254.54 to 172.16.254.10). I don't know if this is expected behavior (i.e. why doesn't the traffic go through the switch trunk port to the gateway, but this may be due to the Unifi infrastructure. Both the USG and the APs are managed under the UniFi controller, so perhaps there is something working on the back end with the controller. Not sure about this one)
Here is the output from a wired 172 client to the wireless 172 client
When I try to ping from wired client to wired client, I get nothing at all on the trunk port. Additionally, even when I drill down and monitor JUST the port the wired client that is the source of the ping, I get no traffic at all. I do get traffic on a successful ping to the gateway of 172.16.254.1 however .
One of my theories (besides the switch just being screwed up, or has a major bug), is that the arp tables are causing problems, which reside on the USG and I think that is because the USG can't get to the switch. All of my 172. entries are listed as incomplete.
The reason I say this, is that the arp entries on the USG are all listed as incomplete:
However, that is a thin theory at best, because when I inspect the dst portion of the packet of the failed USG to wireless client, it has the proper MAC address.
So, what I have sort of proven, I think, is that the packets are tagged, but don't make it from the USG side to the switch side, when initiated from the USG side. However, the packets are also tagged but DO make it from the USG side to the switch side, if the traffic is initiated from the switch side.
This is a tough one for sure. I have more information but less answers!
EDIT: After further testing this evening, I have discovered the same problems exist with any VLAN. I tested using 2 other different VLANs (using different subnets) and I get the same exact results, which are:
Traffic flows from VLAN to untagged (native) subnet, it does not flow the opposite way. Anything that is tagged does not seem to be working at all.
EDIT: You can certainly read through all of my replies below to see the madness that I descended into, but if you want to know what I found the problem was and hopefully get a good laugh, jump to the last reply.
I agree that I think the arp is significant because the underlying problem can be summed in in that "the traffic cant get to its destination because the destination is unknown" Since routes look correct, ARP entries are the only other real decision point. Based on the response from the ICMP, it seems to be getting to the network just fine (so routing is fine), but it can't get to the host on that network...well, because it doesn't know where the heck it is at.
Here is something I came up with on a tcpdump of the USG on the VLAN 172 interface. From it, I gather 2 things:
1.) It does not appear that the USG is responding to the ARP request that the RPi (wired 172 client) is so desperately asking for. As you can see below, the request comes into the USG, but it never goes back out with information.
2.) This arp request has no tag, even though it is on an access port for 172 and it traverses the tagged trunk from the switch.
Granted, I am assuming the tcpdump is including that information since the USG is running linux kernel, though unsure if there is some sort of setting to turn that off, but I bet not. Although, I do see it tagged on the trunk port of the switch going out. So, since the USG port is effectively a tagged port, when I see he traffic come in, I should still see the tag, no?
The above is the USG dump, I failed to screenshot the frame info below it, showing no tag, but believe me when I say it isn't there.
Here is the same traffic on the switchport trunk, and it is tagged.
Just when I was about ready to throw in the towel, new info came up! I will try your suggestion. While I am back and forth, I now am thinking that the problem is in the USG, some sort of bug. I may try rolling back the firmware at some point to see if that solves anything. Sadly, I updated the firmware just before implementing the VLANs, so no clue if the same problem existed before the update. I will also look to make sure the USG should be capturing TAG info.
So new info. Either the USG isn't displaying the 802.1q tagging or it is somehow stripped before the packet is capture (which I don't understand why it would do that or how).
I setup a mirror both on the switch trunk port and also ran a USG tcpdump in tandem.
I pinged from the USG (172.16.254.1) to a wireless client (a new one, which is my mobile phone, 172.16.254.26). First interesting thing, this responded to ping appropriately (and is on a DIFFERENT AP than my laptop hooks to, which may be something). As you can see, the traffic on the trunk port shows tagged, as expected:
On the USG, it shows the successful frames, but does not have them tagged
So, I ALSO pinged from the USG to the wired client (172.16.254.10). It does not show up on the switchport trunk capture, but here is the kicker - IT DOESN"T SHOW UP ON THE USG PORT EITHER!
This tells me, the USG is thinking, OK, Ping this address, check ARP for where it is, (arp says incomplete), has no idea where to send it, so doesn't send it out of the USG port. This is significant, I believe. At least in narrowing down where the problem might lie. I wonder if I can statically add an ARP entry, but that still may not help. I know I can on Windows (which didn't help, tried yesterday), but maybe doing so on the USG might do something. At least so we could say once and for all, it is for sure related to ARP.
There is not an entry at all in the USG for this wired device
So I tried your suggested test.
I have a completely different VLAN (ID 2). Trunks are passing. USG doing DHCP. I have 2 fixed wire ports, both access VLAN 2.
Connected, and immediately received IP addresses for both. 10.100.100.6 and 10.100.100.7.
When I ping from 10.100.100.6 to 10.100.100.7, I get replies appropriately.
However, when I ping from 10.100.100.7 to 10.100.100.6, I get the same unreachable error! I do not see this traffic on the trunk port of the switch, which as you mentioned, is expected.
Here is the arp entries in windows:
That is from the 10.100.100.7 pc. I am not going to include the other PC because it would be a pain to do so, but it looks basically the same. It has the entry for 10.100.100.7.
I also completely turned off the APs just to be sure they weren't somehow interfering, made no difference.
I feel I may have exhausted all options at this point. Everything seems so random, in a specific way, if that makes sense.
While I go back and forth with where the problem lies, I am not back to the issue being on the switch somehow.
Strangely, I get the following entries on the switchport trunk, I am not initiating this traffic.
The 10.100.100.1 is the USG.
The really weird thing is that if I actually initiate a ping from the gateway to the 10.100.100.7 client, it is successful. If I ping from 10.100.100.6 to the USG on 10.100.100.1, it is successful, but if I ping 10.100.100.6 from the USG, it fails!
I keep finding this message in the wireshark logs and I haven't quite figured it out. Meaning, I basically know what it is, but not sure why it is happening. Not sure if it is related to an actual issue, or if the issue is because the USG doesn't actually have a fully formed ARP entry for this (wire VLAN 172) client.
I do see ARP broadcasts flowing from the client looking for the USG and I also see ARP broadcasts coming from the USG asking for the address of the client, but never see any answers either way.
That source is the USG and the client is just Linux server on my network, but it is going across VLAN 1.
