Cannot get VLAN Working SG500 (L2) with UniFi USG (L3)

[shelzmike]

As we can all probably relate, I am at my wits end at this point with this setup. It is in a home lab, so nothing major, but it should be working. I cannot tell where my problem lies. I am quite familiar with Cisco, but mainly Catalyst and haven't really ever worked with an SG500 before, but it doesn't seem that much different. I am quite new to the Unifi platform though. Because of that, I am not sure where my problem is.

Since the Unifi USG handles L3 routing pretty darn efficiently and by default with a network and VLAN creation, I am using the SG500 in L2 mode for simplicity.

Currently most everything is on the default VLAN 1. What I want to do is create a VLAN that I plan on using to test PiHole. Since I don't want every device to use it as its DNS and I eventually want the PiHole to also handle DHCP for that VLAN subnet, I am creating a separate VLAN for it. (Note, currently, for initial setup, I have the PiHole set to a static IP, and the Unifi SG is handling DHCP.

Here is the topology:

Unifi:

VLAN 1 - 192.168.10.0/24

GW - 192.168.10.5

VLAN 172 - 172.16.254.0/26

GW - 172.16.254.1

(I currently also have a hidden SSID that is on the VLAN ID 172 as well).

SG500:

(Note, I have other VLANs created the same way that I don't mention in the post. This is because these are in the same boat and will be used for other things later. Once I get this one figured out, those will be setup the same way)

The client is on Gi1/9

The USG is on Gi1/11

The APs are on Gi1/12 and Gi1/20

switch01#show running-config

config-file-header
switch01
v1.4.8.6 / R800_NIK_1_4_202_008
CLI v1.0
set system mode switch queues-mode 4 

file SSD indicator encrypted
@
ssd-control-start
ssd config
ssd file passphrase control unrestricted
no ssd file integrity control

!
vlan database
vlan 2,172,192,2092
exit
voice vlan oui-table add 0001e3 Siemens_AG_phone________
voice vlan oui-table add 00036b Cisco_phone_____________
voice vlan oui-table add 00096e Avaya___________________
voice vlan oui-table add 000fe2 H3C_Aolynk______________
voice vlan oui-table add 0060b9 Philips_and_NEC_AG_phone
More: <space>,  Quit: q or CTRL+Z, One line: <return> 
                                                      
voice vlan oui-table add 00d01e Pingtel_phone___________
voice vlan oui-table add 00e075 Polycom/Veritel_phone___
voice vlan oui-table add 00e0bb 3Com_phone______________
ip dhcp relay enable
hostname switch01
line console
exec-timeout 30
exit
line ssh
exec-timeout 30
exit
line telnet
exec-timeout 30
exit
logging host 192.168.10.10 severity debugging
logging buffered debugging
logging origin-id ip
username admin password encrypted 580078003fd4025c65e privilege 15
username cisco password encrypted 8f0eeb8542065ada069 privilege 15
ip ssh server
ip ssh password-auth
ip ssh pubkey-auth auto-login
                                                      
crypto key pubkey-chain ssh
user-key admin rsa
key-string row AAAAB3NzaC1yc2EAAAABJQAAAQEA
key-string row evmMERjGzDKMdXR0OMlaRItT40Rn
key-string row Qtdn0CUaFVHa8coRECpa/xrYWMrY
key-string row OBY667ReQoTi0RAWWEO7HZfu
key-string row oLWmmlvc1GVYekwjk5JSvgpxxbfP
key-string row 0wieTa0pCFVyuOIkuJmL4d++V9uN
key-string row vgJSzRLcZaZQCc+Pq8zcFbG9Z34G
key-string row LwHTaGgNSvOUa41l1qyzhjSCyg7g
key-string row ky1CqaPEpAbtnCxz076SNFG79gDj
key-string row jnQykPVxkTbmU2yydw==
exit
exit
snmp-server server
snmp-server location Haer
snmp-server contact "Mike"
snmp-server community int_m ro 192.168.10.10 view Default
ip http timeout-policy 1800
clock timezone " " 0 minutes 0
clock summer-time web recurring usa
!
                                                      
interface vlan 1
 ip address 192.168.10.2 255.255.255.0
 no ip address dhcp
 no ipv6 address autoconfig
 no ipv6 unreachables
 no ipv6 dhcp client stateless
!
interface vlan 2
 name netman
 shutdown
!
interface vlan 172
 name pihole
!
interface vlan 192
 name primary-lan
 shutdown
!
interface vlan 2092
 name iot
 shutdown
!
                                                      
interface gigabitethernet1/9
 description RaspberryPi
 switchport mode access
 switchport access vlan 172
!
interface gigabitethernet1/11
 switchport trunk allowed vlan add 2,172,192,2092
!
interface gigabitethernet1/12
 switchport trunk allowed vlan add 2,172,192,2092
!
interface gigabitethernet1/20
 switchport trunk allowed vlan add 2,172,192,2092
!
exit

ip dhcp snooping
switch01#

The problem I am having is as follows. When I connect a laptop to the SSID with VLAN 172, I get an IP Address from the USG. I can ping both the 172.16.254.1 and 192.168.10.5 addresses (by default, USG is all inter-vlan allowed..will change that later)

I cannot ping the PiHole client from the laptop, nor can I ping it from the USG.

I cannot ping anything from the PiHole client, not even the 172.16.254.1 address.

From the switch I am able to ping 192.168.10.5 but cannot ping 172.16.254.1 (immediately get PING: net-unreachable).

I believe my problem is at the link between the USG and the switch. However, my frustration is that I do not know where the problem lies - if it is on the switch side or the USG side. While I'd appreciate an actual solution, if, at a minimum someone can at least confirm, yes, my switch config is correct and should be working, then I think I will at least know my problem is at the USG.

Thanks!

[PH42420]

I've done something like this in the past, but not using the SG500 in L2 mode.

I was using this device behind an ISP router from ATT, circa 2013 time-frame.

I can't confirm your switch config is correct, but I can tell you this can work in L3 mode.

In this prior setup I configured a port to use Port Address Translation, which was connected to the edge router. Each wireless access point was provided it's own sub-net with a default route leading to the internet and no routes provided between sub-nets (shared internet access, with preservation of VLAN isolation).

I don't have access to that router/config right now, or I could post it for you. I've preferred using the SG500's routing function because device traffic can be managed at a finer level, allowing more filtering of traffic prior to packets reaching the edge router's WAN interface, and allowing for more devices to share a given internet connection. I understand this may not be the conventional/preferential way of doing things in the wider community, for other reasons.

[Richard Burts]

One issue described in your post seems straight forward: "From the switch I am able to ping 192.168.10.5 but cannot ping 172.16.254.1". Looking through the posted config I do not see that there is any configuration for default-gateway for your L2 switch. With no default gateway on a L2 switch the expected behavior is that the switch is able to access IP addresses in the subnet of its vlan interface but not access IP addresses in other subnets/networks.

If this were a Catalyst switch I would ask for the output of show interface status to be able to verify the interface that your pi is connected to. I am not sure what the equivalent is on the SG (or even if there is an equivalent). Does the SG have a show command that would provide information about g1/9? Does the pi have an equivalent to arp -a? If so would you post the content of the arp table on the pi? 

[shelzmike]

What are saying does make sense; however, with the SG500, this is a little less clear. I have 2 options, with the default being User Defined (or not) .

I specified it, but even with it marked None (which is the default), it still listed the "Operational Gateway" as the same IP, so not sure what difference it made. Well, actually, it made no difference!

As far as the show run int command, it is the same on the SG, output below.

interface gigabitethernet1/9
 description RaspberryPi
 switchport mode access
 switchport access vlan 172
!

I am sure there is an equivalent arp -a on the RPi, but the problem is I can't quite get to it easily at the moment :) (Hence the problem). I need to get out a monitor and keyboard to check, which I won't be able to do until later on.

[Richard Burts]

Thanks for the additional information. It is interesting that the screen shot does show a default gateway. But that did not show up in the output of show run, which is surprising. I see that there is an "apply" button on that screen. Is it possible that the default gateway was not applied?

Thanks for the show run for the interface. Does that switch support the command show interface status? If so please post its output.

Another thought is that the output of show vlan would be helpful (if that switch supports that command).

[shelzmike]

So I finally fired up Wireshark and found some information.

First, I started a sniff on the RPi port only and tried to ping from several places. Never saw it get there at all.

Then, I switched to sniffing on the trunk port from the switch to the USG and I was able to see the ICMP packets getting to the switch, but they go no further, so at least I know it is getting there.

I have never really investigated VLAN tags in Wireshark and will have to continue to research, an initial search seems as though the packets aren't actually being tagged FROM the UniFi USG, which would for sure cause this problem.

It is highly possible that I am not reading Wireshark properly as it relates to VLAN tags though. I'll keep looking.

[Richard Burts]

In the wireshark did you see any frames being sent from the RPi? I am wondering about the possibility that there is some problem with that connection.

If wireshark is capturing traffic to/from g1/9 I would not expect to see any vlan tags since it is an access port. If wireshark is looking at any of the trunk interfaces then I would expect to see vlan tags, and I would expect that wireshark would recognize and interpret them.

[shelzmike]

To clarify, when monitoring the RPi port, I was seeing no traffic, at all. I wasn't looking for tags here, just traffic in general, and there was none.

When monitoring the trunk port, I was seeing the traffic, but no tags. Then I remembered that I am on a Windows machine and 802.1q headers are sometimes captured and sometimes not on Windows machines. I cannot tell in this instance. Therefore, I will have to get Wireshark on a linux box to know I should be seeing tags, if they are being sent. Don't have any extra NICs on my Linux box and it is headless, so I can't mirror remotely as I cannot connect if the same NIC used for management is the same that is used for mirroring.

Had to order a USB NIC to use with it that should be here tomorrow. I was wondering though, can a linux VM on a Windows machine WITH a dedicated NIC possibly work? Not sure how the drivers work in that instance.

I also did some further testing.

When I have a wireless client connect to the SSID that is on VLAN 172, I DO reach the USG and get a DHCP address. I can ping from the client to the USG 172.16.254.1 address. I do see that traffic on the trunk port, so it is flowing through the AP port and then through the trunk port. However, the opposite is not true. I am unable to ping from the USG to the client. I see the traffic on the trunk port, but with host unreachable. It is possible this is a USG problem.

I may end up just saying screw it and put the SG500 in L3 mode and have better control and insight on the routing. Still, I don't like to solve problems like that, which isn't an actual solution at all. It should work like this, and I really want to figure out why it isnt. 

Oh, I also pinged from the USG to the RPi while sniffing the trunk port and I didn't see the traffic hit the switch port from the gateway at all, which also lends itself to the gateway being the problem possibly.

[Richard Burts]

Thanks for the update. Interesting that a client connected wireless can ping USG but USG can not ping the client. Is there any possibility of a firewall or security policy on the client that does not allow ping?

I continue to wonder about the connection for RPi. Is there any command on the switch that will show status of the interface (rather than looking at configuration of the interface)? Perhaps something like show interface status or show interface g1/9?

[shelzmike]

OK. So this is quite interesting and something that had be nagging me a little in my brain as I was looking...I checked the firewall logs realtime, and got the following interesting tidbit:

I don't know a ton about martian sources, but from what I remember, it is some sort of reserved IP Address and thus it thinks it is being spoofed (or something along those lines).

That being said, perhaps this is actually why my traffic isn't routing bc my firewall thinks the traffic is spoofed. I am not locked into this subnet. I am going to change it to something less obscure (192.168.172.0/26) to see if that makes a difference.

BUT the strange thing is that this still may not be the case, because I know that 172.16.0.0/12 is in the RFC 1918 for private use. Perhaps it thinks it is martian but not from Public, from somewhere else on my LAN. Very strange indeed though.

RPi interface is fine as it worked fine with the VLAN 1 subnet when it was on that.

Sh int status:

Port     Type         Duplex  Speed Neg      ctrl State       Pressure Mode
-------- ------------ ------  ----- -------- ---- ----------- -------- -------
gi1/1    1G-Copper      --      --     --     --  Down           --     --
gi1/2    1G-Copper    Full    100   Enabled  Off  Up          Disabled Off
gi1/3    1G-Copper    Full    100   Enabled  Off  Up          Disabled Off
gi1/4    1G-Copper      --      --     --     --  Down           --     --
gi1/5    1G-Copper      --      --     --     --  Down           --     --
gi1/6    1G-Copper      --      --     --     --  Down           --     --
gi1/7    1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off
gi1/8    1G-Copper      --      --     --     --  Down           --     --
gi1/9    1G-Copper    Full    100   Enabled  Off  Up          Disabled Off
gi1/10   1G-Copper    Full    1000  Enabled  Off  Up          Disabled On
gi1/11   1G-Copper    Full    100   Enabled  Off  Up          Disabled Off
gi1/12   1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off
gi1/13   1G-Copper      --      --     --     --  Down           --     --
gi1/14   1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off
gi1/15   1G-Copper      --      --     --     --  Down           --     --
gi1/16   1G-Copper      --      --     --     --  Down           --     --
gi1/17   1G-Copper      --      --     --     --  Down           --     --
gi1/18   1G-Copper    Full    1000  Enabled  Off  Up          Disabled On
gi1/19   1G-Copper      --      --     --     --  Down           --     --
gi1/20   1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off
gi1/21   1G-Copper      --      --     --     --  Down           --     --
gi1/22   1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off
gi1/23   1G-Copper      --      --     --     --  Down           --     --
gi1/24   1G-Copper    Full    1000  Enabled  Off  Up          Disabled Off
gi1/25   1G-Combo-C     --      --     --     --  Down           --     --
gi1/26   1G-Combo-C     --      --     --     --  Down           --     --
gi1/27   1G-Fiber       --      --     --     --  Down           --     --
gi1/28   1G-Fiber       --      --     --     --  Down           --     --

No show int Gi1/9 command available.

[PH42420]

I don't want to speak out of turn or anything, but I have spent a lot of time with this particular Cisco device.

Some long term problems I ran into were similar to what you seem to be having.

1st, it took almost 2 full weeks to get the router approved to operate on ATT's network (just to add some context), and about half as much time for ATT to figure out how to form an adjacency with it. Cisco's protocols for doing this are ubiquitous and very good as industry standards. For whatever reasons, however, even with the ATT router supporting the intended configuration there was something of a struggle getting reciprocal traffic to NAT through it.

The management interface of the edge router was accessible from any network defined on the Cisco device, but no internet packets passed out through the WAN would return. There was some sort of tagging and re-tagging issue there, on ATT's router.

[shelzmike]

@PH42420You could certainly be onto something.

My previous message was incorrect (related to martian packets.

After some further research, it basically boils down to the fact that the USG seems to be receiving some traffic from my client, but has no route back to it, thus martian.

The real weird thing is..where the heck is it coming from? I am mirroring the port going from my switch to the USG, all traffic, and I am not seeing this traffic the firewall is reporting.

What fun is a lab though if not for problems like this? It isn't impeding my normal traffic flow, so I don't have end users at my door (i.e. my wife and kids :) ) . I haven't given up yet. I will wait until I get the dongle for the extra NIC to add to my linux machine and wireshark from there so I know that if the VLAN tags are there, I should see them. Will help me better diagnose, I think.

[shelzmike]

All that being said, I still DO NOT see a single packet going from the USG to the Switchport when I ping from the CLI of the USG to the client. It doesn't appear to ever leave the USG. When I do a tracert, it simply traces it to the virtual int of 172.16.254.1 on the USG and stops. So even a linux wireshark instance won't help me with that. It has got to be something on the USG based on that fact alone.

[shelzmike]

Even stronger evidence there is something amiss with that trunk port and here is why.

So I connected my phone AND my laptop to the SSID on VLAN 172. I can ping both both ways, BUT i never see it going over that trunk port. Since the USG is managing the L3 traffic, I assume that the reason there is communication between the 2 wireless devices is because it never needs to hit the switch for any reason.

So conclusion for now it is that link between the 2 (USG and switch) and traffic doesn't seem to ever leave the USG on the LAN.

[Richard Burts]

I have looked back through this thread and have several observations/suggestions.

- as far as martian source is concerned I believe that sometimes it reflects receiving an IP packet with an address that is not valid, but sometimes it might reflect receiving an IP packet on an interface with an address that the device does not associate with that interface. In looking at the capture I see 3 IP addresses, 192.168.10.5 which is USG vlan 1 address, 172.16.254.1 which is USG vlan 172 address, 117.16.254.10. What network is this? And is it associated with the receiving interface which appears to be eth1.172?

- you tell us that you have an SSID that uses 172.16.254 and that a device connected through the SSID works and can access the USG. I wonder if it complicates things when you also configure a switch interface to be in that subnet? I wonder what would happen if you used a different vlan/different IP subnet for RPi?

- sort of related thought - devices using the SSID use DHCP to get an IP address. But RPi has a static IP. And I note that the posted config indicates that dhcp snooping is enabled. I wonder if dhcp snooping might be impacting the traffic from RPi? Perhaps this is another reason to try a different vlan/different IP subnet?