Re: SGs switches and odd wrong gateway issue via DHCP

Marc66 · ‎03-18-2018

Hi All!

I don't know if anyone has ever had this issue before, it has been happening since we started using SGs switches as layer 3 and as DHCP server.. The setup we do at our clients' homes is mainly 4-5 VLANs, intervlan routing at the layer 3 switch (usually SG300 but it can be SG500), DHCP server on that layer 3 switch, DHCP snooping to only allow the main switch to deliver DHCP packets

This is random, it can happen at any time, months, sometimes weeks but the issue is devices on some VLANs will start getting their IP address normally but the gateway would be wrong, it would be an IP address of the switch on a different VLAN!

So for example, I have VLAN 30 (main switch's IP address on that VLAN is 10.0.30.1) for client's VLAN and VLAN 80 (10.0.80.1) for media devices such as Apple TVs, IPTV receivers etc. Devices on that VLAN 80 would at some point start getting a gateway of 10.0.30.1 instead of 10.0.80.1! The only way to solve this is either restarting that main switch or disabling the DHCP server on it and enabling it again but by doing this, we lose all the DHCP bindings. Activating ARP proxy solves the problem but obviously hides the issue

The switches are running the latest firmware. This problem doesn't happen if we use the firewall as DHCP server (the switches would all be layer 2 then)

Any ideas? Do I have to use a Raspberry Pi as a DHCP server (serving all VLANs, with DHCP relay then on the main switch) instead of those switches to avoid this issue, if it is one?

I asked Cisco support at the time but they were not helpful, they talked about the firmware not having an internet connection or something like that and that I should just reupload it, I can't remember now, but that's not the problem because if I restart the switch or disable the DHCP server and re-enable it, it works, so it is obviously a bug in the switch's firmware (from 1.4.5 I believe, at least).. And there is no rogue DHCP whatsoever, DHCP snooping is configured anyway and the devices that we have on the VLANs cannot be DHCP servers

Thank you for your help,

Regards

Lee Cox · ‎03-27-2018

I have been running a SG300-28 layer 3 switch in layer 3 mode with 3 VLANs for at least 3 years. I have never once seen this issue. I do not run DHCP snooping. I also run 2 other Cisco switches in layer 2 mode connected with trunk ports to the layer 3 switch.

I would say you need to narrow it down more. You might try without DHCP snooping. If it is always the same 2 networks then try changing 1 of the network IPs.

I guess I should add I upgrade firmware as it comes out. I have Cisco Findit configured to let me know when new firmware's come out.

Marc66 · ‎03-28-2018

Thanks for your reply Lee, but do you run DHCP server on those too or is the DHCP server on another switch or device?

The problem is only when I run those in DHCP server, it has happened on one of our jobs this morning actually, one TV box on vlan 80 was receiving a gateway of 10.0.30.1 (switch's IP address in vlan 30) instead of 10.0.80.1, I restarted the switch and that solved the issue but obviously it is not a long-term solution.. I always upgrade to the latest firmware

It does it more with vlan 80 but I don't think it is related to the IP subnet itself, maybe more to an "array" issue in programming, on one of our jobs last year the issue happened on vlan 30 where devices were getting a gateway IP address of vlan 10's IP address.. vlan 10 is "before" vlan 30, which is "before" vlan 80, if it makes sense, I am not sure the order is important but maybe..

Let me know if you run the SG300 layer-3 switch in DHCP server mode or not

Thanks

Aleksandra Dargiel · ‎03-28-2018

Hi Marc,

I have seen this issue before but quite difficult to reproduce so I would suggest to open support case with Small Business team and PM the case number so I can get involved in the process of gathering forensic .

Regards,

Aleksandra

Marc66 · ‎03-28-2018

Hi Aleksandra,

Thanks for your reply! I will create a ticket with SB support team then and PM you the case number ;) The only thing is it is difficult to do any factory reset or reboot as these are (high profile) customers' homes and well this is what I had tried in the last ticket with the support team and it didn't help, as I expected, because you disable and enable again the DHCP server (or reboot the switch) it works..

Do you actually remember this issue to be firmware related or is it a "bug" in the switch's config file? You're right, it is difficult to reproduce as it can take weeks or months to happen again

Regards,

Marc

Lee Cox · ‎03-28-2018

I run DHCP server on the layer 3 switch. I use 192.168.x.x IP addresses. I use 192.168.x.254 as my switch default gateways. My router VLAN is not 254.

I also run a router VLAN which I have the router isolated in on a 252 mask which is a point to point connection between the switch and router. . I think it works better than having the router in a default VLAN 1 the management VLAN. The only thing which does not work is Cisco FIndit since it's scope is 1 network VLAN.

I also run 3 Cisco wireless units. I had 3 WAP321 which I am now replacing with WAP371 units. They have multiple VLANs defined where the switch provides DHCP service.

Marc66 · ‎03-28-2018

DHCP server on the SG300-28? So you have maybe 3 pools?

This issue is indeed strange, I am not sure why this happens, it actually has not happened (if I am not mistaken) on jobs with no DHCP snooping yet but that would be really strange because the purpose of DHCP snooping is to provide more security... not mess with DHCP

The SVIs on the main switch ends with .1 (10.0.X.1), most VLANs have their pool on that switch, the switch's default gateway is a Draytek router (or it can be something else sometimes) and I use a dedicated transit VLAN for this.

Lee Cox · ‎03-28-2018

I think you need to use your layer 3 switch as your default gateway for all VLANs not the router. Your router is just an extra device when using a layer 3 switch. The layer 3 switch should handle all the local LAN traffic and only hand off the internet bound traffic to the router.

The router should connect to an access port not a trunk port.

I think we are saying the same thing.

Marc66 · ‎03-28-2018

Yes the Cisco switch is the default gateway for most VLANs (except for some which are on the Draytek firewall, I run those VLANs, mainly CCTV and BMS ones, through the link between the switch and the Draytek so the link is trunk) but as it is a layer 3 switch, it needs to have a default route configured and it is the firewall ;)

As I said, it is a basic setup, 3-4 VLANs on the main Cisco SG switch (layer 3) with interVLAN routing on this switch, DHCP server enabled on it for most VLANs and on the firewall if the corresponding VLAN run back to the latter, DHCP snooping activated when possible, for most VLANs on many of our jobs (not the old ones, it is disabled, but pretty much same setup)

I hope it is clearer :)

Lee Cox · ‎03-28-2018

Sounds we are close to the same config. I really have 4 VLANs defined on the layer 3 switch as I forgot about the router VLAN.

I run a Cisco RV320 router with DHCP turned off and connected to an access port defined with a 30 bit mask.

Hope you can narrow down the problem. I will report back if I ever see this problem but I have been running this setup for a long time. The SG300-28 which I have is a great switch. It never gives me any problems.

Marc66 · ‎03-28-2018

Thanks Lee, yes we seem to have a close setup with different DHCP pools etc. the only difference I see with your setup is DHCP snooping which you don't use but I do. I would be very surprised if the issue comes from DHCP snooping as it shouldn't. When I restart the switch, or disable the DHCP server on it (usually SG300-52P but it happened on a SG500-52P too) and enable it again, even though I lose all the bindings in this case, the devices get the correct IP gateway, it's odd.

But besides of this issue (and possibly another one but not the subject here), these switches are great yes

I will contact support asap and I will update this post :) the only problem (possibly) is many of those jobs are beyond 1 year time so not under warranty now, let's see

bdp-cisco · ‎04-06-2023

I have exactly this issue. My SG300 has worked error-free for several years handling 10 vlans with L3 services including DHCP. I added a new vlan the other day, and since doing so the DHCP server is handing out IP addresses with the gateway being that of the new vlan I just added. It is pretty obviously a bug. Currently running 1.4.9.4 on this SG300-10pp, looks like it is time to upgrade.

bdp-cisco · ‎04-06-2023

For clarity, in the above post I am saying the DHCP server that used to work on 10 separate vlans worked fine, but since adding a new vlan the DHCP server is handing out gateway address of the new vlan for all the vlans that used to work just fine for years. Definitely a bug in 1.4.9.4