Solved: DHCP not working on default VLAN1 but works on other VLANS

mlord · ‎07-28-2023

Hello,

I've spent about 2 weeks reading threads on this subject but I've yet to find what solves my issue. We recently have an issue where our VLAN1 stopped doing DHCP, well, not entirely. Sometimes it takes a long time for an IP to be issued, or not at all. And sometimes it issues an IP that's already in use even though most of the IP addresses are available. The configurations we're using have been in use for years; we've had this problem maybe twice in the past but a core stack reboot would typically clear it up. We deleted exclusions (bad idea) as I assumed having reservations was sort of the same thing; they've been re-added so no more conflicts. My laptop is connected to a small switch, which is then connected to our Admin Core stack of 3 9300-48T's (16.8.1a). An ipconfig /renew does nothing but return an error it couldn't find our DHCP server. I'm trying to get an IP from our scope (192.168.0.1-254/24) from default VLAN1 (192.168.0.150) to our DHCP server VM (10.1.90.3). The DHCP Server VM is on a VMWare machine; this VMWare machine has two Nics occupied and each are a trunk with all VLAN's. Our Sonicwall Firewall is 192.168.0.1, our switch stack is 192.168.0.150 (if it even should be) and our DHCP server vm is 10.1.90.3.

I'm sure both switch stacks don't have matching configurations; these were set up prior to me and so with reverse engineering how these were set up, plus not being fully educated on cisco switches and managing them this has been an uphill climb. I appreciate any and all questions and insights. Thank you.

Peter Paluch · ‎08-02-2023

Hi @mlord ,

So, this capture proved most insightful!

Indeed, the PCAP from the switch was filled with 50MB of transit data within mere 0.2 seconds. Such amount of traffic hitting the CPU is absolutely abnormal; a switch should not be handling transit traffic at all in the CPU.

It took me a while to understand why this was happening: The hosts we see in the PCAP from the switch are from VLAN1 and its address space 192.168.0.0/24 and obviously they are using the switch - the 192.168.0.150 - as their default gateway. However, the switch itself has its default route configured to point to 192.168.0.1. Hence, the traffic from hosts in VLAN1 and destined to internet is getting "tromboned" or "hairpinned" - it comes to the switch through VLAN1 and the switch needs to send it out through VLAN1 again to 192.168.0.1.

Now, IP rules dictate: Whenever you need to route a packet out the same routed interface that you received it on, you must also send an ICMP Redirect to the sender to inform it that it can save one hop by using the proper gateway directly.

And this is the problem! With hardware-based switches, ICMP Redirects are generated in the operating system of the switch, not in its switching/routing hardware, and so for the switch to be able to generate those redirects, the hairpinned traffic must be punted to the CPU for the operating system to see it and generate the ICMP Redirects. This is why the traffic from VLAN1 exiting toward internet is badly hammering your CPU on the switch, causing it to not have time to process more useful traffic in VLAN1 - like your DHCP clients, for example.

It's clear that the end hosts do not honor the ICMP Redirect messages originated by the switch because they are not changing their routing - they're still blasting the traffic across the switch. Ignoring ICMP Redirect messages is the usual behavior of today's operating systems, so no surprise here. Hence, while the proper solution would be to fix the routing on the hosts inside VLAN1, it may not be easily possible, and as a quick interim fix, we can disable generating the ICMP Redirects in VLAN1. This will prevent the switch from punting the traffic to the CPU.

The interim fix is easy:

configure terminal
interface Vlan1
 no ip redirects
end

Please try to configure this and then try getting an IP address from your laptop in VLAN through DHCP again.

Best regards,
Peter

View solution in original post

Flavio Miranda · ‎07-28-2023

Hi @mlord

I would like to see the config from Core. And ask one question, did you try to use another vlan instead 1 ?

If you setup a static IP on your laptop can you ping the DHCP server uninterrupted ?

If you ping with bigger packet size?

mlord · ‎07-28-2023

Hello Flavio,

I can indeed ping uninterrupted when a static IP is set. A tracert shows 1ms 1 ms 1ms 192.168.0.150, 1ms 1ms 1ms 10.1.90.3. I did a ping with a packet size of 1024 and was successful at time=2ms.

A few minutes ago I just created a new DHCP scope of 10.1.30.1/24; this VLAN was already configured in our switch stacks there just wasn't a scope set up as it wasn't needed. DHCP worked without issue from the second stack over the port-channel.

Edit: The config from Core is attached to this post as "AdminStackRunningConfig.txt".

Flavio Miranda · ‎07-28-2023

So we can conclude that the problem is related to vlan 1 only. If I understood correctly this traffic does not make it to the firewall right?

The Core have SVI on Vlan 1 and have IP helper-address pointing to DHCP server which is connected to the Core.

If you connect your laptop to the core directly in one interface on vlan 1, do you face the same problem?

MHM Cisco World · ‎07-28-2023

sorry, VM DHCP server is in VLAN1 so there is no routing there is only bridging, and I see trunk allow vlan.
so there is break bridge between host in VLAN1 and DHCP server in VLAN1.
first check is manual assign IP to any host in VLAN1 and from that host ping DHCP server and see the ping is success or not.
check if there is any VLAN ACL or Port ACL prevent DHCP traffic

mlord · ‎07-28-2023

Hello MHM,

Thank you for your reply. I'm confused by what you mean by saying the VM DHCP server is in VLAN1, when the VM itself has an ip of 10.1.90.3. My Laptop is set static to 192.168.0.36 and can successfully ping and tracert to 10.1.90.3.

MHM Cisco World · ‎07-28-2023

OK I confuse so DHCP server not in VLAN1
the GW I think is FW ?
if Yes then
allow traffic unicast for UDP port 67/68

mlord · ‎07-28-2023

A tracert from my laptop goes 192.168.0.36 > 192.168.0.150 > 10.1.90.3. The 10.1.90.3 is assigned from VLAN90. The DHCP Server is a virtual machine with a virtual NIC. The Virtual NIC is created within VMWARE who's phsyical NIC is a physical switch interface that is trunked to allow all VLANS.

The Firewall is 192.168.0.1, but the Gateway should be 192.168.0.150 which should be our core stack correct?

Allow traffic for UDP port 67/68 on what? Globally or per VLAN? I would assume 67/68 are already allowed as DHCP works on other VLANS without issue.

MHM Cisco World · ‎07-28-2023

from traceroute not pass through the FW no need UDP 67/68 in FW
ok, so you use IP helper in 192.168.0.3 vlan SVI ?

mlord · ‎07-28-2023

I do have the running configurations attached as .txt files that show that information. I'll paste all the VLAN info below as well.

At the bottom of this running configuration (Admin Core Stack) I do see ip-default gateway of 192.168.0.1.

I noticed in the running configuration of our News Core Stack it shows ip-default gateway of 192.168.0.150. These two stacks are attached by a port-channel. I'm assuming that's a 'global' switch configuration for ip-default gateway?

interface Vlan1
description default vlan
ip address 192.168.0.150 255.255.255.0
ip helper-address 10.1.90.3
ip directed-broadcast
no ip proxy-arp
!
interface Vlan11
description WindstreamWANVLAN
no ip address
!
interface Vlan20
description WindstreamMPLS
ip address 63.246.204.158 255.255.255.252
no ip redirects
no ip proxy-arp
!
interface Vlan21
description "Windstream P2P to Erie"
ip address 10.10.21.11 255.255.255.0
!
interface Vlan30
description NewsVLAN
ip address 10.1.30.1 255.255.255.0
ip helper-address 10.1.90.3
ip directed-broadcast
no ip proxy-arp
!
interface Vlan40
description WENY_WiFi
ip address 10.1.40.1 255.255.255.0
ip helper-address 10.1.90.3
ip directed-broadcast
no ip proxy-arp
!
interface Vlan50
ip address 10.1.50.1 255.255.255.0
no ip proxy-arp
!
interface Vlan52
description SpectrumELAN
ip address 10.10.52.11 255.255.255.0
ip directed-broadcast
no ip proxy-arp
!
interface Vlan60
description VizOpusVLAN
ip address 10.1.60.1 255.255.255.0
ip directed-broadcast
no ip proxy-arp
!
interface Vlan70
description ImagineVLAN
ip address 10.1.70.1 255.255.255.0
ip directed-broadcast
no ip proxy-arp
!
interface Vlan80
description CreativeVLAN
ip address 10.1.80.1 255.255.255.0
ip helper-address 10.1.90.3
ip directed-broadcast
no ip proxy-arp
!
interface Vlan88
description TestP2PVLAN
ip address 192.168.88.1 255.255.255.0
!
interface Vlan90
description Server/ManagementVLAN
ip address 10.1.90.1 255.255.255.0
ip helper-address 10.1.90.3
ip directed-broadcast
no ip proxy-arp
!
interface Vlan100
description GuestVLAN
ip address 10.1.100.1 255.255.255.0
ip helper-address 10.1.90.3
ip directed-broadcast
no ip proxy-arp
!
interface Vlan110
description SalesAdminTrafficVLAN
ip address 10.1.110.1 255.255.255.0
ip helper-address 10.1.90.3
ip directed-broadcast
no ip proxy-arp
!
interface Vlan120
description MasterControlVLAN
ip address 10.1.120.1 255.255.255.0
ip helper-address 10.1.90.3
ip directed-broadcast
no ip proxy-arp
!
interface Vlan130
description StreamingVLAN
ip address 10.1.130.1 255.255.255.0
ip helper-address 10.1.90.3
!
interface Vlan150
description EngineeringVLAN
ip address 10.1.150.1 255.255.255.0
ip helper-address 10.1.90.3
ip directed-broadcast
no ip proxy-arp
!
interface Vlan254
description SW_MGMT
ip address 10.1.254.1 255.255.255.0
ip directed-broadcast
no ip proxy-arp
!
interface Vlan1000
description Test Multicast VLAN
ip address 192.168.100.1 255.255.255.0
!
ip default-gateway 192.168.0.1
ip forward-protocol nd

MHM Cisco World · ‎07-28-2023

no ip directed-broadcast <<- disable the direct-broadcast for VLAN1 SVI
and if you can
share
show ip int brief

mlord · ‎07-28-2023

Thank you MHM I will try your suggestion. We have 24/7 video feeds that need to be kept up and I'm currently not on site. I will do this Monday morning and report back promptly.

MHM Cisco World · ‎07-28-2023

From my side I will run lab and check the effect of ip direct-broadcast on dhcp.

MHM Cisco World · ‎07-30-2023

I check ip direct-broadcast and it have no effect on dhcp reply

MHM Cisco World · ‎07-30-2023

ip forward-protocol nd <<- change this to
ip forward-protocol <<- THIS WILL ALLOW udp known port include the DHCP UDP ports