DHCP not working on default VLAN1 but works on other VLANS

[mlord]

Hello,

   I've spent about 2 weeks reading threads on this subject but I've yet to find what solves my issue. We recently have an issue where our VLAN1 stopped doing DHCP, well, not entirely. Sometimes it takes a long time for an IP to be issued, or not at all. And sometimes it issues an IP that's already in use even though most of the IP addresses are available. The configurations we're using have been in use for years; we've had this problem maybe twice in the past but a core stack reboot would typically clear it up. We deleted exclusions (bad idea) as I assumed having reservations was sort of the same thing; they've been re-added so no more conflicts. My laptop is connected to a small switch, which is then connected to our Admin Core stack of 3 9300-48T's (16.8.1a). An ipconfig /renew does nothing but return an error it couldn't find our DHCP server. I'm trying to get an IP from our scope (192.168.0.1-254/24) from default VLAN1 (192.168.0.150) to our DHCP server VM (10.1.90.3). The DHCP Server VM is on a VMWare machine; this VMWare machine has two Nics occupied and each are a trunk with all VLAN's. Our Sonicwall Firewall is 192.168.0.1, our switch stack is 192.168.0.150 (if it even should be) and our DHCP server vm is 10.1.90.3.

   I'm sure both switch stacks don't have matching configurations; these were set up prior to me and so with reverse engineering how these were set up, plus not being fully educated on cisco switches and managing them this has been an uphill climb. I appreciate any and all questions and insights. Thank you.

[MHM Cisco World]

you not being able to talk to the News stack<<- but this another symptom that he have routing issue.

He access vlanX using IP from vlanY' to make retrun traffic back if SW dont have SVI of vlanY then access is failed.

He need ip routing and default route NOT defualt gateway in SW face issue.

Have a nice day

MHM

[Peter Paluch]

MHM,

but this another symptom that he have routing issue

There may be a routing issue, yes, but not related to the problem with DHCP because eventually, the client in VLAN1 will get an IP address from the DHCP server. If the routing issue involved the DHCP connectivity between the DHCP relay and the DHCP server, the client would never get an IP address, no matter how long he waited. This is clearly not the case.

He need ip routing and default route NOT defualt gateway in SW face issue.

But these elements are configured already - the configurations were shared right in the opening post of this entire thread. Let's keep with the clear evidence only, let's make sure we have acquainted ourselves in detail with all of it, and let's not jump to hasty suggestions.

Best regards,
Peter

[MHM Cisco World]

Ok' let waiting his reply.

[mlord]

@Peter Paluch ,

   Attached are the two PCAPS in the .zip file; one from my laptop labeled as DHCPClient, and one from the DHCP Server labeled as HUBDC1.

[Peter Paluch]

Hello @mlord ,

I've reviewed the packet captures. What is extremely weird is that only a few of the client DHCP packets were relayed all the way to the DHCP server, and that bothers me because somewhere, for some reason, the packets are getting dropped. What is even more disconcerting for me is that both the DHCP client and the DHCP server are directly connected to the Admin switch stack, client in VLAN 1, server in VLAN 90. So for the communication between these two, there is absolutely no routing issue present - both IP networks are directly connected to the switch.

I would like to ask for another capture, but this time, we will capture at three points at the same time: Your laptop, your DHCP server, and the Admin switch stack itself. I need to understand if the switch is receiving all the DHCP packets from the client and relaying them.

You will need to prepare the switch for this capture before you start it, and you do it with these commands:

configure terminal
ip access-list extended acldhcp
permit udp any any eq 67
permit udp any any eq 68
end

! Following commands come in the EXEC mode of the switch, not in the configuration mode
monitor capture dhcp buffer size 10
monitor capture dhcp access-list acldhcp
monitor capture dhcp limit packet-len 9500
monitor capture dhcp control-plane both

Afterwards, please start the Wireshark on the server, on the client, and on the Admin switch, execute this command:

monitor capture dhcp start

Then please instruct your laptop to obtain the IP address from DHCP and leave the capture running for a few minutes. Afterwards, stop the Wireshark on your laptop and the server, and on the Admin switch, do this:

monitor capture dhcp stop
monitor capture dhcp export location bootflash:dhcp.pcap

Finally, use a SCP or SFTP client (WinSCP, for example) to download the dhcp.pcap file from the switch. Finally, please attach all these files here again.

Thank you!

Best regards,
Peter

[MHM Cisco World]

What is even more disconcerting for me is that both the DHCP client and the DHCP server are directly connected to the Admin switch stack, client in VLAN 1, server in VLAN 90.

This key here if it routing issue or not' he must confirm that he connect pc and server in same SW (admin).

This may last post here in community' I will not have time anymore to reply.

So good luck to all 

Have a nice days for all friends.

Good bye.

MHM

[mlord]

@Peter Paluch 

Thanks, I'm fascinated as well, and I do agree that those packets are being dropped. I'll prep for the capture immediately.

In the meant time, if this information is of value, I do see DHCP traffic in our Sonicwall; with dropped packets. Not sure if that's a concern, or just due to the Sonicwall being on the VLAN1 subnet and it's just normal UDP stuff.

[mlord]

@Peter Paluch 

   Look at what I found in my Sonicwall. My spare NIC is still trying to acquire DHCP on my laptop; that Source mac address is my NIC in the Packet Detail section.

[Peter Paluch]

Hello @mlord ,

I don't think that the findings on the SonicWall are important. You see, the SonicWall is in VLAN 1, so is the AdminStack, and so is your laptop. When your laptop tries to acquire its IP address from DHCP, it shouts across the entire VLAN - that's the point of the broadcast - and everyone else in the VLAN receives it. The SonicWall is dropping it, but the AdminStack should still pick it up and relay it based on its own configuration from the directly attached network in VLAN1 to the directly attached network in VLAN90 and there to the DHCP server. So I'm not too worried about the SonicWall - that is expected.

Best regards,
Peter

[mlord]

@Peter Paluch 

   Understood, I figured as much. Wireshark captures are currently capturing, and I should have them available shortly. Thank you again for your time and attention to my issue.

[mlord]

@Peter Paluch 

Finally, I have the files requested. Please see the attached .zip file.

[Peter Paluch]

Hello @mlord ,

Thank you for the files!

Looking into the packet capture, the one obtained from the switch somehow contains only 3 packets which are ARP messages. That somehow does not make sense - I hoped to see way more traffic on the switch capture.

Can we do one more capture, just on the switch, with the same commands please? You do not need to re-enter the ACL into the configuration if it is still there. Just please also include the output that gets displayed on the CLI of the switch as you create and start the capture session, and when you stop it and export it. I want to be sure we're not doing something wrong when running that capture session.

Without the proper capture session on the AdminStack, it'll be considerably tougher to troubleshoot this, that's why I'm so insistent...

Best regards,
Peter

[mlord]

@Peter Paluch 

   I believe the only output the terminal showed were prompts that running the capture could cause performance issues; it didn't display the ARP messages as it captured them. Someone also came in to chat with me so I believe the captures all ran about 7 to 10 minutes give or take. I can try running them longer. I can also wait a minute to trigger a /renew from my NIC adapter while all captures are running. I do see the below lines in show run.

ip access-list extended acldhcp
permit udp any any eq bootps
permit udp any any eq bootpc

EDIT: Looks like my NIC adapter finally pulled DHCP sometime after I left yesterday. Looks to be more than an hour after issuing the /renew command. It's also the same IP I've used static for a long time before all this; while doing this testing I seem to only get 1 of 2 IP's...either .0.10, or .0.36. I'm also seeing "BAD_ADDRESS" in some of the DHCP leases. I did a /release command on my NIC and it took immediately, updating the scope and instantly removing the address.

Ethernet adapter Ethernet:

Connection-specific DNS Suffix . : weny.lb.ads
Description . . . . . . . . . . . : Killer E3100 2.5 Gigabit Ethernet Controller
Physical Address. . . . . . . . . : 00-D8-61-E6-F5-84
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IPv4 Address. . . . . . . . . . . : 192.168.0.36(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Lease Obtained. . . . . . . . . . : Tuesday, August 1, 2023 6:41:05 PM
Lease Expires . . . . . . . . . . : Thursday, August 3, 2023 6:41:03 AM
Default Gateway . . . . . . . . . : 192.168.0.150
DHCP Server . . . . . . . . . . . : 10.1.90.3
DNS Servers . . . . . . . . . . . : 192.168.0.144
                                             10.1.90.3
NetBIOS over Tcpip. . . . . . . . : Enabled

[Peter Paluch]

Hello @mlord ,

Let's try to do the whole capture exercise one more time. I have modified the ACL to be vastly more permissive and catch all UDP traffic hitting the control plane. So the commands to be executed on the AdminStack will be as follows:

no monitor capture dhcp

!
configure terminal
 no ip access-list extended acldhcp
 ip access-list extended acldhcp
  permit udp any any
 exit
end
!

monitor capture dhcp control-plane both
monitor capture dhcp buffer size 50
monitor capture dhcp access-list acldhcp

After this, start Wireshark on the client and the server, and on the switch, run

monitor capture dhcp start

The rest is like yesterday - wait a few minutes, then stop the Wiresharks and on the switch, please do:

monitor capture dhcp stop
monitor capture dhcp export location bootflash:dhcp.pcap

Then please attach all the files here again.

Thank you!

Best regards,
Peter

[mlord]

@Peter Paluch 

   Running into an issue when I enter "monitor capture dhcp stop" and it says "Capture dhcp is not active" and "Unable to deactivate Capture". I deleted the old capture file in case it couldn't overwrite or something and am trying again.

WENYCore#monitor capture dhcp start

Enabling Control plane capture may seriously impact system performance. Do you want to continue? [yes/no]:
% Please answer 'yes' or 'no'.

Enabling Control plane capture may seriously impact system performance. Do you want to continue? [yes/no]: yes
Started capture point : dhcp

WENYCore#monitor capture dhcp stop
Capture dhcp is not active

Unable to deactivate Capture.