Solved: DHCP offer dropping at the switch

Graham Murison · ‎07-20-2023

I believe I am coming up against this bug: https://bst.cisco.com/bugsearch/bug/CSCvo31022.
I do not have DHCP snooping turned on and I as far as I have read it is disabled by default.

When I debug ip udp:

*Jul 20 2023 21:21:55.271: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:21:58.274: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:22:00.072: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:22:00.072: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:22:05.057: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:22:05.057: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:22:08.098: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:22:08.099: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:22:10.270: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:22:13.044: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:22:13.045: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:22:13.276: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:22:16.277: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:22:16.932: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:22:16.932: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:22:25.901: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:22:25.901: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:22:28.274: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:22:31.279: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:22:34.280: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:22:42.639: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:22:42.640: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:22:46.277: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:22:49.282: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:22:52.283: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:23:04.279: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:23:07.285: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:23:10.285: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:23:15.122: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:23:15.122: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:23:19.121: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:23:19.122: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:23:22.282: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:23:25.288: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:23:28.074: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:23:28.075: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:23:28.290: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:23:40.285: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:23:43.288: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:23:44.059: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=308
*Jul 20 2023 21:23:44.060: UDP: sent src=10.96.20.1(67), dst=10.96.20.20(67), length=309
*Jul 20 2023 21:23:46.292: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:23:58.288: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:01.291: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:04.294: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:16.290: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:19.294: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:22.297: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:34.293: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:37.296: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:40.300: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:52.296: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:55.300: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584
*Jul 20 2023 21:24:58.303: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584

Wireshark from the Windows DHCP Server shows both the DHCP discover and DHCP Offer
4329 512.716357 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0x4301746e
4377 520.659425 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0x4301746e
4379 520.663389 10.96.20.20 10.96.100.1 DHCP 354 DHCP Offer - Transaction ID 0x4301746e
4470 537.637299 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0x4301746e
4471 537.637859 10.96.20.20 10.96.100.1 DHCP 354 DHCP Offer - Transaction ID 0x4301746e
6964 877.509726 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0xeb7b149b
6993 882.494355 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0xeb7b149b
7426 885.535618 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0x18384edd
7458 890.481533 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0x18384edd
7492 894.368806 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0x18384edd
7941 903.337633 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0x18384edd
7942 903.338102 10.96.20.20 10.96.100.1 DHCP 354 DHCP Offer - Transaction ID 0x18384edd
8306 920.075577 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0x18384edd
8307 920.076069 10.96.20.20 10.96.100.1 DHCP 354 DHCP Offer - Transaction ID 0x18384edd
8566 952.556690 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0xdf55d080
8632 956.556204 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0xdf55d080
8798 965.508682 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0xdf55d080
8799 965.509335 10.96.20.20 10.96.100.1 DHCP 354 DHCP Offer - Transaction ID 0xdf55d080
8913 981.493313 10.96.20.1 10.96.20.20 DHCP 343 DHCP Discover - Transaction ID 0xdf55d080
8914 981.493953 10.96.20.20 10.96.100.1 DHCP 354 DHCP Offer - Transaction ID 0xdf55d080

The Windows DHCP server is in VLAN 20 and the client is in VLAN 100
Config attached.

When I set a static IP on the client, it is able to ping the DHCP server so routing is there.

Wondering if I am missing any thing obvious.

Graham Murison · ‎07-24-2023

Thanks to some tips from @Harold Ritter @Flavio Miranda I figured this out and it came down to the Security team a long time ago created a non-logging firewall rule that blocks ANY/ANY traffic for bootp and Netbios. Usually this works well and good practice until you are using the same firewall for inter-VLAN traffic. The broadcast relay (DISCOVER) was working Client > Relay > FW > Server, but the unicast OFFER failed at the firewall due to above (Server > FW). Once the Security team allowed bootp/dhcp & netbios between the required segments, the issue is fixed.

Below is the working config. Also multiple VLANs are reaching the same DHCP server and getting the correct IP lease. Posting for completeness.



========
no dhcp relay information option vpn 
interface Vlan100
description SVI for IT VLAN
vrf forwarding IT-vrf
ip address 10.96.xx.1 255.255.255.0
ip helper-address global 10.96.ab.20
no ip redirects
!
interface Vlan110
description SVI for First Floor VLAN
vrf forwarding Users1-vrf
ip address 10.96.xy.1 255.255.254.0
ip helper-address global 10.96.ab.20
!
interface Vlan120
description SVI for Second Floor VLAN
vrf forwarding Users2-vrf
ip address 10.96.yy.1 255.255.254.0
ip helper-address global 10.96.ab.20
!
interface Vlan130
description SVI for Third Floor VLAN
vrf forwarding Users3-vrf
ip address 10.96.yz.1 255.255.254.0
ip helper-address global 10.96.ab.20
!
interface Vlan140
description SVI for Fourth Floor VLAN
vrf forwarding Users4-vrf
ip address 10.96.zz.1 255.255.254.0
ip helper-address global 10.96.ab.20
========

I believe this is working since the IP helper address is also the gateway address for the DHCP scope. The DHCP server is Windows 2019.

View solution in original post

MHM Cisco World · ‎07-20-2023

Is the SW is 9000 series?

If Yes then check below link

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-series-switches/217429-troubleshoot-slow-or-intermittent-dhcp-o.html

Graham Murison · ‎07-20-2023

Thanks I saw that too, hence why I disabled ip redirect on the SVI's involved, but unfortunately that hasn't resolved the issue.

The SW is a 3850 bought from a 3rd party reseller, so SmartNet is not an option.

MHM Cisco World · ‎07-20-2023

 ip dhcp relay information trusted
 ip dhcp relay information option-insert
 ip dhcp relay information check-reply

why you config these dhcp relay option even so you dont have DHCP snooping
you need only
ip helper address

Flavio Miranda · ‎07-20-2023

Hi @Graham Murison

The bug mention drop dhcp discovery and it mention "some times". Dont think is the bug.

Wondering, when the server reply to client the reply get into vrf for vlan 20 and it is not getting to vrf on vlan 100.

You said you can ping but maybe the dhcp is being handled diferently.

Did you try to use the same vrf or no vrf?

Harold Ritter · ‎07-20-2023

Hi @Graham Murison ,

As others mentioned, this is not related to the above bug but rather to the fact that the DHCP server and client are in different VRFs.

This is feasible but you need to change the configuration a bit.

!

ip dhcp relay information option vpn (to set option 82)

interface vlan100

ip helper-address vrf Servers-vrf 10.96.20.20

You also need to set the VRF name (IT-vrf) on the DHCP server pool.

Regards,

Harold Ritter
Sr Technical Leader
CCIE 4168 (R&S, SP)
harold@cisco.com
México móvil: +52 1 55 8312 4915
Cisco México
Paseo de la Reforma 222
Piso 19
Cuauhtémoc, Juárez
Ciudad de México, 06600
México

shabanaiqbal265 · ‎07-20-2023

When DHCP offers are dropping at the switch, it typically indicates a problem with the communication between the DHCP server and the client devices. DHCP (Dynamic Host Configuration Protocol) is responsible for dynamically assigning IP addresses and other network configuration parameters to devices on a network.

Here are some common reasons why DHCP offers may be dropping at the switch:

DHCP Server Configuration Issues: Verify the configuration of the DHCP server to ensure it is set up correctly with a sufficient pool of available IP addresses to assign to client devices.
Switch Port Misconfiguration: Ensure that the switch ports connected to the DHCP server and client devices are correctly configured. They should be in the appropriate VLAN and have the correct settings for DHCP communication.
DHCP Relay Configuration: If the DHCP server is located on a different network segment (subnet) from the client devices, DHCP relay agents might be used to forward DHCP messages between segments. Check the DHCP relay configuration to ensure it is set up correctly.
IP Address Conflicts: Check for any IP address conflicts on the network. If multiple devices have the same IP address, it can cause DHCP offers to be dropped.
Network Connectivity Issues: Ensure that the DHCP server and the switch have proper network connectivity. Check for any cable or physical connection issues.
Firewall or Security Policies: Firewall rules or security policies might be blocking DHCP traffic, preventing DHCP offers from reaching the client devices. Review the firewall and security settings to ensure they allow DHCP communication.
Faulty Hardware: A faulty switch or network card could disrupt DHCP communications. Check the hardware components for any issues.

To troubleshoot and resolve DHCP offer dropping issues, consider the following steps:

Check DHCP Server Logs: Examine the logs on the DHCP server to see if there are any error messages or indications of issues.
Verify Network Configuration: Double-check the network configuration, VLAN settings, and DHCP relay configurations to ensure they are correctly set up.
Check for IP Address Conflicts: Use network scanning tools to identify any IP address conflicts on the network.
Test Connectivity: Verify network connectivity between the DHCP server and the switch by pinging them from each other.
Check Switch Port Settings: Review the configuration of the switch ports connected to the DHCP server and client devices.
Check Firewall and Security Policies: Review firewall and security settings to ensure they are not blocking DHCP traffic.
Restart DHCP Services: Consider restarting the DHCP server and the switch to clear any temporary issues.

If the issue persists and you're unable to resolve it on your own, consider seeking assistance from a network administrator or IT professional with expertise in networking and DHCP troubleshooting.

Graham Murison · ‎07-24-2023

Thanks to some tips from @Harold Ritter @Flavio Miranda I figured this out and it came down to the Security team a long time ago created a non-logging firewall rule that blocks ANY/ANY traffic for bootp and Netbios. Usually this works well and good practice until you are using the same firewall for inter-VLAN traffic. The broadcast relay (DISCOVER) was working Client > Relay > FW > Server, but the unicast OFFER failed at the firewall due to above (Server > FW). Once the Security team allowed bootp/dhcp & netbios between the required segments, the issue is fixed.

Below is the working config. Also multiple VLANs are reaching the same DHCP server and getting the correct IP lease. Posting for completeness.



========
no dhcp relay information option vpn 
interface Vlan100
description SVI for IT VLAN
vrf forwarding IT-vrf
ip address 10.96.xx.1 255.255.255.0
ip helper-address global 10.96.ab.20
no ip redirects
!
interface Vlan110
description SVI for First Floor VLAN
vrf forwarding Users1-vrf
ip address 10.96.xy.1 255.255.254.0
ip helper-address global 10.96.ab.20
!
interface Vlan120
description SVI for Second Floor VLAN
vrf forwarding Users2-vrf
ip address 10.96.yy.1 255.255.254.0
ip helper-address global 10.96.ab.20
!
interface Vlan130
description SVI for Third Floor VLAN
vrf forwarding Users3-vrf
ip address 10.96.yz.1 255.255.254.0
ip helper-address global 10.96.ab.20
!
interface Vlan140
description SVI for Fourth Floor VLAN
vrf forwarding Users4-vrf
ip address 10.96.zz.1 255.255.254.0
ip helper-address global 10.96.ab.20
========

I believe this is working since the IP helper address is also the gateway address for the DHCP scope. The DHCP server is Windows 2019.