DHCP Serving through Helper-Address intermittent issue

[datacenter]

Hello guys!

I'm facing an issue that is really awkward related to DHCP IPv4 serving through a 9200L stackpile that acts like a L3 CORE.

Intermittently a device takes a lot of time to get an IP or even times out and stay at 169.x.x.x, demanding a reconnection to get IP.

That happens every 5-6 attempts, in different vlans, and also something similar on a LOCAL DHCP POOL sitting in that same core switch.

It was running an old IOS version (17.0.x), tried upgrading to our standard version (17.6.5 Bengaluru) but made no difference.

Interesting is that I moved that same LOCAL DHCP POOL to another stackpile we have (2x 3850x sfp switches) and in that switch stack the issue is not seen.

Relevant config on 9200L stack pile:

SWXXXXX#sh run | i dhcp
ip dhcp pool VLAN902
ip dhcp pool VLAN901

ip dhcp pool VLAN902 (this is the local pool we faced the issue)
network 192.168.1.0 255.255.255.0
domain-name weg.net
default-router 192.168.1.254 (svi on 3850x fiber switch pile currently)
dns-server 8.8.8.8 1.1.1.1
lease 0 4

interface Vlan604 (this is a DHCP server dependant network we faced issues)
ip address 10.160.4.254 255.255.255.0
ip helper-address 10.160.1.15 (our dhcp server)
no ip redirects
end

Server interfaces:

interface GigabitEthernet3/0/6
description xxxxxxxxxx
switchport mode trunk
ip dhcp snooping trust
end

We've opened a MS ticket too, they stated that some of the DHCP requests are not reaching the server, that is why we started investigating network layer too.

PS: Added ip dhcp snooping trust to test, but ip dhcp snooping is not active in global config.

PS2: DHCP server is also sitting in that same 9200 switch stack

Thanks in advance!

[MHM Cisco World]

Switch(config-if)#no ip unreachables

Disable this under SVI and check again.

[datacenter]

What is this command for?
I can try it, just need to understand it better as it is already in production.

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-series-switches/217429-troubleshoot-slow-or-intermittent-dhcp-o.html

Sure check above Cisco doc. About why we need disable this command under svi.

MHM

[Richard Burts]

There are several things in the original post that I would like to understand better.

- is the server interface really a trunk port? What vlans are carried by that trunk?

- you indicate that vlan 604 is one where you experience the issue. But the subnet associated with that interface does not relate to any of the information that you provide about the server or the dhcp pool. Can you provide appropriate details?

- I suggest that you remove ip dhcp snooping trust from the config. And at least temporarily I suggest removing ip dhcp snooping trust.

When the issue is happening are there any log messages that relate to dhcp?

[datacenter]

- is the server interface really a trunk port? What vlans are carried by that trunk?

It is a VMWare host with multiple VMs, so it carries a trunk all. DHCP server is sitting on vlan 601.

- you indicate that vlan 604 is one where you experience the issue. But the subnet associated with that interface does not relate to any of the information that you provide about the server or the dhcp pool. Can you provide appropriate details?

VLAN604 is an example of DHCP server based VLAN, so it solely relies on helper-address + dhcp server to serve IP

VLAN902 is an example of local DHCP pool based on the switch, so it serves IP without sending request to DHCP server

Mentioned both because I've seen issues on both scenarios (local dhcp pool and dhcp server pool), and in both scenarios if I move the SVI + pool (in VLAN902) to a different switch problem ceases, so it really looks like there is someting on that 9200 stackpile.

That is strange, as we have dozens of branches running same setup with 9200L stackpile doing core switch and we dont face that same issue.

I can share details on the 3850X stackpile we are currently hosting VLAN902 pool if that helps. That stackpile is not facing any issue.

- I suggest that you remove ip dhcp snooping trust from the config. And at least temporarily I suggest removing ip dhcp snooping trust.

It was running without and dhcp snooping at all, globally or to the interface, I've added just to do some testing.

But I'll remove eithe way, as it did not help at all.

It looks like I'm missing something, but I've cross checked config with other piles we have and it all looks good.