DHCP Severe Problem on Cisco Catalyst 9K Switches

[Rami Ibrahim]

Hello,

Please help me because I have a serious problem that I don't know the cause of it yet. I have a general enterprise network with Collapse Core Design where clients connected on access switches are not able to get IP addresses through DHCP from the DHCP server.

When endpoints request IP through DHCP sometimes it works sometimes not, some IP Phones able to renew others won't, some endpoints even couldn't get an IP at all. I have been troubleshooting this for 2 days and upgrade Access SW IOS-XE from 17.3.4 to the latest 17.9.3 and the same problem persists.

My network setup is as follows:

Endpoints (Cisco IP Phone, PCs) >> Access Switch (2 x 9300) Stack >> Core Switch (2 x 9606) Stackwise-Virtual >> DHCP server (Microsoft Windows Server 2012) connected also directly on the Core SW.

I have Access SW connected to the Core SW through a port-channel (2 x 10G) links and finally I realized that when I shutdown the second uplink (between member no 2 in the Access SW stack & Standby Core SW) all endpoints were able to get IP through DHCP and all start to work as expected. When I revert both links to the port-channel the issue repeated again. 

Any ideas on what could result in this behavior since the port-channel considered a single link logically from an STP perspective and it should not cause this issue. 

Note: TAC case opened 2 days ago and the engineer still checking with no findings yet.

Thank you,

Kindest Regards,  

[M02@rt37]

Hello @Rami Ibrahim,

You have noticed that shutting down the second uplink in the port-channel between the access switch stack and the standby core switch resolves the problem temporarily:

Do you check if that the channel-group configuration and load-balancing settings are consistent on both ends ?

[Rami Ibrahim]

Dear M02@rt37 

 Thank you for your reply,

 The port-channel on both sides is simple trunk, and the load balancing method is different, on access switch it is dst-ip and on the Core its  src-dst-mixed-ip-port. 

Core SW is running 17.2.1

Regards,

[Flavio Miranda]

Hi

 Do you have dhcp snooping?

[Rami Ibrahim]

Dear Falvio,

 Before upgrading the Access SW, I tried to enable DHCP snooping and all works, I thought that was a bug and I disable DHCP snooping save the config and upgrade the SW, after upgrading same issue happens re-enabling DHCP snooping also did not make any change.

Regards, 

[MHM Cisco World]

Server connect to SW that you keep it UP during test' other SW that you down it link it not direct connect to Server?

[Rami Ibrahim]

Dear MHM, 

 I am sorry I did not get you point Sir

[MHM Cisco World]

Server is connect to one of Core SW virtual stack'

Then you mention that you shut one link of Port channel' are the link you shut down connect to core that it not direct connect to dhcp server 

[Rami Ibrahim]

Yes the server is on a different port channel 

[MHM Cisco World]

There two think here either the dhcp is drop becuase of queue of cpu is full'

I share below link to solve queue issue

Or the packet drop when pass over svl from one core to other.

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-series-switches/217429-troubleshoot-slow-or-intermittent-dhcp-o.html

[Rami Ibrahim]

Thank you sir 

Actually, I checked the queue found no drops, also on Core SW SVI I have no ip redirect no ip unreachable and no ip proxy arp.

Thanks again.

[MHM Cisco World]

You check queue for both Cores ?

[Rami Ibrahim]

Yes both core don't have drops in Broadcast/ICMP redi queues. 

Checked also the access SW queue found also no drops. 

[MHM Cisco World]

Show controllers ethernet-control <svl>

Chech if there is drop

[Rami Ibrahim]

I attached the output of show platform hardware fed switch 1 qos queue stats internal cpu policer on Core SW #1 and Core SW #2, there drops but for a different queue (Sw forwarding). 

Also attached the show controllers ethernet-control on both links of Core SW going downstream to Access SW.

No drops for any queue on Access SW.

Thanks a lot.