4507R+E DHCP weird issue

[Neil Sheridan]

Hi,

We have a 4507R+E acting as the core L2/L3 switch with a number of 3650's dotted around the campus. Each 3650 is connected to the core via Fibre. For reasons I won't go into, we have around 100 PVLAN's, basically one per classroom. Our DHCP server is an MS 2016 server and has been working perfectly for the longest time. The past few weeks we've seen some strange issues.

From a clean reload, the clients will obtain a DHCP lease and work happily. After about 2 to 2 1/2 days, they stop receiving an address. The Primary PVLAN is on a 10.0.16.x/21 subnet. The DHCP server is on a 10.1.1.x/16 subnet and the correct IP Helper address has been added to the .16 SVI.

The 4507R+E has two SUP8 modules. If I reload slot 2, IP addresses are immediately handed out, if I reload the entire switch, the same occurs. Testing this morning, I managed to get an IP, and I could ping the GW (SVI IP), I could also ping the IP of the Switch, but I couldn't get any further. I've checked the routing and it all looks fine, not to mention after the reload all works as it should.

I haven't run any packet captures as the pressure is on when the issue is apparent. To add to the weirdness, we have a number of other Wireless SSID's on different subnets, when the client cannot get a .16.x/21 address, it does via a different SSID and subnet...

I can also confirm that there are a large number of available IP's left in the scope.

I'm hoping someone else may have a different thought process on how to dig further.

Cheers

[paul driver]

Hello

---

@Neil Sheridan wrote:

Hi,

We have a 4507R+E acting as the core L2/L3 switch with a number of 3650's dotted around the campus. Each 3650 is connected to the core via Fibre. For reasons I won't go into, we have around 100 PVLAN's, basically one per classroom. Our DHCP server is an MS 2016 server and has been working perfectly for the longest time. The past few weeks we've seen some strange issues.

From a clean reload, the clients will obtain a DHCP lease and work happily. After about 2 to 2 1/2 days, they stop receiving an address. The Primary PVLAN is on a 10.0.16.x/21 subnet. The DHCP server is on a 10.1.1.x/16 subnet and the correct IP Helper address has been added to the .16 SVI.

The 4507R+E has two SUP8 modules. If I reload slot 2, IP addresses are immediately handed out, if I reload the entire switch, the same occurs. Testing this morning, I managed to get an IP, and I could ping the GW (SVI IP), I could also ping the IP of the Switch, but I couldn't get any further. I've checked the routing and it all looks fine, not to mention after the reload all works as it should.

I haven't run any packet captures as the pressure is on when the issue is apparent. To add to the weirdness, we have a number of other Wireless SSID's on different subnets, when the client cannot get a .16.x/21 address, it does via a different SSID and subnet..

---

I tempted to say its related to your dhcp and dns but that would be just an assumption at this time.

What are your lease times for each dhcp scope in relation to the wired and wifi users, also is dhcp set to update dns or do are the clients?

Does this just occur on wired clients or wifi , if just either one I would say communication to your dhcp is okay

Do you have any duplicate dhcp and dns entries being logged?

[Georg Pauwen]

Hello,

when your clients don't get a lease, can the DHCP server ping the SVIs where the helper addresses are configured (and/or vice versa) ?

[Stuart C]

Have you run wireshark on the local machine and on the DHCP server to see if your requests are hitting the server?

[Neil Sheridan]

Thank you for your replies, and my apologies for the late reply, it was quite hectic last week.

In answer to the suggestions:

I haven't been able to run any packet captures. As soon as it goes down 1200+ people are offline, so the pressure is on the get it back up. I'd hoped I could 'nurse' it along until the next term break and runs some captures then. I know the clients can ping the SVI IP, and the management IP of the switch, but nothing further. 

DNS is successfully updated via the DHCP server, the logs confirm this. There are no other DHCP servers serving the same subnet or a portion of it. I also installed a new 2016 DHCP server with the same problems.

In an effort to get some stability, I moved back to the old physical DHCP server and placed it into a PVLAN Promiscuous port mode. So far so good. I'm a little concerned it could be a band-aid solution, and I would prefer to get to the bottom, but until I can have the WLAN down for a period of time, it'll have to do.

Thanks again

N

[paul driver]

Hello

You still have not confirmed if this issue is affecting all clients (wired/wifi) and if you are seeing any duplicate dns/dhcp entries.

Can you also confirm what software is running on core switches and any historical logging such as cpu/memory utilization

[Neil Sheridan]

Hi Paul,

From what I can tell it's only WiFi clients on the .16 subnet. Having said that, the lease time for wired clients is considerably longer, 3 days opposed to WiFi clients at 8 hours.

There are no duplicate DHCP addresses or DNS entries.

During the issue, the resources of the switch are normal (confirmed via Prime) and it's running cat4500es8-universalk9.SPA.03.10.01.E.152-6.E1, and has been for a few months.

Yesterday was the first day in 4 weeks with no issue reported, hopefully, it continues.