BRIEF
- Anyone else seeing WiFi clients issuing bursts of DHCP Requests which trigger 'ip dhcp snooping rate limit xx' thresholds on Catalysts?
DETAIL
- The flock of Catalyst 2960X at the access-layer have several hardening features enabled, including the DHCP Snooping feature
interface GigabitEthernet1/0/1
[...]
storm-control broadcast level 1.00
storm-control multicast level 1.00
storm-control action shutdown
storm-control action trap
spanning-tree portfast edge
spanning-tree guard root
ip dhcp snooping limit rate 25
Intermittently, some ports see bursts of DHCP and trigger the err-disable behavior
12:27:34 5n-2-esx dhcp-rate-limit error detected on Gi1/0/15, putting Gi1/0/15 in err-disable state
12:29:34 5n-2-esx-mgmt Attempting to recover from dhcp-rate-limit err-disable state on Gi1/0/15
Tracking these down, I find that the Catalyst ports affected by this event feed Wireless Access Points (Meraki MR33)
A typical day might include ~5-25 of these events (from a population of ~70 WAPs servicing ~600 WiFi clients)
I have run ethanalyzer on the upstream L3 gear to capture DHCP frames on the Sup card (aka 'in-band')
nxos# ethanalyzer local interface inband capture-filter "udp port 67 or (vlan and udp port 67)" limit-captured-frames 500000 capture-ring-buffer files 64 write bootflash:2020-02-16/mdf-a-rtr-inband-dhcp.pcap
In these pcaps, I see bursts of DHCP Discovers (and corresponding DHCP Offers from the DHCP server) -- 25 or 26 in a row and then silence... I interpret this behavior as consonant with the dhcp-rate-limit behavior, i.e. the Cat2960X forwarded ~25 DHCP frames within a single second, then disabled the WAP port.
For grins, I increased the dhcp-rate-limit parameter from 25 to 200 ... and captured a pcap in which 200 DHCP Discovers appeared in a single second
BTW: in these pcaps, the offending client sets the DHCP Transaction ID in an apparently random fashion. [Normally, a client which wants a DHCP address will set the DHCP Transaction ID to some starting number and then increment it by one with each fresh DHCP Discover.] This smells like buggy behavior to me: (a) excessively rapid series of Discovers, plus (b) chaotic Transaction ID
==> Question: does anyone know how I would go about filing a bug report with the Android or iOS folks (I have seen both Android phones and iPhones/iPads as sources of these DHCP bursts)
Total downtime for the WAP is ~3-4 minutes, i.e. (2) minutes during which the Cat2960X has the port disabled, and ~2 minutes for the MR33 to reboot, once the Cat2960X re-enables the port
IMPACT
I'm not clear on the impact to WiFi clients. For areas serviced by a single WAP, I suspect that the impact is substantial (i.e. no WiFi connectivity for ~4 minutes for clients located in that area). For areas serviced by multiple WAPs, I don't know what the end-user experience looks like, as the device roams away from the now silent WAP to an active one
WISHFUL THINKING
What I really want is the 'dhcp-rate-limit' feature implemented on the Meraki WAPs -- they don't currently support this feature. [I have submitted an enhancement request.] i.e. I want the WAPs to shut off aberrant clients
==> Question: Does anyone know if Cisco WAPs (aka the Catalyst 9000s) implement dhcp-rate-limit?
REMEDIATION
Seems to me that I could pursue one or more of the following:
(a) Investigate traffic-shaping on the MR33s (perhaps this feature would support throttling DHCP bursts to something that the Domain Controllers can tolerate)
(b) Disable dhcp-rate-limit on the Catalysts and just let the Domain Controllers get pounded by DHCP Discovers. Perhaps they can handle this just fine
(c) Live with the intermittent loss of WAPs -- it's only for a few minutes per incident
==> Question: does anyone see another remediation option?
ANYONE ELSE?
==> Question: Anyone else seeing this behavior?
--sk
