Periodic DHCP conflicts on whole subnet

[mario.jost]

We manage several seperated networks that provide simple internet acces to guest clients. There are about 50 locations. In the past 6 months, we see a strange behaviour. Sometimes, the DHCP service declines all available IP addresses due to conflicts. it looks like this in the log:

Jun  9 13:24:46.428: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.188.
Jun  9 13:56:58.110: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.189.
Jun  9 13:57:01.606: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.190.
Jun  9 13:57:05.106: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.131.
Jun  9 13:57:08.570: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.132.
Jun  9 13:57:11.954: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.133.
Jun  9 13:57:15.426: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.134.
Jun  9 13:57:18.806: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.136.
Jun  9 13:57:22.098: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.137.
Jun  9 13:57:25.778: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.138.
Jun  9 13:57:29.062: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.139.
Jun  9 13:57:37.378: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.140.
Jun  9 13:57:40.678: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.141.
Jun  9 13:57:43.934: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.142.
Jun  9 13:57:47.230: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.143.
Jun  9 13:57:50.482: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.144.
Jun  9 13:57:53.866: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.145.
Jun  9 13:57:57.038: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.146.
Jun  9 13:58:00.330: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.147.
Jun  9 13:58:03.578: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.148.
Jun  9 13:58:06.874: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.149.
Jun  9 13:58:10.138: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.150.
Jun  9 13:58:13.534: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.151.
Jun  9 13:58:16.774: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.152.
Jun  9 13:58:20.086: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.153.
Jun  9 13:58:23.346: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.154.
Jun  9 13:58:26.626: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.155.
Jun  9 13:58:29.926: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.156.
Jun  9 13:58:33.430: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.157.
Jun  9 13:58:36.774: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.158.
Jun  9 13:58:40.078: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.159.
Jun  9 13:58:43.662: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.160.
Jun  9 13:58:47.038: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.161.
Jun  9 13:58:50.426: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.162.
Jun  9 13:58:53.710: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.163.
Jun  9 13:59:18.858: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.164.
Jun  9 13:59:22.086: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.165.
Jun  9 13:59:25.358: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.169.
Jun  9 13:59:28.698: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.171.
Jun  9 13:59:29.954: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.174.
Jun  9 13:59:31.178: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.176.
Jun  9 13:59:32.422: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.173.
Jun  9 13:59:33.694: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.168.
Jun  9 13:59:34.938: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.167.
Jun  9 13:59:36.150: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.170.
Jun  9 13:59:37.402: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.178.
Jun  9 13:59:38.638: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.179.
Jun  9 13:59:39.946: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.182.
Jun  9 13:59:41.230: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.183.
Jun  9 13:59:42.478: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.130.
Jun  9 13:59:43.706: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.180.
Jun 11 07:05:52.548: %DHCPD-4-DECLINE_CONFLICT: DHCP address conflict:  client 9849.14ea.649f declined 10.12.227.166.

The conflicts are always logged as gratuious ARP:

roWAL01#show ip dhcp conflict 
IP address        Detection method   Detection time          VRF
10.12.227.188     Gratuitous ARP     Jun 09 2021 01:24 PM    DSL                             
10.12.227.189     Gratuitous ARP     Jun 09 2021 01:56 PM    DSL                             
10.12.227.190     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.131     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.132     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.133     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.134     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.136     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.137     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.138     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.139     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.140     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.141     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL                             
10.12.227.142     Gratuitous ARP     Jun 09 2021 01:57 PM    DSL 

Its always the same MAC address per location that is responsible for this. I did not put in all the logs, but it usually does around 15 conflicts per minute until all the IPs are exhausted and clients cant get any IP addresses anymore. There are workarounds with deactivating IP dhcp conflict logging and stuff, but i would rather get to know the cause for this issue.

irst it appeared in Jan/Feb 2021 in just 2 locations and we thought this must be a local issue. But now we have this issue on and off (maybe two incidents a week) in various locations (about 15 at this point). My fear is, that this issue is getting worse and spreads to all locations eventually.

As this is a guestnet, various clients can walk by and connect, there is no restriction on it except for a login that is displayed on our counter. My guess is that something like mac randomisation could be responsible for something like this. We have an internal WiFi with our own devices in every location and the issue never appears on that network. Im just putting this out there to see if we are the only one with this kind of behaviour or if this is something that alot of networkadmins have to deal with because some software changes are being rolled out by Apple/Google/Microsoft. I did some digging in the logs 

to get all the MAC adresses that caused the issue. They all look very similar:

9849.1415.3595
9849.142b.5d1e
9849.1451.37af
9849.14ae.b0ce
9849.14ea.649f

The fact that these are multicast addresses explains the issue that i cant locate them to an AP within prime. Prime will show me the unicast address. It is very difficult to try to pinpoint the issue as the issue just occurs for several minutes and then disappears for multiple days or even weeks. Has anyone seen anything like this before or can help me how we could taccle such an issue? We use C897VA-K9 & C892FSP-K9 both running Version 15.7(3)M4a. DHCP servers should not respond to multicast source mac addresses, so maybe there is a bug but i couldnt find anything that matches.

[paul driver]

Hello

For the logging i would suggest use a EEM script to clear these logs on a regualr basis so not to fill up the buffer and as such negate the dhcp server from denying address allocation or just turn off dhcp conflict logging off.

event manager applet dhcp-conflict
event timer cron cron-entry "23 59 * * *"
action 1.1 cli command "enable"
action 1.2 cli command "clear ip dhcp conflict *"
action 1.3 syslog msg "DHCP conflict records cleared"

or

no ip dhcp conflict logging 

[mario.jost]

Dear Paul

Thanks for your reply, but this i just a workaround. As i wrote in the post, i would rather fix the issue than just konfigure a quick fix to never have to deal with it again:

---

@mario.jost wrote:

There are workarounds with deactivating IP dhcp conflict logging and stuff, but i would rather get to know the cause for this issue.

---

Regards,

Mario

[paul driver]

Hello

It s indeed a workaround, however you need to make sure your client obtain dhcp allocation and with the conflict logging filling up they wont obtain leases so the suggested config should do that, long term you can also negate those mac-address on the switches

mac address-table static 9849.1415.3595 vlan xx drop
mac address-table static 9849.142b.5d1e vlan xx drop
mac address-table static 9849.1451.37af  vlan xx drop
mac address-table static 9849.14ae.b0ce vlan xx drop

mac address-table static 9849.14ea.649f vlan xx drop

or deny at port level

mac access-list exteneded nomac
deny 9849.1400.0000 0000.00ff.ffff any
permit any any

int x/x
mac access-group nomac in

Lasty do you have clients that have their wired and wifi nics enabled at the same time?

[mario.jost · Accepted solution]

After intense troubleshooting, i could get to the root of the problem. The devices at fault here, are the newer mercedes-benz cars that have these new screens called MBUX (mercedes-benz user experience) built in. These can connect to Wifi Networks, but dont have a browser built in. So they connect to our guest wifi which does not have a password, but a splash page. They cant login because they dont have a browser to display the splash page. The wireless controller does not let them connect to the internet as long as they havent authenticated on the splash page. The MBUX recognizes, that it has no internet connection with the given IP address. If it would do a DHCP release and renew, it would just get the same IP address as most DHCP servers have a remember feature built in. So it does a DHCP DECLINE and a new DHCP DISCOVER right afterwards. With this, it gets a new IP address because the old one has been set on the conflict table by the router. Im guessing the device (or rather the programmers behind it) is hoping that the new IP provides connectivity. Because non of the IPs do (cause no splash page auth) all the IPs in a subnet are quickly tried and land on the conflict table.

I wrote everything in a summary and placed it at Daimler to fix this. They are clearly not following the RFC. So we are a Mercedes-Benz dealership and therefore experience this issue on all locations. We created a script that runs hourly, counts the amount of conflicts and if they exceed a certain threshold, blocks the mac address like Paul suggested. I dont think that alot of businesses would have to deal with such an issue as it is very specific (MBUX & unprotected Wifi with splash page). Nevertheless, i wanted to share the result here in case anyone stumbles upon the same issue in the future. Thanks for the steer in the right direction.

[balaji.bandi]

Is the DHCP in the router or you have dedicated DHCP Servers ?

[Giuseppe Larosa]

Hello @mario.jost ,

actually the MAC addresses are unicast

98:49:14 Wistron Neweb Corporation from wireshark OUI search page

The explanation is the following:

a MAC address is muticast if the I/G bit is set to one . But to be noted the I/G bit is the less meaningful bit in the most significant byte so 0x098 = 10011000

Each byte is sent on wire on reverse order less meaningful bit first most meaning bit last.

Hope to help

Giuseppe