Bogus MAC addresses generated from docks/dongles

[Josh Morris]

I have been investigating this issue for a while now. I have about 20-30 different switchports around my campus where thousands of bogus mac addresses show up in ISE attached to specific ports. We're talking about MACs like '00:00:00:00:00:01'. Ones that are clearly bogus. Until now, I've had to develop profiles matching everything from those switchports so I could purge them nightly. I purge something between 2,000-6,000 each night. 

We believe to be narrowing in on certain docks or dongles causing this. One in general has been the Dell 'hockey puck' style dongle. It seems especially problematic. I would love to just eliminate the dongles but that's not always going to work. What other ways can I use to deal with this? Port-security would work, but it seems silly to just add mac address limitations on only a few ports across campus. 

[Arne Bier]

I see this too. Apart from purging them, there's nothing else you could do from the switch or ISE. And you're also forced to use muti-auth mode on the switch, due to all the additional MAC addresses in the DATA domain. 

Do you have a support contract with Dell to bring this to their attention?

I see all sorts of misbehaviour on end devices, that you normally wouldn't see or care about in a non-NAC environment - e.g. devices that come with 802.1X enabled from factory and just send garbage EAPOL requests every 60 seconds. That causes a lot of noise in ISE and I am constantly fighting to have these supplicants disabled (for customers who use MAB instead). The other perennial pain is the broken Ethernet device drivers in Windows that cause workstations to create 100 to 1000 successful 802.1X requests a day. Normally one requests is enough, but if the device driver is broken it will decide at random intervals to start over again. 

If you look closely at your failed RADIUS requests in ISE, you'll see this underworld in action.  And if you also look at your top 10 busiest endpoints, you might be amazed at how many Windows supplicants are just spamming all day long.

it's a waste of resources on ISE, and it's not ISE's job to be a type of NMS - but I often report these issues to the end user, because it creates a better experience all around.  

[Cristian Matei]

Hi,

   When you have AUTH enabled on layer 2 port, don't recommend using port-security, as AUTH is a dynamic port security method in the end, you might run into weird challenges along the way. You either let those dongles get network access and face the consequences, or configure auth violation action to restrict traffic.

Best,

Cristian.

[MHM Cisco World]

no need port-security 
run 
host mode single host 
this make port accept one MAC per one Port for data domain 

you seem use multi-auth or multi-host that make problem 

MHM

[Arne Bier]

I have to disagree with the point about host-mode. Host-mode will not prevent the problem of the endpoints generating spurious MAC addresses. If anything, host-mode will err-disable the interface as soon as the 2nd MAC address comes along. Which will cut the user off. That’s not viable.
The problem cannot be solved by any config on the switch itself or the RADIUS server. It’s an endpoint issue.

[MHM Cisco World]

But if he additional add authc violations restrict  the port will not go to err-disable.

Correct me if I am wrong 

MHM

[Arne Bier]

The problem with detecting "violations" is that the Ethernet frames can arrive in any order - the arrival is non-deterministic. For example, assume that MAC address A0 is the valid one, and that A1-A255 are the invalid/spurious MAC addresses. Josh doesn't tell us when this problem occurs (i.e. immediately, or over a long period of time, or just randomly). Anyway ... the switch has no way of knowing that MAC address A0 is the only valid one - that's ISE's job.  So if MAC A1 arrives first, followed by A0, then the err-disable/violation kicks in. Err-disable or restrict has the same end result - the innocent end user is blocked. That is bad news. It's possible that this could happen, especially since we cannot control what order the endpoint generates the traffic - and - this is a genuine case, if a switch reboots while the endpoint is connected, you will have connected end stations still generating frames, and once the interfaces are ready to process frames, you will have no idea what frame is going to come first - A0 or A1-255. Hence, chances are you will lock up that interface. 

Trust me - I have been there and done that with customers who have ESXi servers on access layer. They didn't want to add a MAB entry into ISE for all the VMs, because it was a very dynamic environment, and MAB was the only auth method.  I thought I was smart by using multi-host mode (the vmKernel frames Authorize the interface, and the VM traffic piggy backs on that).  It all worked according to plan until someone rebooted the switch. After switch came up, the VMs are still sending traffic like crazy, and guess which frames arrives first ... and lock up the interface.  it gets even worse with NIC teaming, because the secondary interface is UP/UP (no MAC address) and only when NIC Teaming is activated, the first MAC address will arrive ... but which one?  You have no control over that. Bottom line ... don't use multi-host mode if you don't know the ORDER in which traffic will arrive to authorize the interface.

Single-host mode is in a similar situation of falling foul of the non-determinism issue. 

Fix the end devices. Basta!

[MHM Cisco World]

I read about dongle in one of ISE cisco doc. But I forget where' if I found doc. I will share here.

MHM

[Arne Bier]

@Josh Morris have you looked into firmware updates for the dock? 

Try that first. If still persists, then update the Ethernet Device drivers on your PC.

A Dell laptop on a Dell dock should be using MAC passthrough mode, where the source Ethernet MAC address seen on the switch, is that contained in the laptop's BIOS. If passthrough is not enabled, then the MAC address of the dock would appear on the switch, which is not ideal. 

I have a theory that these bogus endpoints are actually valid Ethernet frames from the host, but where the dock (or device driver) has overwritten the source MAC address with some random register value. You could run a circular buffer "monitor capture" on that interface and analyse in Wireshark, to see what data is actually contained in one of these frames that have weird source MAC addresses. If it looks like traffic from the PC (check the source IP address in Wireshark) then my theory could be correct.

I would update things in this order, testing each time after each update to see what solved it (if that were the case)

1. Dock FW
2. PC BIOS update
3. Ethernet Device Driver update (preferably from the PC vendor website, not Microsoft)
4. Force a Windows Update

Changing dock types/models might also resolve it - the device driver chosen for the Windows Ethernet Adapter is based on the chipset used in the dock (Realtek or Intel, etc)

[Mark Potter]

Have also come across this with Apple Mac laptops and HP Docks, G2 shaped ones.
They can generate up to 4500 mac addresses on a port or just a few hundred.

As mentioned the only fix I've found is sometimes a firmware update on dock and laptop sometimes fixes it, if not replace the dock.

Pretty annoying to fluff around with a purge policy or manually deleting them.