MAC addresses appear on incorrect ports

[MattD2010]

I am having a problem with several workstations not being able to access the network. Upon further investigating I found that the MAC address is appearing on different ports throughout the VLAN. The MAC may appear on different switches, and even different buildings (as long as they have ports on that VLAN). I am at a loss as to what is happening. We don't have port-security enabled on any ports and are currently using dot1x port authentication.

Example: Say "Computer A" is connected to port gi1/0/2 on switch 1 and "Computer B" is connected to port gi1/0/4 on switch 2, "Computer B" MAC address may at anytime show up on switch 1 port gi1/0/2. This will cause "Computer B" to be unable to access anything on the network as the layer 2 traffic is being passed through the wrong port.

To resolve this temporarily, I would shutdown gi1/0/2 on Switch 1 and restart the port. Then "Computer A" will appear correctly as a dynamic entry in the mac address-table, and "Computer B" will be listed on the correct interface.

When this happens the entries are listed as "Static" in the address-table.

Standard port config

------------------------
switchport access vlan XXX
switchport mode access
switchport voice vlan XXX
authentication control-direction in
authentication event fail action next-method
authentication event server dead action authorize
authentication event server dead action authorize voice
authentication event server alive action reinitialize
authentication host-mode multi-auth
authentication order dot1x mab
authentication priority dot1x mab
authentication port-control auto
authentication violation restrict
mab
dot1x pae authenticator
dot1x timeout tx-period 10
dot1x max-reauth-req 1
spanning-tree portfast
spanning-tree bpduguard enable

Any help is appreciated.

[brselzer]

Hello,

When you see a bad mac on the wrong port, is it authenticating with dot1x or mab? You can check with show authentication session or show access session. 

It sounds like you have port reflection going on. Maybe a bad nic driver or something. The hosts on your network are looping packets back into the network without changing the source mac so you learn it on many different ports. 

Hope that helps!

[MattD2010]

---

@brselzer wrote:

Hello,

When you see a bad mac on the wrong port, is it authenticating with dot1x or mab? You can check with show authentication session or show access session. 

It sounds like you have port reflection going on. Maybe a bad nic driver or something. The hosts on your network are looping packets back into the network without changing the source mac so you learn it on many different ports. 

Hope that helps!

---

I will verify tomorrow when I am back at the office; however, when I looked at the ISE this afternoon it appeared that it was using mab when it was authenticating on the wrong ports.

[MattD2010]

The incorrect MAC addresses are being authenticated using mab. The correct authentications are all dot1x.

[brselzer]

Hello,

That's what I expected. Since mab only needs a packet source mac to authenticate. You should probably start gathering data on the clients connected to the ports where macs are learned in the wrong place. Same model, OS, NIC, driver, etc. I believe your hosts are doing packet reflection. For whatever reason, the NIC of your hosts is looping a packet back into the network so a packet with source mac A comes in a port connected to a PC with mac B. This causes it to authenticate with MAB and get statically assigned to that interface. Until it times out, the valid host on that port will not be able to communicate. You could do a SPAN of the interface to verify but that might be difficult if the interface is jumping around.

Hope that helps!

[MattD2010]

---

@brselzer wrote:

Hello,

That's what I expected. Since mab only needs a packet source mac to authenticate. You should probably start gathering data on the clients connected to the ports where macs are learned in the wrong place. Same model, OS, NIC, driver, etc. I believe your hosts are doing packet reflection. For whatever reason, the NIC of your hosts is looping a packet back into the network so a packet with source mac A comes in a port connected to a PC with mac B. This causes it to authenticate with MAB and get statically assigned to that interface. Until it times out, the valid host on that port will not be able to communicate. You could do a SPAN of the interface to verify but that might be difficult if the interface is jumping around.

Hope that helps!

---

That makes sense. I went and gathered information on a few of the workstations and have not found any similarities between them. We are working with different OS (win 8 and win 10), different NICs (realtek and intel), different drivers, etc. I have updated the NIC drivers and reinstalled the NICs yesterday, however it appears the problem returned this morning (or overnight). Would it make sense to disable MAB on those ports until we figure out the NIC problem?

[brselzer]

Hello,

Sorry for the delay in response. You can remove authentication it if you want but I am not sure it will solve your problem, just change the symptoms. For example, now that authentication isn't there any mac can be learned on your access port. If a broadcast packet from your gateway is reflected then you learn the default gateway mac on that port, all traffic on that switch will be black holed for a period of time. I would expect another packet to come in from the gateway on the right interface shortly so it would be smaller intervals of loss.

The only solution I see from the network side is to either static the mac addresses which isn't scalable or turn off MAB and leave just dot1x which might break devices that don't support dot1x. However, if you know the handful of clients that are seeing the issue, maybe just turn off MAB on those ports but leave dot1x enabled. 

Hope that helps!

[MattD2010]

---

@brselzer wrote:

Hello,

Sorry for the delay in response. You can remove authentication it if you want but I am not sure it will solve your problem, just change the symptoms. For example, now that authentication isn't there any mac can be learned on your access port. If a broadcast packet from your gateway is reflected then you learn the default gateway mac on that port, all traffic on that switch will be black holed for a period of time. I would expect another packet to come in from the gateway on the right interface shortly so it would be smaller intervals of loss.

The only solution I see from the network side is to either static the mac addresses which isn't scalable or turn off MAB and leave just dot1x which might break devices that don't support dot1x. However, if you know the handful of clients that are seeing the issue, maybe just turn off MAB on those ports but leave dot1x enabled. 

Hope that helps!

---

I did remove MAB from the ports that appear to be reflecting the MAC addresses with no luck. It is still populating the address-table with the wrong macs; however now they are “unauthorized” not “authorized” with MAB...the end user is still losing network connectivity until I shutdown the port.

Would statically assigning the Mac stop that, as a temp solution?

[brselzer]

Hello,

Yea. I would think putting in static macs would stop the problem because you are no longer relying on dynamic learning. However, if you do that, I would static that mac address of the default gateway to the uplink just to be safe. 

Hope that helps!

[MattD2010]

I have looked into this problem further and found that each of these workstations have "Kaspersky Network Agent" installed on them. I have uninstalled that network agent on the workstations. It appears to have helped the problem, however, it has not completely taken care of the problem. I have also re-imaged one of the workstations that appears to be passing the wrong packets; again with no change.

Are there any other possibilities that could explain this problem? Maybe a bug in the firmware somewhere (ex. CSCuq42892 on the core switch)?

Thank you

[MattD2010 · Accepted solution]

I found that the problem was related to SCCM "Wake up Proxy" being enabled.

https://docs.microsoft.com/en-us/sccm/core/clients/deploy/plan/plan-wake-up-clients

Once this was disabled the problem appears to be resolved.