Lots of MAB errors with "phantom" MACs..some on multiple interfaces

BillyBaroo · ‎12-06-2019

So I had this dropped into my lap and I can't really make any headway on it. We started noticing some MAB failures and started looking for the devices. Although being reported on a certain interface there is nothing but what is supposed to be there; an ip phone and a pc behind it. Logs will even show the same MAC on multiple interfaces across different switches and vlans. It's like "phantom" devices are trying to access an interface, but get dropped at the MAB. Here's a snipit of one of our logs:

16694398: Dec 6 14:55:48.880 EST: %MAB-5-FAIL: Authentication failed for client (b4b6.8690.6566) on Interface Gi2/16 AuditSessionID 0A4B266E00033041A38758D8
16694399: Dec 6 14:55:51.957 EST: %MAB-5-FAIL: Authentication failed for client (54bf.6495.e670) on Interface Gi5/25 AuditSessionID 0A4B266E0003AB3E8C35253C
16694400: Dec 6 14:55:55.024 EST: %MAB-5-FAIL: Authentication failed for client (10e7.c6a7.83e3) on Interface Gi1/38 AuditSessionID 0A4B266E000391DF1E06D7B4
16694401: Dec 6 14:55:56.057 EST: %MAB-5-FAIL: Authentication failed for client (b4b6.8646.cafa) on Interface Gi5/7 AuditSessionID 0A4B266E00032D0598BC6DAC
16694402: Dec 6 14:55:57.087 EST: %MAB-5-FAIL: Authentication failed for client (b4b6.8690.6545) on Interface Gi2/20 AuditSessionID 0A4B266E0003426FE0374E68
16694403: Dec 6 14:55:57.087 EST: %MAB-5-FAIL: Authentication failed for client (b4b6.8690.6db0) on Interface Gi1/6 AuditSessionID 0A4B266E00033D48CF503284
16694404: Dec 6 14:55:57.087 EST: %MAB-5-FAIL: Authentication failed for client (b4b6.8648.415c) on Interface Gi1/31 AuditSessionID 0A4B266E00034071D9992C4C
16694405: Dec 6 14:55:58.104 EST: %MAB-5-FAIL: Authentication failed for client (80ce.621e.6f0a) on Interface Gi5/17 AuditSessionID 0A4B266E0003ECD9E88B7958

We're getting upwards of 1300 failures on some interfaces daily. If I pull the mac list for the interface only the two that should be on there are listed. Where could these other MACs be coming from? My research hasn't produced much for me as of yet.

pieterh · ‎12-09-2019

"Logs will even show the same MAC on multiple interfaces across different switches and vlans"

mac-vendor lookup associates the given MACs with valid vendors like DELL HP etc.

first check ip these PC's do not have a management interface shared with the data interface (iDRAC, CIMC, ILO)

if not -> looks like packets come from a software modified mac address -->> do a serious malware scan !

if yes these management interfaces may be the source (but guess not with same mac on multiple switchports)

other option is to create a SPAN port and do a packet capture to see what data is sent in the packets with these MAC addresses.

BillyBaroo · ‎12-10-2019

I majority of the MACs involved in the failed authentication have an OUI from HP.

Georg Pauwen · ‎12-09-2019

Hello,

could be the bug below. Known fixed releases are:

15.2(2)E
15.0(2)SE8
15.2(6.3.0i)E
15.2(5.0)ST
15.2(4.0)ST
15.2(4.0.64a)E
15.2(2.2.32)EA
3.6(0)E

abnormal dot1x authentication failure msg from some specific mac address
CSCum75962
Description
Symptom:
Frequent dot1x authentication failure msg came out at many different port, all of them has same source mac address, such as:
053909: Jan 23 02:56:39.597 BJ: %AUTHMGR-5-START: Starting 'dot1x' for client (0025.4619.7214) on Interface Gi1/0/28 AuditSessionID AC014579000001B524053E46
053910: Jan 23 02:56:55.073 BJ: %DOT1X-5-FAIL: Authentication failed for client (0025.4619.7214) on Interface Gi1/0/28 AuditSessionID AC014579000001B524053E46
053911: Jan 23 02:56:55.073 BJ: %AUTHMGR-7-STOPPING: Stopping 'dot1x' for client 0025.4619.7214 on Interface Gi1/0/28 AuditSessionID AC014579000001B524053E46
053912: Jan 23 02:56:55.073 BJ: %AUTHMGR-5-START: Starting 'mab' for client (0025.4619.7214) on Interface Gi1/0/28 AuditSessionID AC014579000001B524053E46
053913: Jan 23 02:56:55.099 BJ: %MAB-5-FAIL: Authentication failed for client (0025.4619.7214) on Interface Gi1/0/28 AuditSessionID AC014579000001B524053E46
053914: Jan 23 02:56:55.099 BJ: %AUTHMGR-7-STOPPING: Stopping 'mab' for client 0025.4619.7214 on Interface Gi1/0/28 AuditSessionID AC014579000001B524053E46
053915: Jan 23 02:56:55.099 BJ: %AUTHMGR-5-FAIL: Authorization failed or unapplied for client (0025.4619.7214) on Interface Gi1/0/28 AuditSessionID AC014579000001B524053E46

056306: Jan 23 12:23:24.894 BJ: %AUTHMGR-5-START: Starting 'dot1x' for client (0025.4619.7214) on Interface Gi1/0/23 AuditSessionID AC014579000001D42811BF05
056307: Jan 23 12:23:40.370 BJ: %DOT1X-5-FAIL: Authentication failed for client (0025.4619.7214) on Interface Gi1/0/23 AuditSessionID AC014579000001D42811BF05
056308: Jan 23 12:23:40.370 BJ: %AUTHMGR-7-STOPPING: Stopping 'dot1x' for client 0025.4619.7214 on Interface Gi1/0/23 AuditSessionID AC014579000001D42811BF05
056309: Jan 23 12:23:40.370 BJ: %AUTHMGR-5-START: Starting 'mab' for client (0025.4619.7214) on Interface Gi1/0/23 AuditSessionID AC014579000001D42811BF05
056310: Jan 23 12:23:40.395 BJ: %MAB-5-FAIL: Authentication failed for client (0025.4619.7214) on Interface Gi1/0/23 AuditSessionID AC014579000001D42811BF05
056311: Jan 23 12:23:40.412 BJ: %AUTHMGR-7-STOPPING: Stopping 'mab' for client 0025.4619.7214 on Interface Gi1/0/23 AuditSessionID AC014579000001D42811BF05
056312: Jan 23 12:23:40.412 BJ: %AUTHMGR-5-FAIL: Authorization failed or unapplied for client (0025.4619.7214) on Interface Gi1/0/23 AuditSessionID AC014579000001D42811BF05

this happened at every customer switch stack and the source mac has regular pattern:

SO-OAW3750-18F-C1 0064.4001.949c
SO-OAW3750-19F-C1 0064.4001.949d
SO-OAW3750-20F-C1 0025.4619.7213
SO-OAW3750-21F-C1 0025.4619.7214
SO-OAW3750-22F-C1 0025.4619.7215
SO-OAW3750-23F-C1 0025.4619.7216
SO-OAW3750-25F-C1 0025.4619.7217

Conditions:
enable dot1x authentication

Workaround:
none

Further Problem Description:
this problem happened randomly at many ports of every switch stack and the mac address OUI is Cisco. we used monitor session to try to capture those frames but can't capture.