Mysterious Cisco MAC that moves form port to port

Cory Peterson · ‎08-16-2018

I have been seeing an odd issue where a MAC address will show up in the authentication sessions on ISE and the switch but when I try to track it down on the switch only to find it is not on the same port that is reported in ISE. And if I go back to find it again later in the day or the next it will show up on a different port.

The MAC in question at this point is: 2C:5A:0F:41:61:B3

So the MAC only shows up in the authentication sessions. It never shows up in the CAM table.

While trying to find out what this device is the below outputs threw me for a loop:

Switch-4506#sho auth sess mac 2C:5A:0F:41:61:B3

Interface    MAC Address    Method  Domain  Status Fg Session ID
----------------------------------------------------------------------
Gi5/16       2c5a.0f41.61b3 mab     DATA    Auth      0AB4FF0A000154524CCE1530

Switch-4506#sho cdp ne gi 5/16 det
-------------------------
Device ID: SEP0013C3458FE4
Entry address(es): 
  IP address: 10.180.111.30
Platform: Cisco IP Phone 7940,  Capabilities: Host Phone Two-port Mac Relay 
Interface: GigabitEthernet5/16,  Port ID (outgoing port): Port 1
Holdtime : 133 sec
Second Port Status: Up

Version :
P0030801SR02

advertisement version: 2
Duplex: full
Power drawn: 6.300 Watts


Total cdp entries displayed : 1
Switch-4506#                      
Switch-4506#sho mac add add 2C:5A:0F:41:61:B3
No entries present.

Switch-4506#sho mac add in gi 5/16
Unicast Entries
 vlan     mac address     type        protocols               port
---------+---------------+--------+---------------------+-------------------------
  10      c85b.76c3.3b58   dynamic ip,ipx,assigned,other GigabitEthernet5/16        
 110      0013.c345.8fe4   dynamic ip,ipx,assigned,other GigabitEthernet5/16        


Switch-4506#sho auth sess in gi 5/16

Interface    MAC Address    Method  Domain  Status Fg Session ID
----------------------------------------------------------------------
Gi5/16       2c5a.0f41.61b3 mab     DATA    Auth      0AB4FF0A000154524CCE1530
Gi5/16       c85b.76c3.3b58 dot1x   DATA    Auth      0AB4FF0A0001008841F5D09C
Gi5/16       0004.f2e6.2176 N/A     UNKNOWN Unauth    0AB4FF0A0001546F4CCFED88

Anyone else seen this or have an explanation of what this is?

paul · ‎08-16-2018

Where is the phone you showed on that port in the show CDP.

what does the detailed output of show auth session look like?

Cory Peterson · ‎08-17-2018

Here is the detailed output, note the port is different this time.

Switch-4506#sho auth sess mac 2C:5A:0F:41:61:B3 det
            Interface:  GigabitEthernet4/39
          MAC Address:  2c5a.0f41.61b3
         IPv6 Address:  Unknown
         IPv4 Address:  Unknown
            User-Name:  2C-5A-0F-41-61-B3
               Status:  Authorized
               Domain:  DATA
       Oper host mode:  multi-auth
     Oper control dir:  both
      Session timeout:  N/A
      Restart timeout:  N/A
Periodic Acct timeout:  3300s (local), Remaining: 2749s
       Session Uptime:  572s
    Common Session ID:  0AB4FF0A0001678F502BA498
      Acct Session ID:  0x0001621E
               Handle:  0x880008AB
       Current Policy:  POLICY_Gi4/39

Local Policies:
        Service Template: DEFAULT_LINKSEC_POLICY_SHOULD_SECURE (priority 150)
      Security Policy:  Should Secure
      Security Status:  Link Unsecure

Server Policies:

Method status list: 
       Method           State 

       dot1x            Stopped
       mab              Authc Success

Good question on the phone, I just went back and looked and it looks like the phone was moved to a different port. But if it was moved in the time between those two commands I would be very surprised, just adds to the mystery. And the new port the phone is on is not the same as the new port the mystery MAC is on, so it is not following the phone.

I also noticed this in ISE:

paul · ‎08-17-2018

It almost looks like you have ports looped or something. That is very odd.

Cory Peterson · ‎08-17-2018

Very Odd... From what I have gathered so far I only see it happen on 4500 with Sup 6 or 7.

Damien Miller · ‎08-17-2018

When I saw your post I thought this seemed very familiar, so we discussed this briefly via email, but I figured I would jump in and share it here too. We are also seeing this the moment we add dot1x port config on two 4506's with sup6's running 15.2(2)E7, WS-X4648-RJ45V+E line cards.

Like you, the live logs also show the endpoints "beaming" around the switch. Authenticating multiple times on port x, moving, authenticating some more, moving again etc. We had the managed services team open a case with TAC because we cannot find a cause for it. Happened to us on both 4506/sup6 chassis we applied dot1x to. Lucky for us only four 4500/sup6 exist out of 10k switches so it is a relatively contained issue.

sho mac address-table int g 5/39
Unicast Entries
vlan mac address type protocols port
---------+---------------+--------+---------------------+-------------------------
1 001b.216f.49c4 dynamic ip,ipx,assigned,other GigabitEthernet5/39

Interface MAC Address Method Domain Status Fg Session ID
----------------------------------------------------------------------
Gi5/39 0004.f29a.19f0 mab VOICE Auth C7407716005950B130011852
Gi5/39 7cad.7450.382b N/A UNKNOWN Unauth C7407716005950C430017902
Gi5/39 a0e0.af34.002e mab DATA Auth C7407716005950B030010A6A
Gi5/39 001b.216f.49c4 mab DATA Auth C74077160056B54E204D6AD9
Gi5/39 bc16.6516.5660 mab DATA Auth C7407716005950BB30013EEA
Gi5/39 0004.f29a.3242 N/A UNKNOWN Unauth C7407716005950D33001D3B2
Gi5/39 0004.f29a.330d N/A UNKNOWN Unauth C7407716005950CC3001AD46
Gi5/39 7cad.74fb.f6e4 mab DATA Auth C74077160059508B30003395
Gi5/39 c89c.1dae.4de1 mab DATA Auth C7407716005950BE300149EA
Gi5/39 0004.f298.e536 mab VOICE Auth C7407716005950AB3000E6C6
Gi5/39 c89c.1dae.4de0 mab DATA Auth C7407716005950A43000BB5A
Gi5/39 3c08.f6db.db01 N/A UNKNOWN Unauth C7407716005950C530017DDA

hslai · ‎09-01-2018

This looks somewhat similar to an old issue seen on 3K running IOS 15.0(2)SE train, which related to "ip helper" configured on SVI.

Damien Miller · ‎09-01-2018

The MSP opened a tac case to troubleshoot. I have not heard back but I do know it is still an issue. The client has a handful of sup 6e's left in the environment and it doesn't seem to have created any end user impact, it has been a low priority for us.

Cory Peterson · ‎09-02-2018

The devices we are seeing this on does not have any SVI's with Helper addresses.

Also, I have not opened a case on this as it is not really affecting authentications, but it makes remediation a bit of a hassle as it is just more failing endpoints that we keep chasing for no reason.

hslai · ‎09-02-2018

Right. The 3K defect was on device-sensor and did not impact regular authentications, either. It might also help to check the MAC address pattern and see what type of interfaces it resembled on the switch.