Catalyst switches: Endless access-session for failed MAC addresses

[Johannes Luther]

Hi community,

I'm using Catalyst switches and want to perform open mode (NAC monitor) mode using IBNS 2.0 configuration.
The RADIUS server is a Cisco ISE.

Switch ports: The ports are configured in open mode in the host-mode multi-auth.
ISE: The default rule at the end of the policy is a DENY. Unknown MACs should be denied.
I know there is also the strategy to deploy a PERMIT rule with a corresponding DACL - but I don't like this approach.

I'm not using any kind of Profiler ... knows MACs are permitted - unknown are not ... simple as that.

I'm running some lab tests and want to test the monitor mode functionality. This means, that multiple MAC addresses may be allowed behind a switch port. So I have a legitimate MAC / Client behind my switch port and simulate other clients by MAC spoofing (scapy snippet). I want to prove, that the port is not going into error disable and the MACs are learned on the port.

So far so good... when I launch my MAC spoofing the following happens:
- MAC addresses are learned on the port (show mac address-table int <ID>)
- Access-session for MAC address is triggered and after some time authentication fails (MAB immediately, 802.1X after 90 seconds) for the MAC address

=> Still, so far so good...

After the session fails, the authentication is restarted after 60 seconds. Here's a shortened config snipped of the policy:

policy-map type control subscriber MY_POLICY
[...]
  event authentication-failure match-first
  [...]
    40 class MAB_FAILED do-until-failure
      10 terminate mab
      20 authentication-restart 60
    60 class always do-until-failure
      10 terminate dot1x
      20 terminate mab
      30 authentication-restart 60

So for a given failed MAC the auth is retried and retried again and again ....

However after some time (300 seconds), the MAC address of the failed client is removed from the MAC address table. However, the access session is restarted and restarted again for this MAC, although it's no longer in the MAC address table.

So the MAC address is tried to reauthenticate via MAB forever ... or at least until the port is bounced or the access session is cleared.

How to overcome this? Is there any config knob? I know, that I could enable a PERMIT Rule at the end of the ISE authz rules to solve it in combination with the "idle-timeout" ... however there must be another way to remove the access sessions for not existing MACs....

[Arne Bier]

Do you have Device-Tracking enabled?  If yes, then perhaps the symptoms you're seeing are a software defect (in my opinion). 

In a healthy system I have never seen a switch create a session when there is no MAC address on the interface (show mac address interface xxx) - the presence of the MAC address is the trigger for the session manager (SMD) to do some processing.

I have also tried in the past, to debug Catalysts 802.1X/MAB but the logs can be hard to interpret. It might be worth giving this a try

My cheat sheet below

IOS-XE split out the 802.1X Session Management into a separate Linux process called 'smd' (Session Manager Daemon)

Forget everything you have learned in last 25 years of IOS debugging - it no longer works (the classic debug commands are still there, but they are defunct)

Set the debugs

===================

set platform software trace smd switch active R0 dot1x-all debug

set platform software trace smd switch active R0 radius-authen debug

set platform software trace smd switch active R0 aaa-authen debug

set platform software trace smd switch active R0 eap-all debug

set platform software trace smd switch active R0 auth-mgr-all debug

View the trace levels

=========================

show platform software trace level smd switch active R0

View the logs with

==============================

show platform software trace message smd switch active R0

After test complete, reset the debugs to normal again!!!

===========  

set platform software trace smd switch active R0 dot1x-all notice

set platform software trace smd switch active R0 radius-authen notice

set platform software trace smd switch active R0 aaa-authen notice

set platform software trace smd switch active R0 eap-all notice

set platform software trace smd switch active R0 auth-mgr-all notice

[Johannes Luther]

Hi Arne,
thank you for the reply. So it's the same behavior in IOS 15.2(7) (2960-X) and IOS-XE 17.6 (9300)

---

@Arne Bier wrote:

Do you have Device-Tracking enabled?  If yes, then perhaps the symptoms you're seeing are a software defect (in my opinion). 

Yes Device Tracking is enabled on both platforms and works like expected. However for these MAC addresses, the IP addresses are not learned, because I just send arbitary traffic from them, which is not ARP or ND. Device tracking won't learn any IP for the MAC from the data plane. But this should not influence the behavior from my point of view.

---

@Arne Bier wrote:

In a healthy system I have never seen a switch create a session when there is no MAC address on the interface (show mac address interface xxx) - the presence of the MAC address is the trigger for the session manager (SMD) to do some processing.

---

And this is still the case. So the access session is created as soon as the MAC is learned on the switch port.
However, the access session for a given MAC is not cleared when the MAC address is removed (MAC timeout).

[andrewswanson]

Could the following thread help with your issue?

https://community.cisco.com/t5/switching/idle-timeout-and-probing-for-access-session-ibns2-0/td-p/3693787

• Create a service template with inactivity-timer set
• When a client fails authentication, this service-template is applied to the session
• when the inactivity-timer expires, the session is cleared

hth
Andy

service-template INACTIVITY-TIMER
inactivity-timer 60 probe
......
event authentication-failure match-all
10 class always do-until-failure
10 activate service-template INACTIVITY-TIMER
event inactivity-timeout match-all
10 class always do-until-failure
10 clear-session

[Johannes Luther]

Hi @andrewswanson 
thank you very much! Very, very helpful. Tried it in my lab and it's working with little modifications.
I didn't use the "probe" keyword in the "inactivity-timer", because there might be endpoints, which are not in den device-tracking database, because their IP has not been learned through device tracking compatible protocols (like ARP).

However, the keyword "inactivity" implies, that the event is triggered when the endpoint (MAC) does not send frames. Unfortunately this is incorrect. The inactivity timer is reset as soon as a new event for the endpoint is triggered, regardless whether the endpoint is active.

Example

policy-map type control subscriber MY_POLICY
[...]
  event authentication-failure match-first
  [...]
    40 class MAB_FAILED do-until-failure
      10 terminate mab
      15 activate service-template INACTIVITY-TIMER
      20 authentication-restart 60
    60 class always do-until-failure
      10 terminate dot1x
      20 terminate mab
      25 activate service-template INACTIVITY-TIMER
      30 authentication-restart 60

In the example above, the inactivity-timer must be less than 60 seconds, because this is the authentication restart time.
As soon as the authentication restarts, the inactivity timer is reset. If the inactivity timer is reset the newly introduced event "inactivity-timeout" never triggers....

I thought about the whole topic and played around with the absolute timeout as well.... However I wasn't happy with the outcome.
Why even restart after a failed MAB attempt? The endpoint sends frames if it want to communicate. So I don't need to restart anything - I just need a way to backpressure failed MAB attempts for some time, right?

One way would be to tweak (somehow) the policy or using EEM:
So the EEM approach would be

event manager applet REMOVE_AUTH_MANAGER_SESSION
 event mat interface regexp .* type delete hold-down 1 ratelimit 5
 action 1.0 syslog msg "MAC address deleted: $_mat_mac_address (interface: $_mat_intf_name). Try to clear access-session"
 action 2.0 cli command "clear access-session mac $_mat_mac_address"

[Johannes Luther]

Nevermind the EEM applet ... it's kinda useless, because it's triggered on MAC changes from dynamic to static as well... Clearing freshly authorized sessions, resulting in an endless auth loop.

Could be solved with some clever conditions within the EEM script ... but the more I think of it, I find it kind of "dirty"

[martin.fischer]

Hi, set the command

 authentication timer unauthorized 300

under the interface. This removes sessions which were in the unauthorized state for 300 seconds (or whatever time you like).

[andrewswanson]

I think "authentication timer unauthorized 300" is a legacy (ibns 1) command. It would interesting to see the result of configuring an interface with this command (in ibns 1 mode) and checking if/how "authentication display new-style" converts this to ibns 2.

[andrewswanson]

Just checked this in CML - the interface retains the "authentication timer unauthorized 300" command after conversion to ibns 2 so this may be an option worth looking at?

[martin.fischer]

Yes, it's an IBNS 2.0 compatible command like most of the other authentication timer xyz commands.

[Johannes Luther]

switch(config-if)#authentication timer inactivity 300
Command deprecated(authentication timer inactivity 300 ) - use cpl config

Does not work with IBNS 2.0 on my 2960X. And (even if it would work), the command is not compatible with interface templates.

[martin.fischer · Accepted solution]

Hi @Johannes Luther 
The command you entered is not the same as I suggested (you used timer inactivity instead of unauthorized).

[Johannes Luther]

Silly me ... command is working on the interface level (not template).

And... it's working ... crazy...

Now it would be good to have something similar in the IBNS 2.0 policy ..

[andrewswanson]

Perhaps another option would be for ISE to send the RADIUS Idle-timeout attribute for failed MAB authentications? I assume ISE currently responds only with Access-Reject.

If that works, it would shift policy decisions onto ISE rather than the switch ibns 2 policy.

hth
Andy

EDIT - just read that this attribute is only sent in an Access-Accept or Access-Challenge so wouldn't work in this scenario