topic Re: Catalyst switches: Endless access-session for failed MAC addresses in Network Access Control

Catalyst switches: Endless access-session for failed MAC addresses

Johannes Luther — Thu, 01 Dec 2022 11:31:27 GMT

Hi community,

I'm using Catalyst switches and want to perform open mode (NAC monitor) mode using IBNS 2.0 configuration.
The RADIUS server is a Cisco ISE.

Switch ports: The ports are configured in open mode in the host-mode multi-auth.
ISE: The default rule at the end of the policy is a DENY. Unknown MACs should be denied.
I know there is also the strategy to deploy a PERMIT rule with a corresponding DACL - but I don't like this approach.

I'm not using any kind of Profiler ... knows MACs are permitted - unknown are not ... simple as that.

I'm running some lab tests and want to test the monitor mode functionality. This means, that multiple MAC addresses may be allowed behind a switch port. So I have a legitimate MAC / Client behind my switch port and simulate other clients by MAC spoofing (scapy snippet). I want to prove, that the port is not going into error disable and the MACs are learned on the port.

So far so good... when I launch my MAC spoofing the following happens:
- MAC addresses are learned on the port (show mac address-table int <ID>)
- Access-session for MAC address is triggered and after some time authentication fails (MAB immediately, 802.1X after 90 seconds) for the MAC address

=> Still, so far so good...

After the session fails, the authentication is restarted after 60 seconds. Here's a shortened config snipped of the policy:

policy-map type control subscriber MY_POLICY
[...]
  event authentication-failure match-first
  [...]
    40 class MAB_FAILED do-until-failure
      10 terminate mab
      20 authentication-restart 60
    60 class always do-until-failure
      10 terminate dot1x
      20 terminate mab
      30 authentication-restart 60

So for a given failed MAC the auth is retried and retried again and again ....

However after some time (300 seconds), the MAC address of the failed client is removed from the MAC address table. However, the access session is restarted and restarted again for this MAC, although it's no longer in the MAC address table.

So the MAC address is tried to reauthenticate via MAB forever ... or at least until the port is bounced or the access session is cleared.

How to overcome this? Is there any config knob? I know, that I could enable a PERMIT Rule at the end of the ISE authz rules to solve it in combination with the "idle-timeout" ... however there must be another way to remove the access sessions for not existing MACs....

Re: Catalyst switches: Endless access-session for failed MAC addresses

Arne Bier — Mon, 05 Dec 2022 21:43:40 GMT

Do you have Device-Tracking enabled? If yes, then perhaps the symptoms you're seeing are a software defect (in my opinion).

In a healthy system I have never seen a switch create a session when there is no MAC address on the interface (show mac address interface xxx) - the presence of the MAC address is the trigger for the session manager (SMD) to do some processing.

I have also tried in the past, to debug Catalysts 802.1X/MAB but the logs can be hard to interpret. It might be worth giving this a try

My cheat sheet below

IOS-XE split out the 802.1X Session Management into a separate Linux process called 'smd' (Session Manager Daemon)

Forget everything you have learned in last 25 years of IOS debugging - it no longer works (the classic debug commands are still there, but they are defunct)

Set the debugs

===================

set platform software trace smd switch active R0 dot1x-all debug

set platform software trace smd switch active R0 radius-authen debug

set platform software trace smd switch active R0 aaa-authen debug

set platform software trace smd switch active R0 eap-all debug

set platform software trace smd switch active R0 auth-mgr-all debug

View the trace levels

=========================

show platform software trace level smd switch active R0

View the logs with

==============================

show platform software trace message smd switch active R0

After test complete, reset the debugs to normal again!!!

===========

set platform software trace smd switch active R0 dot1x-all notice

set platform software trace smd switch active R0 radius-authen notice

set platform software trace smd switch active R0 aaa-authen notice

set platform software trace smd switch active R0 eap-all notice

set platform software trace smd switch active R0 auth-mgr-all notice

Re: Catalyst switches: Endless access-session for failed MAC addresses

Johannes Luther — Tue, 06 Dec 2022 06:03:01 GMT

Hi Arne,
thank you for the reply. So it's the same behavior in IOS 15.2(7) (2960-X) and IOS-XE 17.6 (9300)

@Arne Bier wrote:

Do you have Device-Tracking enabled? If yes, then perhaps the symptoms you're seeing are a software defect (in my opinion).

Yes Device Tracking is enabled on both platforms and works like expected. However for these MAC addresses, the IP addresses are not learned, because I just send arbitary traffic from them, which is not ARP or ND. Device tracking won't learn any IP for the MAC from the data plane. But this should not influence the behavior from my point of view.

@Arne Bier wrote:

In a healthy system I have never seen a switch create a session when there is no MAC address on the interface (show mac address interface xxx) - the presence of the MAC address is the trigger for the session manager (SMD) to do some processing.

And this is still the case. So the access session is created as soon as the MAC is learned on the switch port.
However, the access session for a given MAC is not cleared when the MAC address is removed (MAC timeout).

Re: Catalyst switches: Endless access-session for failed MAC addresses

andrewswanson — Tue, 06 Dec 2022 16:51:18 GMT

Could the following thread help with your issue?

https://community.cisco.com/t5/switching/idle-timeout-and-probing-for-access-session-ibns2-0/td-p/3693787

Create a service template with inactivity-timer set
When a client fails authentication, this service-template is applied to the session
when the inactivity-timer expires, the session is cleared

hth
Andy

service-template INACTIVITY-TIMER
inactivity-timer 60 probe
......
event authentication-failure match-all
10 class always do-until-failure
10 activate service-template INACTIVITY-TIMER
event inactivity-timeout match-all
10 class always do-until-failure
10 clear-session

Re: Catalyst switches: Endless access-session for failed MAC addresses

Johannes Luther — Wed, 07 Dec 2022 08:34:08 GMT

Hi @andrewswanson
thank you very much! Very, very helpful. Tried it in my lab and it's working with little modifications.
I didn't use the "probe" keyword in the "inactivity-timer", because there might be endpoints, which are not in den device-tracking database, because their IP has not been learned through device tracking compatible protocols (like ARP).

However, the keyword "inactivity" implies, that the event is triggered when the endpoint (MAC) does not send frames. Unfortunately this is incorrect. The inactivity timer is reset as soon as a new event for the endpoint is triggered, regardless whether the endpoint is active.

Example

policy-map type control subscriber MY_POLICY
[...]
  event authentication-failure match-first
  [...]
    40 class MAB_FAILED do-until-failure
      10 terminate mab
      15 activate service-template INACTIVITY-TIMER
      20 authentication-restart 60
    60 class always do-until-failure
      10 terminate dot1x
      20 terminate mab
      25 activate service-template INACTIVITY-TIMER
      30 authentication-restart 60

In the example above, the inactivity-timer must be less than 60 seconds, because this is the authentication restart time.
As soon as the authentication restarts, the inactivity timer is reset. If the inactivity timer is reset the newly introduced event "inactivity-timeout" never triggers....

I thought about the whole topic and played around with the absolute timeout as well.... However I wasn't happy with the outcome.
Why even restart after a failed MAB attempt? The endpoint sends frames if it want to communicate. So I don't need to restart anything - I just need a way to backpressure failed MAB attempts for some time, right?

One way would be to tweak (somehow) the policy or using EEM:
So the EEM approach would be

event manager applet REMOVE_AUTH_MANAGER_SESSION
 event mat interface regexp .* type delete hold-down 1 ratelimit 5
 action 1.0 syslog msg "MAC address deleted: $_mat_mac_address (interface: $_mat_intf_name). Try to clear access-session"
 action 2.0 cli command "clear access-session mac $_mat_mac_address"

Re: Catalyst switches: Endless access-session for failed MAC addresses

Johannes Luther — Wed, 07 Dec 2022 09:28:49 GMT

Nevermind the EEM applet ... it's kinda useless, because it's triggered on MAC changes from dynamic to static as well... Clearing freshly authorized sessions, resulting in an endless auth loop.

😞Could be solved with some clever conditions within the EEM script ... but the more I think of it, I find it kind of "dirty"

Re: Catalyst switches: Endless access-session for failed MAC addresses

martin.fischer — Wed, 07 Dec 2022 09:56:39 GMT

Hi, set the command

authentication timer unauthorized 300

under the interface. This removes sessions which were in the unauthorized state for 300 seconds (or whatever time you like).

Re: Catalyst switches: Endless access-session for failed MAC addresses

andrewswanson — Wed, 07 Dec 2022 10:33:24 GMT

I think "authentication timer unauthorized 300" is a legacy (ibns 1) command. It would interesting to see the result of configuring an interface with this command (in ibns 1 mode) and checking if/how "authentication display new-style" converts this to ibns 2.

Re: Catalyst switches: Endless access-session for failed MAC addresses

andrewswanson — Wed, 07 Dec 2022 10:52:38 GMT

Just checked this in CML - the interface retains the "authentication timer unauthorized 300" command after conversion to ibns 2 so this may be an option worth looking at?

Re: Catalyst switches: Endless access-session for failed MAC addresses

martin.fischer — Wed, 07 Dec 2022 11:59:05 GMT

Yes, it's an IBNS 2.0 compatible command like most of the other authentication timer xyz commands.

Re: Catalyst switches: Endless access-session for failed MAC addresses

Johannes Luther — Thu, 08 Dec 2022 12:01:21 GMT

switch(config-if)#authentication timer inactivity 300
Command deprecated(authentication timer inactivity 300 ) - use cpl config

Does not work with IBNS 2.0 on my 2960X. And (even if it would work), the command is not compatible with interface templates.

Re: Catalyst switches: Endless access-session for failed MAC addresses

martin.fischer — Thu, 08 Dec 2022 13:24:48 GMT

Hi @Johannes Luther
The command you entered is not the same as I suggested (you used timer inactivity instead of unauthorized).

Re: Catalyst switches: Endless access-session for failed MAC addresses

Johannes Luther — Thu, 08 Dec 2022 14:23:38 GMT

Silly me ... command is working on the interface level (not template).

And... it's working ... crazy...

Now it would be good to have something similar in the IBNS 2.0 policy ..

Re: Catalyst switches: Endless access-session for failed MAC addresses

andrewswanson — Thu, 08 Dec 2022 17:14:09 GMT

Perhaps another option would be for ISE to send the RADIUS Idle-timeout attribute for failed MAB authentications? I assume ISE currently responds only with Access-Reject.

If that works, it would shift policy decisions onto ISE rather than the switch ibns 2 policy.

hth
Andy

EDIT - just read that this attribute is only sent in an Access-Accept or Access-Challenge so wouldn't work in this scenario