Devices won't work until port is reset on C9300's after ISE reboot

[dustinn3]

For some reason on our new C9300's anytime we make a change in ISE that requires a node to reboot or just even updating our licenses, all of the devices that are using mac based auth stop working until the ports are reset.  The same devices work fine on our other Cisco switches using ISE. TAC added a re-auth policy that they said would fix it but it didn't.  I upgraded from 2.4 P14 to 2.7 P7 and still have the same issue.  Devices using dot1x continue to work just fine.  Most of these devices are Cisco 8945 and 8845 phones.

Here's our port configs on the 9300's and the policy map.

subscriber aging inactivity-timer 15 probe
switchport access vlan XXX
switchport mode access
switchport voice vlan XXX
device-tracking
logging event nfas-status
logging event trunk-status
logging event subif-link-status
load-interval 30
authentication periodic
authentication timer reauthenticate server
access-session port-control auto
mab
trust device cisco-phone
snmp trap mac-notification change added
snmp trap mac-notification change removed
dot1x pae authenticator
dot1x timeout tx-period 10
dot1x max-reauth-req 3
auto qos voip cisco-phone
source template closed_DOT1x-TEMPLATE
spanning-tree portfast
spanning-tree bpduguard enable
spanning-tree guard root
service-policy input AutoQos-4.0-CiscoPhone-Input-Policy
service-policy output AutoQos-4.0-Output-Policy

policy-map type control subscriber pmap_IDENTITY-CLOSED
event session-started match-all
10 class always do-until-failure
10 authenticate using dot1x retries 3 retry-time 1 priority 10
20 authenticate using mab retries 3 retry-time 1 priority 20
event authentication-failure match-first
10 class cmap_AAA_SRV_DOWN_UNAUTHD_HOST do-until-failure
10 clear-authenticated-data-hosts-on-port
20 activate service-template CRITICAL_AUTH_ACCESS
30 activate service-template DEFAULT_CRITICAL_VOICE_TEMPLATE
40 authorize
50 pause reauthentication
20 class cmap_AAA_SVR_DOWN_AUTHD_HOST do-until-failure
10 pause reauthentication
20 authorize
30 class cmap_DOT1X_FAILED do-until-failure
10 terminate dot1x
20 authentication-restart 30
40 class cmap_DOT1X_NO_RESP do-until-failure
10 terminate dot1x
20 authenticate using mab retries 3 retry-time 1 priority 20
50 class cmap_MAB_FAILED do-until-failure
10 terminate mab
20 authentication-restart 30
60 class always do-until-failure
10 terminate dot1x
20 terminate mab
30 authentication-restart 30
event agent-found match-all
10 class always do-until-failure
10 terminate mab
20 authenticate using dot1x priority 10
event aaa-available match-all
10 class cmap_IN_CRITICAL_AUTH do-until-failure
10 clear-session
20 class cmap_NOT_IN_CRITICAL_AUTH do-until-failure
10 resume reauthentication
event inactivity-timeout match-all
10 class always do-until-failure
5 clear-session
event authentication-success match-all
10 class always do-until-failure
10 activate service-template DEFAULT_LINKSEC_POLICY_SHOULD_SECURE

Thanks,

[Arne Bier]

Hello @dustinn3 

What version is the C9300 running?

What was the output of a "show access-session" on one of those affected C9300 ports?

How did you reset the ports? clear access-session or shut/no shut ?

Do your other non-C9300 switches have the same IBNS 2.0 config? And what about software version?

s112#show access-sess int gi 1/0/12 details
            Interface:  GigabitEthernet1/0/12
               IIF-ID:  0x12312E4D
          MAC Address:  885a.92d9.d0f7
         IPv6 Address:  Unknown
         IPv4 Address:  Unknown
            User-Name:  885a92d9d0f7
               Status:  Unauthorized
               Domain:  UNKNOWN
       Oper host mode:  multi-domain
     Oper control dir:  both
      Session timeout:  N/A
    Common Session ID:  420D020A00000412B04A68FE
      Acct Session ID:  Unknown
               Handle:  0x47000408
       Current Policy:  IDENTITY-POLICY


Server Policies:


Method status list:
       Method           State
        dot1x           Stopped
          mab           Authc Failed

s112#

I observed  a similar situation today on a C9300 running 17.6.2 and the device was also a Cisco phone using MAB auth.

The session seemed to be in a zombie state. No sessions in ISE. And no session countdown timer. Switch said that the MAB had failed AuthN but ISE shows exactly the opposite- all authN attempts resulted in success.

In my case we had a routing issue between switch and ISE (for roughly 4 hours). I think this killed it. 

I was able to restore the session with a "clear access-session int gig 1/0/12"

If this is not a bug, then perhaps someone can point me to the error in the config?

It feels to me that the switch does not re-attempt MAB .. which in my opinion it should if "MAB failed" (which it can't anyway because our ISE Policy never rejects MAB)

interface GigabitEthernet1/0/12
 description PHONE
 switchport access vlan xxx
 switchport mode access
 switchport nonegotiate
 switchport voice vlan yyy
 load-interval 30
 access-session host-mode multi-domain
 access-session closed
 access-session port-control auto
 mab
 dot1x pae authenticator
 dot1x timeout tx-period 10
 storm-control broadcast level 1.00
 storm-control multicast level 1.00
 no lldp transmit
 no lldp receive
 spanning-tree portfast
 spanning-tree bpduguard enable
 service-policy type control subscriber IDENTITY-POLICY
 service-policy input INGRESS-MARKING
 service-policy output EGRESS-MARKING
 ip dhcp snooping limit rate 15
end

s112#show policy-map type control subsc IDENTITY-POLICY
IDENTITY-POLICY
  event session-started match-all
    10 class always do-until-failure
     10 authenticate using dot1x priority 10
     20 authenticate using mab priority 20
  event authentication-failure match-first
    10 class AAA_SVR_DOWN_UNAUTHD_HOST do-until-failure
     10 activate service-template RESTRICTED_AUTH_VLAN
     20 activate service-template CRITICAL_VOICE_VLAN
     30 authorize
     40 pause reauthentication
    20 class AAA_SVR_DOWN_AUTHD_HOST do-until-failure
     10 pause reauthentication
     20 authorize
    30 class DOT1X_NO_RESP do-until-failure
     10 terminate dot1x
     20 authenticate using mab priority 20
    40 class MAB_FAILED do-until-failure
     10 terminate mab
     20 authentication-restart 60
    50 class DOT1X_FAILED do-until-failure
     10 terminate dot1x
     20 activate service-template RESTRICTED_AUTH_VLAN
     30 authorize
    60 class always do-until-failure
     10 terminate dot1x
     20 terminate mab
     30 authentication-restart 60
  event agent-found match-all
    10 class always do-until-failure
     10 terminate mab
     20 authenticate using dot1x priority 10
  event aaa-available match-all
    10 class IN_CRITICAL_AUTH do-until-failure
     10 clear-session
    20 class NOT_IN_CRITICAL_AUTH do-until-failure
     10 resume reauthentication
  event authentication-success match-all
    10 class always do-until-failure
     10 activate service-template IA-TIMER

[Arne Bier]

I was curious about this and did some testing

I used an extended ACL to block UDP/1812 and UDP/1813 between ISE nodes and my C9300.  I could confirm that the ACL was getting hits, and my "test aaa .." commands were failing, as was the endpoint on Gig1/0/12.

I was expecting the endpoint to end in the restricted VLAN as defined in IBNS policy (for unauth'd session) or to for forced Authorized in the case of an existing auth session,  But neither case happens. Perhaps there is a bug with the switch's interpretation of "aaa-timeout".

class-map type control subscriber match-all AAA_SVR_DOWN_AUTHD_HOST
 match result-type aaa-timeout
 match authorization-status authorized
!
class-map type control subscriber match-all AAA_SVR_DOWN_UNAUTHD_HOST
 match result-type aaa-timeout
 match authorization-status unauthorized

This also led me to look at the "show aaa servers" and I can't make any sense of it. The "UP" and "DEAD" status are not obvious and even when I used the automate-tester, which I was sure would detect ISE as down.

radius-server dead-criteria time 5 tries 2
radius-server deadtime 15
radius server nac1
 address ipv4 10.50.32.30 auth-port 1812 acct-port 1813
 automate-tester username testuser ignore-acct-port probe-on
 key ****
radius server nac2
 address ipv4 10.50.32.31 auth-port 1812 acct-port 1813
 automate-tester username testuser ignore-acct-port probe-on
 key *****

What am I to make of this? Is it dead or alive??

s112#show aaa servers

RADIUS: id 1, priority 1, host 10.50.32.30, auth-port 1812, acct-port 1813, nac1
     State: current UP, duration 714s, previous duration 808s
     Dead: total time 5158s, count 4
     Platform State from SMD: current DEAD, duration 595s, previous duration 253s
     SMD Platform Dead: total time 5805s, count 4
     Platform State from WNCD (1) : current UP
     Platform State from WNCD (2) : current UP
     Platform State from WNCD (3) : current UP
     Platform State from WNCD (4) : current UP
     Platform State from WNCD (5) : current UP
     Platform State from WNCD (6) : current UP
     Platform State from WNCD (7) : current UP
     Platform State from WNCD (8) : current UP, duration 0s, previous duration 0s
     Platform Dead: total time 0s, count 0UP

[Arne Bier]

Slight update to the issue above.  The problem was resolved with a config change on the switch. As it turned out, someone had accidentally removed the VLAN which was required when the AAA was down. So the switch was trying to apply a non existent VLAN and that's the problem. Once I added the VLAN back into the switch, all was well.

Now I am still wondering how to interpret the "show aaa servers" output.

From what I can tell, on the 9300 the critical part of the output is the line 

Platform State from SMD: current UP

or

Platform State from SMD: current DOWN

It's probably worth mentioning that even a successful "test aaa" command doesn't awaken the SMD into "UP" state. I can't tell which trigger causes the SMD to go "UP" again - it's either the probe (after deadtime expired) or it's the endpoint performing a re-auth.

[dustinn3]

Arne,

Thanks for all the testing you did.  

Looks like were running 16.12.04.  The config appears to be the same on our 4500's as what it is on the 9300's and I don't have the same issue.  

When I looked at ISE it appeared that the ports were still authorized, but on the port themselves it showed unauthorized similar to what you were seeing.  

I didn't attempt a clear access-session, but a shut no shut or unplugging the device and reconnecting worked.

I may setup another 9300 in the lab and see if I can reproduce without actually taking ISE down. 

[dustinn3]

I just setup another 9300 and had the same issue after applying the ACL.  If I apply it and wait for it to fail and then remove it while it's still retrying it will re-auth and come back up.  However if I wait a few minutes it never retries again.  

As soon as I ran the clear access-session command it re-authed and came back up.

[Arne Bier]

Hi @dustinn3 

I think I have an idea what the problem is. When the ACL is applied, have you checked the state of the "show aaa servers" command?

Notice the line with the "SMD" in it as I mentioned in an earlier post. That is the Session Manager Daemon - the processes that is responsible for handling the comms to RADIUS server when doing MAB/802.1X - if that is DOWN, then there will be no attempt to talk to that remote IP address (i.e. RADIUS server). The IOS will keep that IP "DEAD" for the time specified by the deadtime - e.g. 5 minutes

radius-server dead-criteria time 5 tries 2
radius-server deadtime 5

After the 5 minutes have elapsed, the IOS will try that IP address again.

A "dead" server is one where there have been two retries with a 5 second in-between. (the first access-accept is not counted - only RETRIES are counted).

I also found that if ALL the servers are dead, then the IOS will still wait for the deadtime to expire before it tries again. Therefore I make the deadtime 5 minutes. It's unlikely that both ISE nodes will be unreachable at the same time - but 5 minutes is a loooong time to have no ISE services! It would be nice to have an IOS feature that gets more aggressive in trying, when it has no servers alive. Then again, if the servers are still dead, then all the trying in the world won't help you. Best to use IBNS 2.0 and its contingency plans.

Have you tried the deadtime mechanism? And do you use automated-tester on the radius statements?

I also found that over time, the commands between IOS and even older IOS-XE varies a lot. It's best to adapt the commands to each release.

How often the automate tester is run, is probably up for debate - but perhaps in a busy network the RADIUS servers will be kept busy and the idle probes won't fire. Have a look in ISE to see how many probes you get and you can hide them from your Live Logs anyway with the Collection Filters (Admin > System > Logging > Collection Filters

radius server nac1
 address ipv4 10.50.32.30 auth-port 1812 acct-port 1813
 automate-tester username testuser idle-time 2
 key *****
radius server nac2
 address ipv4 10.50.32.31 auth-port 1812 acct-port 1813
 automate-tester username testuser idle-time 2
 key *****

If you have exhausted all options then perhaps try a newer IOS-XE version

[dustinn3]

Here's what I'm seeing on show aaa servers.

RADIUS: id 1, priority 1, host 10.1.0.67, auth-port 1812, acct-port 1813, hostname ISE-01-RADIUS
State: current DEAD, duration 8s, previous duration 919s
Dead: total time 848s, count 3
Platform State from SMD: current UP, duration 889s, previous duration 1500s

RADIUS: id 2, priority 2, host 10.1.0.68, auth-port 1812, acct-port 1813, hostname ISE-02-RADIUS
State: current DEAD, duration 135s, previous duration 878s
Dead: total time 795s, count 2
Platform State from SMD: current DEAD, duration 129s, previous duration 930s
SMD Platform Dead: total time 1429s, count 2

After a few minutes of removing the ACL they show they are alive again and I get but the port still doesn't attempt a re-auth. As soon as it get's this message it just stops trying again until you either cycle the port or reset authentication.  I can leave it for several days and it never tries again.

Mar 23 15:40:20.107 CDT: %SESSION_MGR-5-FAIL: Switch 1 R0/0: sessmgrd: Authorization failed or unapplied for client (1ce6.c79b.2723) on Interface FiveGigabitEthernet1/0/1 AuditSessionID 4E040A0A00000029B87EEA5D. Failure reason: Authc fail. Authc failure reason: AAA Server Down.

Mar 23 15:49:05.289 CDT: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.1.0.68:1812,1813 is not responding.
Mar 23 15:50:31.778 CDT: %RADIUS-4-RADIUS_DEAD: RADIUS server 10.1.0.67:1812,1813 is not responding.
Mar 23 16:00:05.650 CDT: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.1.0.68:1812,1813 is being marked alive.
Mar 23 16:01:32.181 CDT: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.1.0.67:1812,1813 is being marked alive.

Here's my configs.  

radius-server dead-criteria time 5 tries 3
radius-server retransmit 5
radius-server deadtime 3

radius server ISE-01-RADIUS
address ipv4 10.1.0.67 auth-port 1812 acct-port 1813
timeout 10
retransmit 3
automate-tester username radius.iS.alive ignore-acct-port idle-time 15

key *****

radius server ISE-02-RADIUS
address ipv4 10.1.0.68 auth-port 1812 acct-port 1813
timeout 10
retransmit 3
automate-tester username radius.iS.alive ignore-acct-port idle-time 15

key *****

I thought it might be version related as well so I upgraded to 17.03.04 with the same issue.  

I went ahead and opened another ticket with TAC.  I'll let you know what they figure out.

Thanks,

[Arne Bier · Accepted solution]

Good luck with the TAC case

For what it's worth, IOS-XE 17.6.2 seems to work well so far (i.e. with failed ISE servers my clients are put into their critical DATA and VOICE vlans.) - when I restore the access to ISE, I wait a few minutes and the service restores to normal. I don't set a session timeout in ISE for Voice devices - but I set a session timeout of 65535 seconds for data devices (which might hang off the back of a phone - and it's extra belts and braces to re-auth data devices periodically because some non-Cisco phones might not send the logoff to the switch when the piggy-backed device is disconnected)

Below is a redacted config that includes all the important stuff. I didn't include the VLAN details - be sure to always define all the VLANs that you have mentioned in your config below. Failure to define the VLAN itself will cause none of this to work

aaa new-model
!
!
aaa group server radius ISE
 server name nac1
 server name nac2
 deadtime 5
 retransmit 2
 timeout 5
!
aaa authentication dot1x default group ISE
aaa authorization network default group ISE
aaa accounting update newinfo periodic 2880
aaa accounting identity default start-stop group ISE
!
!
aaa session-id common
!
!
ip dhcp snooping vlan *** comma delimited list of VLANs to Snoop on *****
no ip dhcp snooping information option
ip dhcp snooping
!
!
!
epm logging
access-session attributes filter-list list FILTER_DS
 cdp
 lldp
 dhcp
access-session accounting attributes filter-spec include list FILTER_DS
device-tracking policy IPDT_POLICY
 security-level glean
 no protocol ndp
 no protocol udp
 tracking enable reachable-lifetime 10
!
service-template DEFAULT_LINKSEC_POLICY_MUST_SECURE
 linksec policy must-secure
service-template DEFAULT_LINKSEC_POLICY_SHOULD_SECURE
 linksec policy should-secure
service-template DEFAULT_CRITICAL_VOICE_TEMPLATE
 voice vlan
service-template DEFAULT_CRITICAL_DATA_TEMPLATE
service-template CRITICAL_VOICE_VLAN
 description ** Apply voice vlan on AAA Fail **
 voice vlan
service-template CRITICAL_AUTH_VLAN
 description ** Apply data vlan on AAA Fail **
 vlan ***critical_VLAN****
service-template RESTRICTED_AUTH_VLAN
 description ** Apply RESTRICTED vlan on AAA Fail **
 vlan **** restricted_VLAN****
service-template IA-TIMER
 description ** Apply inactivity timer and ARP probe **
 inactivity-timer 60 probe
dot1x system-auth-control
!
class-map type control subscriber match-all AAA_SVR_DOWN_AUTHD_HOST
 match result-type aaa-timeout
 match authorization-status authorized
!
class-map type control subscriber match-all AAA_SVR_DOWN_UNAUTHD_HOST
 match result-type aaa-timeout
 match authorization-status unauthorized
!
class-map type control subscriber match-all DOT1X
 match method dot1x
!
class-map type control subscriber match-all DOT1X_FAILED
 match method dot1x
 match result-type method dot1x authoritative
!
class-map type control subscriber match-all DOT1X_MEDIUM_PRIO
 match authorizing-method-priority gt 20
!
class-map type control subscriber match-all DOT1X_NO_RESP
 match method dot1x
 match result-type method dot1x agent-not-found
!
class-map type control subscriber match-all DOT1X_TIMEOUT
 match method dot1x
 match result-type method dot1x method-timeout
!
class-map type control subscriber match-any IN_CRITICAL_AUTH
 match activated-service-template RESTRICTED_AUTH_VLAN
 match activated-service-template CRITICAL_VOICE_VLAN
!
class-map type control subscriber match-all MAB
 match method mab
!
class-map type control subscriber match-all MAB_FAILED
 match method mab
 match result-type method mab authoritative
!
class-map type control subscriber match-none NOT_IN_CRITICAL_AUTH
 match activated-service-template RESTRICTED_AUTH_VLAN
 match activated-service-template CRITICAL_VOICE_VLAN
!
!
policy-map type control subscriber IDENTITY-POLICY
 event session-started match-all
  10 class always do-until-failure
   10 authenticate using dot1x priority 10
   20 authenticate using mab priority 20
 event authentication-failure match-first
  10 class AAA_SVR_DOWN_UNAUTHD_HOST do-until-failure
   10 activate service-template RESTRICTED_AUTH_VLAN
   20 activate service-template CRITICAL_VOICE_VLAN
   30 authorize
   40 pause reauthentication
  20 class AAA_SVR_DOWN_AUTHD_HOST do-until-failure
   10 pause reauthentication
   20 authorize
  30 class DOT1X_NO_RESP do-until-failure
   10 terminate dot1x
   20 authenticate using mab priority 20
  40 class MAB_FAILED do-until-failure
   10 terminate mab
   20 authentication-restart 60
  50 class DOT1X_FAILED do-until-failure
   10 terminate dot1x
   20 activate service-template RESTRICTED_AUTH_VLAN
   30 authorize
  60 class always do-until-failure
   10 terminate dot1x
   20 terminate mab
   30 authentication-restart 60
 event agent-found match-all
  10 class always do-until-failure
   10 terminate mab
   20 authenticate using dot1x priority 10
 event aaa-available match-all
  10 class IN_CRITICAL_AUTH do-until-failure
   10 clear-session
  20 class NOT_IN_CRITICAL_AUTH do-until-failure
   10 resume reauthentication
 event authentication-success match-all
  10 class always do-until-failure
   10 activate service-template IA-TIMER
!
!
template 802.1X
 dot1x pae authenticator
 storm-control broadcast level 1.00
 storm-control multicast level 1.00
 spanning-tree portfast
 spanning-tree bpduguard enable
 switchport access vlan ****restricted_VLAN****
 switchport mode access
 switchport nonegotiate
 trust device cisco-phone
 mab
 access-session host-mode multi-domain
 access-session closed
 access-session port-control auto
 authentication periodic
 authentication timer reauthenticate server
 service-policy type control subscriber IDENTITY-POLICY
 description UserAccess 802.1X
 ip dhcp snooping limit rate 15
!

interface GigabitEthernet1/0/12
 description NAC Controlled Port
 switchport mode access
 switchport voice vlan ***voice_VLAN***
 device-tracking attach-policy IPDT_POLICY
 load-interval 30
 dot1x timeout tx-period 10
 no lldp transmit
 no lldp receive
 source template 802.1X
 spanning-tree portfast
!
ip radius source-interface ****vlan/interface****
radius-server attribute 6 on-for-login-auth
radius-server attribute 6 support-multiple
radius-server attribute 8 include-in-access-req
radius-server attribute 25 access-request include
radius-server dead-criteria time 5 tries 2
radius-server deadtime 5
!
radius server nac1
 address ipv4 10.50.32.30 auth-port 1812 acct-port 1813
 automate-tester username testuser idle-time 2
 key 0 ************
!
radius server nac2
 address ipv4 10.50.32.31 auth-port 1812 acct-port 1813
 automate-tester username testuser idle-time 2
 key 0 *************
!
no access-session mac-move deny
mac address-table notification change

[dustinn3]

Arne,

Thanks for your config.  I loaded it on my switch and can confirm it worked on my switch as well so I went back through the config line by line and found the issue. We had a VAR setup the templates and it appears they made an error in the policy-map and were referencing a non-existent service template in the class AAA_SVR_DOWN_UNAUTHD_HOST do-until-failure. Once I changed the name to the correct template it started working.  Looks like I've still got a bit of work to do though because it's clearing the authentication about every 30 seconds which takes it down for a ping or two. Thanks again for all your help.