AP stop to authenticate by dot1x into LAN

[Mirek_Tichy]

C9800-40 17.12.x, C91xx local mode, EAP-FAST authentication to C9300 with a MAB fallback.

After some weeks or months some APs failed to authenticate themself to dot1x secured LAN. Typicaly after an upgrade some APs are lost. ISE reports:

• Event 5440 Endpoint abandoned EAP session and started new
• Failure Reason 22063 Wrong password

We have a fallback configuration so all that APs will pass by MAB and are working well. That means they give a new config from the WLC during the Join. But APs stay in the wrong password state.

AP#sho ap authentication status
key_mgmt=IEEE 802.1X (no WPA)
wpa_state=ASSOCIATED
address=ac:4a:56:00:00:00
Supplicant PAE state=HELD
suppPortStatus=Unauthorized
EAP state=FAILURE
selectedMethod=43 (EAP-FAST)
eap_tls_version=TLSv1.2
EAP TLS cipher=ADH-AES128-SHA
tls_session_reused=0
EAP-FAST Phase2 method=MSCHAPV2

"clear authentication session" on the switchport or "capwap ap reset" on the AP don't solve the problem.

The only solution is a full reload of the AP. After that each AP authenticate itself correctly and re-authenticate continuously for an unknown time period.

I think it is a task for the TAC but I am trying to ask here first.

[Leo Laohoo]

Ever since we've upgraded to 17.12.4 and 17.12.5, we have seen a lot of our APs doing this.  

Another behaviour we've seen are the APs losing their IP addresses.  I have a TAC case for this (APs losing IP address) and I have been demanding they publish an APSP to fix CSCwp20385.

Our APs are 3700, 2800/3800/4800/1560, 9124, 9130, 9136 and 9166.  

[MHM Cisco World]

We are talking about AP authc not wifi client authc ?

MHM

[Leo Laohoo]

---

@MHM Cisco World wrote:
We are talking about AP authc not wifi client authc ?

---

Check out CSCwp20385.  To put it simply, AP is receiving traffic but blackholing all outbound traffic.  This includes DHCP DORA and 802.1x-related traffic.  

[MHM Cisco World]

Sorry but 

• Failure Reason 22063 Wrong password <<- this meaning ISE is received some packet

MHM

[Leo Laohoo]

---

@MHM Cisco World wrote:
this meaning ISE is received some packet

---

ISE received "corrupt" or incomplete packets and deemed it as "wrong password".  

[Mirek_Tichy]

Hi Leo, thanks, it may relate but unfortunately there are allmost no info in the bug.

[Leo Laohoo · Accepted solution]

Raise a TAC Case.  

Use my TAC Case #699156079 as a reference.

[Mirek_Tichy]

Exactly we have no problem with cients. We are talking about dot1x secured LAN ports connecting APs.

[MHM Cisco World]

How you add username/password to AP ?

I think this need WLC 

So issue between WLC and ISE not between AP and ISE

MHM

[Mirek_Tichy]

Hi, The creds are in the Site TAGs on the WLC. The WLC serves about 1000 AP but 400 of them got wrong and ISE reports bad password. Rejoin and reboot AP solves the problem with no config at the WLC.

[MHM Cisco World]

I think you need only rejoin no need reboot.

In end ISE will only authc device after re-auth time is end' so from view of WLC/ISE the AP still authc 

Rejoin will clear authc session of AP and force AP to re-auth

MHM

[MHM Cisco World]

Any update from TAC team 

MHM

[Leo Laohoo]

TAC wants us to capture packet capture from the AP.  As I've told TAC and the developers, our APs are installed at 2.6 metres above ground.  Having someone do packet captures, from a ladder, is a dangerous task.  

So we are waiting for an opportunity to find an AP which is safe for us to do.  

In the meantime, the developers are refusing to hand over the APSP to fix this bug.  

[MHM Cisco World]

Sorry' you have exact same issue ?

AP is authc as supplicant?

MHM