AP issue

[frederick.mercado]

Hello we have a local 9120axi AP with a 9800LC WLC that seems to be losing connection. The AP is in flex connect, all config looks fine: 

Advanced

Country Code

US  

Multiple Countries

US

Statistics Timer

180

CAPWAP MTU

1485

AP Link Latency

Enabled

AP PMK Propagation Capability

Enabled

TCP Adjust MSS Option

AP TCP MSS Adjust

Enabled

AP TCP MSS Size

1250

AP IPv6 TCP MSS Adjust

Enabled

AP IPv6 TCP MSS Size

1250

AP Retransmit Config Parameters

AP Retransmit Count

8

AP Retransmit Interval

5

Version

Primary Software Version

17.9.3.50

Predownloaded Status

N/A

Predownloaded Version

N/A

Next Retry Time

N/A

Boot Version

1.1.2.4

IOS Version

17.9.3.50

Mini IOS Version

0.0.0.0

Time Statistics

Up Time

191 days 23 hrs 8 mins 37 secs

Controller Association Latency

25 secs

Is the AP currently connected to controller

JOINED

Time at which the AP joined this controller last time

04/22/2024 11:35:11

Type of error that occurred last

Run

Time at which the last join error occurred

04/22/2024 11:34:21

Last AP Disconnect Details

Reason for last AP connection failure

Max Retransmission to AP

Last Reboot Reason (Reported by AP)

No reboot reason

Control DTLS Statistics

DTLS Session request received

2983

Established DTLS session

2977

Unsuccessful DTLS session

6

Reason for last unsuccessful DTLS session

DTLS Handshake Certificate Authentication Failed

Time at last successful DTLS session

04/22/2024 11:35:07

Time at last unsuccessful DTLS session

03/22/2024 20:11:21

Join phase statistics

Join requests received

2979

Successful join responses sent

2977

Unsuccessful join request processing

2

Reason for last unsuccessful join attempt

NA

Time at last successful join attempt

04/22/2024 11:35:11

Time at last unsuccessful join attempt

02/29/2024 00:35:29

Configuration phase statistics

Configuration requests received

8931

Successful configuration responses sent

8931

Unsuccessful configuration request processing

0

Reason for last unsuccessful configuration attempt

NA

Time at last successful configuration attempt

04/22/2024 11:35:12

Time at last unsuccessful configuration attempt

NA

Data DTLS Statistics

DTLS Session request received

0

Established DTLS session

0

Unsuccessful DTLS session

0

Reason for last unsuccessful DTLS session

DTLS Handshake Success

Time at last successful DTLS session

NA

Time at last unsuccessful DTLS session

NA

No ICMP drops from concurrent ping. No other reported issues on the 9300 Switch or other APs. There were some loss of connectivity for users at 0834 hrs and 1051 hrs. 

Closest thing I ran into was this error in the radioactive tace: "AP disconnect initiated. Reason: Max Retransmission to AP, Phase: Run" 

[frederick.mercado]

Cosmetic - but looks at NTP...tried reloading AP - will do a full power on/off as you indicated on port now.

[frederick.mercado]

Tried clearing the config and reset to factory defaults trough WLC...still get these errors:

Apr 24 16:33:21 ntpd[10853]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

Apr 24 16:27:43 capwapd[5995]: failed writing /click/pmk_tracker/add_cache_param_via_tlv len 192 data: "MAC 00:00:00:00:00:00, ACL \"\", CACHE_..."
Apr 24 16:27:43 kernel: [*04/24/2024 12:27:43.5026] In write handler 'add_cache_param_via_tlv' for 'pmk_tracker :: PMKTracker':
Apr 24 16:27:43 kernel: [*04/24/2024 12:27:43.5026] not a valid cache_type in add_cache_param_via_tlv
Apr 24 16:27:43 kernel: [*04/24/2024 12:27:43.5026]
Apr 24 16:27:43 kernel: [*04/24/2024 12:27:43.5026] !!!!! {/opt/cisco/bin/capwapd} failed writing /click/pmk_tracker/add_cache_param_via_tlv len 192 data: "MAC 00:00:00:00:00:00, ACL \"\", CACHE_..."

Apr 24 16:27:42 kernel: [*04/24/2024 12:27:42.8416] *** Unable to connect to: 127.0.0.1:4040 - [Errno 111] Connection refused

Apr 24 16:27:27 ntp_update: NTP: Wed Apr 24 16:27:27 2024 :Can not create ntp process log file.

Apr 24 16:27:27 ntpd[8355]: ntpd exiting on signal 15 (Terminated)

Apr 24 12:37:44 sshd[15551]: error: Could not get shadow information for admin

[marce1000]

 - Have a checkup of the 9800LC WLC configuration with the CLI command show tech wireless and feed the output to : Wireless Config Analyzer
                                This is fundamental

   Further look at and or 'execute' : https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/218396-troubleshoot-catalyst-9800-ap-join-or-di.html
  You can also trouble shoot with commands from : https://logadvisor.cisco.com/logadvisor/wireless/9800/9800APJoin

  And use these commands to check if you are making progress : https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc4

 M.

[frederick.mercado]

Checked config - nothing out of the ordinary. 

I noticed quite a few bugs in the 17.9.3 version - so we will schedule a upgrade this evening to 9.5.

[Leo Laohoo]

@frederick.mercado 

If the controller is on 17.9.4/17.9.4a or 17.9.5, I would strong recommend downgrading to 17.6.6 or upgrading to 17.12.3.  My reasons can be found HERE.

Console into one of the problematic AP.  Is it complaining about "Max retransmission count exceeded, going back to DISCOVER mode."  If it is, then this is CSCwh61011.

[Gaurav Kansal]

Dear Frederick,

It seems you are hitting a bug CSCwe04602. This bug is reported in your version 17.9.3.50. Please go through this bug and if this relates your issue as it looks like you mentioned in your summary, then please go through with known fixed release mentioned by cisco.

Please vote helpful if you find this working.
Good Luck..

[frederick.mercado]

This is not in relation to us. We do not run SDA Fabric. Furthermore, this is just one AP. We have opened a RMA for the AP.

[kerstin-534]

We have same problem with 9130 and error

kernel: [*08/29/2024 18:19:46.9273] !!!!! {/opt/cisco/bin/capwapd} failed writing /click/pmk_tracker/add_cache_param_via_tlv len 192 data

Version 17.9.5.201