C9800-CL - APs Disjoined

[Dines]

Hi 

Using C9800-CL

Ver 17.15.3

Flexmode, remote sites.

AP model - C9120-AXI

I am getting this error from multiple sites/APs across the network. AP is up, no disconnection, stable network, but on Controller they are getting dijoined msg every 30-50 seconds. DHCP is stable, some APs are working find at site but some are not.

Jul 29 04:10:49.890: %CW_LIC-5-AP_AIR_STATE: Chassis 1 R0/0: wncd: AP with mac address xxxx.xxxx.8c20 has moved to AIR state

Jul 29 04:10:41.605: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: XXCWAP-02 Mac: xxxx.xxxx.8c20 Session-IP:  xxx.xxx.xxx.132[5255] 172.25.159.32[5246] Disjoined DTLS close alert from peer

[masonreed11tt]

This looks like the AP is closing the DTLS session, not necessarily losing connectivity. Since only some APs are affected, compare their logs, switch ports, PoE settings, and software versions with those of working APs. Also check for known bugs in and review CAPWAP debug logs to identify what's triggering the repeated rejoin.

[Mark Elsen]

  - @Dines             The message CW_LIC-5-AP_AIR_STATE ; has a severity level  -5- which means that it is informational.
                              If this message occurs, it implies that the given AP has moved to AIR state , which means that it has
                              joined the controller again. If you are using flexconnect it should not impact clients (?)

                              Consider using the latest advisory release for 17.15. :
                                                 https://software.cisco.com/download/home/286322605/type/282046477/release/IOSXE-17.15.5

  M.

[Dines]

WiFi is not stable though at remote sites, users are getting disconnected 

[Mark Elsen]

  - @Dines                               Added ; look at port counters on switch-interfaces for problematic access points
                                                            and check speed and duplex mode for connected access points;
                                                            make sure there are no anomalies

           
                                                 Go to 17.15.5 first 
                                    - Troubleshoot problematic access points with instructions from :
                                                      https://logadvisor.cisco.com/logadvisor/wireless/9800/9800APJoin
                                    + Use commands from https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#toc-hId--1620538988
                                       to get further insights.
                                    + Check logs on switches where APs get disconnected
                                   + Validate the 9800 controller's configuration using the CLI command 
                                      show tech wireless and feed the output from that into : Wireless Config Analyzer

  M.

[Mark Elsen]

  - @Dines                 I also advice to   enable syslog messages in access points and controller towards a syslog server
                                 according to :
     https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_wl_16_10_cg/enabling-syslog-messages-in-access-points-and-controller-for-syslog-server.html

  M.

[Mark Elsen]

    - @Dines                   Use commands from : https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#toc-hId-866973845
                                                             For troubleshooting

                                      + Use latest advisory release the  for 17.15.x software train

  M.

[Dines]

There were two issues, connectivity lost towards secondary controller and AP started showing Certificate validation chain broken - DTLS issues,

Established connectivity and bounced the AP CAPWAP Tunnel and some of them needed bounce the switchport where APs was connected and its started working.

[Mark Elsen]

   - @Dines              Good that things work again!; but the particular message Certificate validation chain broken ; it indicates
                               a configuration error or conflict somewhere; such as not symmetrical configured N+1 HA controllers,
                               not using NTP. Sometimes when people import certificates also
                               the directive  chain-validation stop    can be used .
                               For the time being I don't know the exact cause. 
                                                      Currently your action must be considered a workaround only

  M.

[Dines]

Thanks Mark

Agree with you, was not expecting users to experience the connectivity issue, 

I was not aware of the "chain-validation stop" this is helpful in this situation, i need to check and validate NTP once.

Regards

[Mark Elsen]

  - @Dines                  Indeed  "chain-validation stop"  is not  immediately relevant here ; it's only when
                                  certificates are imported for special purposes.   Ok, to check NTP

  M.

[Mark Elsen]

   - @Dines              >....There were two issues, connectivity lost towards secondary controller and AP started showing Certificate validation chain broken - DTLS issues,

                                    I have been doing some research on my 9800 controller in the  LAB :
                                    The 'Certificate validation chain broken' message can be ignored , it's always shown when the access 
                                   point looses connection with the controller

                                   To investigate further use :
                                                          WLC # sh wireless stats ap join summary
                                                                  #  sh wireless stats ap mac-address 687d.b45e.ea00 join detailed | b AP re
                                                                                           Replace that mac address with a Base MAC of an access
                                                                                           point under investigation

                                 Other useful commands : 
WLC# show ap ethernet statistics
           show platform hardware chassis active qfp feature wireless capwap datapath statistics drop all
           show platform hardware chassis active qfp feature wireless capwap datapath mac-address <APradio-mac> details
           show platform hardware chassis active qfp feature wireless capwap datapath mac-address <APradio-mac> statistics
           show platform hardware chassis active qfp feature wireless dtls datapath statistics all

 M.