Cicsco 9120AXI-E

[Dominique92100]

Hello, we have a Wifi network with 4 Cisco C9120AXI-E access points and 2 C9115 AXI-E access points.

All access points are in EWC mode and well connected to WLAN Wifi.

The firmware version of the controllers is 16.12.4.31 and the boot version is 1.1.2.4.
The ios version is 16.12.4.31.

We have the following problem :

The primary controller (current primary) seems to restart and we lose wifi during the change of primary controller.

A new controller is assigned, and can also reboot after 2 or 3 hours, with a new change of primary controller and a new wifi network cut.

Have you encountered this problem before?
We tested the POE injectors, the network cables, swapped terminals to test the locations, the problem persists.

Thanks in advance.

Dominique

[marce1000]

 - Check the logs when this happens ,preferably use a syslog-server on all intended controllers as a log-collector (gives better overview for determining a problem pattern (e.g.))

 M.

[Dominique92100]

Hello marce,

thanks four you reply.

I found this in the log :

It seems to be a controller crash, but why, I don't know..

Our configuration is really basic..

Feb  3 18:15:21.382: %WEBSERVER-5-LOGIN_PASSED: Chassis 1 R0/0: nginx: Login Successful from host 10.242.2.4 by user 'logic' using crypto cipher 'ECDHE-RSA-AES128-GCM-SHA256'

Feb  3 18:15:19.508: %WEBSERVER-5-SESS_TIMEOUT: Chassis 1 R0/0: nginx: Session timeout from host 10.242.2.4 by user 'logic' using crypto cipher 'ECDHE-RSA-AES128-GCM-SHA256'

.Feb  3 17:42:32.730: %SMART_LIC-5-SYSTEM_CLOCK_CHANGED: Smart Agent for Licensing System clock has been changed

Feb  3 16:39:34.503: %WEBSERVER-5-LOGIN_PASSED: Chassis 1 R0/0: nginx: Login Successful from host 10.242.2.7 by user 'logic' using crypto cipher 'ECDHE-RSA-AES128-GCM-SHA256'

.Feb  3 16:03:37.128: %SMART_LIC-5-SYSTEM_CLOCK_CHANGED: Smart Agent for Licensing System clock has been changed

.Feb  3 16:03:34.696: %SMART_LIC-5-SYSTEM_CLOCK_CHANGED: Smart Agent for Licensing System clock has been changed

Feb  3 14:26:16.113: %SMART_LIC-5-EVAL_START: Entering evaluation period

 Feb  3 14:26:16.110: %SMART_LIC-5-EVAL_START: Entering evaluation period

Feb  3 14:22:22.697: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

Feb  3 14:22:21.691: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

Feb  3 14:21:28.511: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP1CD1.E0EF.A1C4, MAC: 1cd1.e0ef.a1c4 Joined

Feb  3 14:21:28.301: %APMGR_TRACE_MESSAGE-3-EWLC_AP_LOG_MSG: Chassis 1 R0/0: wncd: AP Remote Log: a49b.cd41.0e20  Last AP AP1CD1.E0EF.A1C4 reboot was due to crash

Feb  3 14:21:08.161: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger

Feb  3 14:21:07.983: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger

Feb  3 14:21:07.399: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process emd

Feb  3 14:21:06.850: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process fman_rp

Feb  3 14:21:04.577: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process mobilityd

Feb  3 14:21:02.516: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process wncmgrd

Feb  3 14:21:01.919: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process wstatsd

Feb  3 14:21:01.363: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger

Feb  3 14:21:01.189: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger

Feb  3 14:21:00.039: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process emd

Feb  3 14:21:00.696: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))

Feb  3 14:21:00.696: %REDUNDANCY-3-REDUNDANCY_ALARMS: Unable to assert REDUNDANCY alarm

Feb  3 14:21:00.696: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))

Feb  3 14:20:59.532: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process fman_rp

Feb  3 14:20:56.705: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process mobilityd

Feb  3 14:20:55.660: %STACKMGR-6-STANDBY_ELECTED: Chassis 1 R0/0: stack_mgr: Chassis 2 has been elected STANDBY.

Feb  3 14:20:55.010: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process wncmgrd

Feb  3 14:20:55.662: %IOSXE_REDUNDANCY-6-PEER: Active detected chassis 2 as standby.

Feb  3 14:20:54.576: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process wstatsd

Feb  3 14:20:54.183: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger

Feb  3 14:20:54.031: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger

Feb  3 14:20:53.072: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process emd

Feb  3 14:20:52.638: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process fman_rp

ss mobilityd

Log Buffer (4096 bytes):

 Logging Source-Interface:       VRF Name:

Trap logging: level informational, 156 message lines logged

 Persistent logging: disabled

Count and timestamp logging messages: disabled

Exception Logging: size (4096 bytes)

filtering disabled

Buffer logging:  level debugging, 153 messages logged, xml disabled,

 filtering disabled

 Monitor logging: level debugging, 0 messages logged, xml disabled,

 filtering disabled

Console logging: level debugging, 143 messages logged, xml disabled,

No Inactive Message Discriminator.

No Active Message Discriminator.

Syslog logging: enabled (0 messages dropped, 13 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

And this morning :

A new primary Controler changing :

Feb 4 07:34:38.245: %WEBSERVER-5-LOGIN_PASSED: Chassis 2 R0/0: nginx: Login Successful from host 10.242.2.2 by user 'logic' using crypto cipher 'ECDHE-RSA-AES128-GCM-SHA256'

Feb 4 07:34:33.513: %WEBSERVER-5-SESS_TIMEOUT: Chassis 2 R0/0: nginx: Session timeout from host 10.242.2.2 by user 'logic' using crypto cipher 'ECDHE-RSA-AES128-GCM-SHA256'

Feb 4 06:07:57.689: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: AP1CD1.E0EF.D5E8, MAC: 1cd1.e0ef.d5e8 Joined

This one : AP1CD1.E0EF.D5E8, MAC dissapear from the pool yesterday and come back this morning...

.Feb 4 01:07:43.913: %SMART_LIC-5-SYSTEM_CLOCK_CHANGED: Smart Agent for Licensing System clock has been changed

[marce1000 · Accepted solution]

 - Looks like a bug you may want to upgrade to https://software.cisco.com/download/home/286322988/type/286323077/release/17.7.1 or the latest one in the 16.x train : https://software.cisco.com/download/home/286322988/type/286323077/release/16.12.6a

 M.

[Dominique92100]

Thank you for this quick feedback, I'm having a little trouble translating your message, you mean I have to do the update ?

Or does the last update still have the bug?

Thanks

[marce1000]

                           >Or does the last update still have the bug?

 - I am not sure because I did not find a specific bug report, the idea is to try out these releases (one to start with) , and check if you still see the same problem or not.

 M.

[Dominique92100]

Thank you

I will try 17.7.1 and just see if it happen again.

[marce1000]

 - Ok, good luck , some more advanced stuff coming up , you may if problems  then persist connect to an EWC (or one of the EWC's) with : https://cway.cisco.com/cli/ (thorugh SSH), you can then on the  upper left run (press) System Diagnostics, and on the right you have Crashdump Analyzer, both could provide more insights.

 M.

[Dominique92100]

Thank you very much, it's really cool to answer my questions. So I have a last one, regarding the licenses, I did not buy any licenses with the access point, and in the license management section, I have this (screenshot). 

At the end of the evaluation, I will lose some advanced functions, or I will have a real problem with the access point and Wifi access ?

ah another thing, if one day you go to Paris, send me an email I owe you a drink in a bar near the Eiffel Tower

[marce1000]

 - To be honest I know very little about 'Cisco licensing' , I am more a technical man. For the time being check if  the upgrade(s) , can solve the crashing problem.

 M.

[Dominique92100]

okay, thank, i will keep you in touch.

Regards

[Scott Fella]

From what I was told by my Cisco sales team is that every AP that will be connected to an IOS controller, requires either a DNA-Essentials or DNA-Advantage license.  This is due to the fact that the IOS controllers require a smart account.  Now I have a lab setup and never did anything to register a token, but my ap's are online.  Will that fail later, I have no clue.  I don't think that would cause your issue since ap's did join the EWC controller.  Licensing is the worse to understand.

[Dominique92100]

Totally agree with you, you need a PhD or a Masters in marketing to understand how Cisco licenses work. My engineer just tells me that if we use CAPWAP access points in connection with other controllers than the integrated EWC, the license is useful. I also read in passing on a website that a particularity of the EWC was not to need a license...

[Scott Fella]

I look at it this way... if in CCW the ap line item has options for DNA license, then it is required when connected to DWC/9800.  If the ap is going to join an AireOS, then the DNA license is not needed because its a different license you use.  I think folks will eventually learn the more they have to use these ap's.

[Rich R · Accepted solution]

EWC does not require any licenses.

https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/nb-o6-embded-wrls-cont-ds-cte-en.html#Licensing

Licensing

Cisco Embedded Wireless Controller on Catalyst Access Points licensing is dependent on each customer’s specific use case.

Use case A: The Embedded Wireless Controller (EWC) can be used for small deployments where EWC is only managed using the smart dashboard or mobile app. For this use case, a Cisco DNA subscription license and Smart Licensing is not required for any access point connecting to the EWC.

That software is quite old now.  Like @marce1000 recommended - best to start with updating to a recent release.