topic Hello all.Today I realized in Wireless

All 5508 controllers just rebooted

sprocket10 — Mon, 05 Jul 2021 06:54:58 GMT

Curious if anyone has seen this.

All of our 5508 Wireless Controllers have just rebooted across our UK sites, all running version 7.4.110.0, all rebooted at the same time for no obvious reason. Our old 4402 controller didnt reboot.

All 5508 controllers just rebooted

Leo Laohoo — Wed, 08 Jan 2014 09:18:58 GMT

Did you find any crashinfo file?

Re: All 5508 controllers just rebooted

Sandeep Choudhary — Wed, 08 Jan 2014 13:36:38 GMT

Agree with Leo....

Paste the crashinfo from WLC :
How..... Check this:

COntroller>Management>Tech Support> Controller Crash

Regards

Sent from Cisco Technical Support iPhone App

Re: All 5508 controllers just rebooted

Rasika Nayanajith — Wed, 08 Jan 2014 18:49:14 GMT

How many controllers all together get rebooted. Check all controllers "show sysinfo" to see last reload reason & time of the reboot match each other.

Post one of them "show sysinfo" output to see

HTH

Rasika

***** Pls rate all useful responses ****

Re: All 5508 controllers just rebooted

sprocket10 — Thu, 09 Jan 2014 09:09:59 GMT

Sysinfo on all controllers shows:
Last Reset....................................... Software reset

I have pasted the first part of the crashinfo from 3 controllers below but seem identical.

All rebooted at same time (within a few seconds)

************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name: Cisco-HQ-5508-Grnd-C
Model:     AIR-CT5508-K9
Version:   7.4.110.0
Timestamp: Wed Jan 8 09:04:58 2014
SystemUpTime: 35 days 13 hrs 25 mins 39 secs
signal:    11
pid:       1070
TID:       1456812880
Task Name: emWeb
Reason: System Crash
si_signo: 11
si_errno: 0
si_code:   1
si_addr:   0x0
timer tcb:   0x83d
timer cb:    0x10e76bd0 ('alarmSendMsgToMsgTask+48')
timer arg1: 0x0
timer arg2: 0x0

Long time taken timer call back inforamtion:
Time Stamp: Wed Jan 8 09:04:58 2014
timer cb : 10e76bd0p('alarmSendMsgToMsgTask+48')
Duration : 320951 usecs, cbCount= 1

------------------------------------------------------------
Analysis of Failure:

Software Failed on instruction at :
pc = 0x1206c7b0 (osapiStrlcpy+72), ra = 0x10af5e38 (osapiStrlcpy+72)

Software Failed while accessing the data located at :0x0

------------------------------------------------------------

************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name: Cisco-GLA-5508-Glasgow
Model:     AIR-CT5508-K9
Version:   7.4.110.0
Timestamp: Wed Jan 8 09:04:49 2014
SystemUpTime: 36 days 23 hrs 33 mins 40 secs
signal:    11
pid:       1070
TID:       1435497296
Task Name: emWeb
Reason: System Crash
si_signo: 11
si_errno: 0
si_code:   1
si_addr:   0x0
timer tcb:   0x83c
timer cb:    0x10e76bd0 ('alarmSendMsgToMsgTask+48')
timer arg1: 0x0
timer arg2: 0x0

Long time taken timer call back inforamtion:
Time Stamp: Wed Jan 8 09:04:49 2014
timer cb : 10e76bd0p('alarmSendMsgToMsgTask+48')
Duration : 255209 usecs, cbCount= 2

------------------------------------------------------------
Analysis of Failure:

Software Failed on instruction at :
pc = 0x1206c7b0 (osapiStrlcpy+72), ra = 0x10af5e38 (osapiStrlcpy+72)

Software Failed while accessing the data located at :0x0

------------------------------------------------------------

************************************************************
*             Start Cisco Crash Handler                *
************************************************************
Sys Name: Cisco-ARW-5508
Model:     AIR-CT5508-K9
Version:   7.4.110.0
Timestamp: Wed Jan 8 09:04:46 2014
SystemUpTime: 36 days 23 hrs 36 mins 42 secs
signal:    11
pid:       1070
TID:       1457722192
Task Name: emWeb
Reason: System Crash
si_signo: 11
si_errno: 0
si_code:   1
si_addr:   0x0
timer tcb:   0x7fd
timer cb:    0x1102a950 ('nfaSyncMsgSendToTask+272')
timer arg1: 0x0
timer arg2: 0x0

Long time taken timer call back inforamtion:
Time Stamp: Wed Jan 8 09:04:46 2014
timer cb : 1102a950p('nfaSyncMsgSendToTask+272')
Duration : 93038 usecs, cbCount= 1

------------------------------------------------------------
Analysis of Failure:

Software Failed on instruction at :
pc = 0x1206c7b0 (osapiStrlcpy+72), ra = 0x10af5e38 (osapiStrlcpy+72)

Software Failed while accessing the data located at :0x0

------------------------------------------------------------

Re: All 5508 controllers just rebooted

Sandeep Choudhary — Thu, 09 Jan 2014 09:15:49 GMT

Hi,

1. Remove the consle cable from Controller.

2.wait for few minutes then reboot the controller after reboot check again.

3. Update the Firmware on WLCs.

4. If still persists then raise a TAC case, it may be ahardware failure so RMA'd.

also chekc this bug: https://tools.cisco.com/bugsearch/bug/CSCuh55653

Regards

All 5508 controllers just rebooted

Leo Laohoo — Thu, 09 Jan 2014 09:23:08 GMT

Hmmmm ... The crash info does not bode well.

On the upper right-hand corner of your screen you will see an option to create a TAC Case. I suggest you do so. In the meantime, get ready to upgrade to the latest 7.4 code currently available for download.

Re: All 5508 controllers just rebooted

Leo Laohoo — Thu, 09 Jan 2014 09:28:14 GMT

Task Name: emWeb

It's Cisco Bug ID CSCub88183.

And this is found in the Release Notes for both WiSM-2 and 5508 for firmware 7.4.110.0.

All 5508 controllers just rebooted

Saravanan Lakshmanan — Thu, 09 Jan 2014 14:46:29 GMT

post stack trace of the crash, need stack trace to get clue, sometimes that also does not help and require coredump of the crash. 7.4.121.0 or 7.4.111.X may have fix, if its an known bug.

All 5508 controllers just rebooted

Rasika Nayanajith — Thu, 09 Jan 2014 18:22:37 GMT

Have you purchased all these controllers at the same time ? check with Cisco TAC if there any issue with the hardware depend on its manufacturing details. We are having 9x5508 in our environment & never came across a situation like this where all controller rebooted at once.(had to RMA specific 5508 batch due to silent crashes time to time)

Make sure your FUS 1.7.0.0 & go for 7.4MR2 (7.4.121.0) for the time being & see what TAC says on this. If I was you, I would push for RMA all controllers.

HTH

Rasika

**** Pls rate all useful responses ****

All 5508 controllers just rebooted

Saravanan Lakshmanan — Thu, 09 Jan 2014 18:31:31 GMT

Generally, RMA is not possible for software reset reasons. Only if there is dataplane crash that can't be fixed via software will be considered as hardware failure besides other true failures.

Last Reset....................................... Software reset

In this case, the crash task was due to emweb, it is possible NMS device may be trying to push an config to all WLCs caused this crash. Stack trace and coredump should speak about the actual reason.

All 5508 controllers just rebooted

Rasika Nayanajith — Thu, 09 Jan 2014 22:07:29 GMT

I am not saying you are hitting this bug. but in certain cases bugs are applicable to HW manufactured since certain date/time. So always worth to check.

CSCul68057

Symptom:

Wireless LAN Controller may encounter unexpected reload without crash file or coredump.

Console log output may include "reaperWatcher rebooting" and "!!!!! Watchdog detected LOCKUP !!!!!",

and there may be "#OSAPI-2-REAPER_WATCHER_INFO" message in syslog.

Conditions:

5508, 2504 or WiSM2 manufactured since 2012.

Known Affected Releases

7.4(110.0)

7.0(240.0)

7.5(102.0)

Known Fixed Releases

7.4.121.0

7.6.100.0

HTH

Rasika

All 5508 controllers just rebooted

Saravanan Lakshmanan — Thu, 09 Jan 2014 22:35:16 GMT

Not trying to debate, at the same time trying to help out

This bug is about CF driver issue on WLC, it is not an hardware issue that require RMA, We can still upgrade the WLC to fixed code. And i don't see single RMA attached to this defect

Example of bug that require WLC RMA. It is not reoccuring so you don't find any external info.

https://tools.cisco.com/bugsearch/bug/CSCub00349

Re: All 5508 controllers just rebooted

Rasika Nayanajith — Thu, 09 Jan 2014 22:42:49 GMT

I agree with you.

What I wanted to say is certain software bugs only applicable to particular harware versions like in this case. Did not try to say RMA is the solution

Rasika

Re: All 5508 controllers just rebooted

Saravanan Lakshmanan — Thu, 09 Jan 2014 22:51:32 GMT

Sure. By the way most of the older AireOS releases will be deferred due to this bug within a week, field notice will be sent out.

Re: All 5508 controllers just rebooted

sprocket10 — Fri, 10 Jan 2014 09:53:21 GMT

What I find strange is that all our 5508 controllers rebooted at the same time. I will schedule an upgrade on them all to the latest version

All 5508 controllers just rebooted

Rasika Nayanajith — Fri, 10 Jan 2014 09:55:58 GMT

Go for the 7.4.121.0 (7.4MR2). Also ensure that your FUS upgraded to 1.7.0.0 as well.

Rasika

All 5508 controllers just rebooted

tonyp8581 — Wed, 15 Jan 2014 20:18:22 GMT

Hi sprocket10,

I'm having the same issue. I have 2 WLC 5508 (HA) running v7.4.110.0. Since September, never had any reboot problem until this week.

First, my primary WLC rebooted. Today, both WLC rebooted around the same time.

Have you had the chance to upgrade and did that fix the problem ?

Thx

Tony

Re: All 5508 controllers just rebooted

sprocket10 — Thu, 06 Mar 2014 09:59:15 GMT

Turns out it was Cisco Network Assistant when backing up

Hello all.Today I realized

Bruno Dinis — Wed, 18 Nov 2015 16:01:06 GMT

Hello all.

Today I realized that yesterday I've got a reload at 11.00am in my primary WLC.

(wlc1) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.0.115.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS

System Name...................................... wlc1
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 10.2.0.68
IPv6 Address..................................... ::
Last Reset....................................... Software reset

I've searched for syslog messages but got no lucky

Nov 17 11:12:30 wlc1: *apfMsConnTask_4: Nov 17 11:12:30.820: #APF-3-CHECK_SUPP_RATES_FAILED: apf_utils.c:418 Could not check supported rates. Missing Supported Rate. Length :0. Mobile MAC: 50:cc:f8:72:3b:45.
Nov 17 11:12:31 wlc1: *apfMsConnTask_4: Nov 17 11:12:31.408: #APF-3-CHECK_SUPP_RATES_FAILED: apf_utils.c:418 Could not check supported rates. Missing Supported Rate. Length :0. Mobile MAC: c0:f2:fb:d6:89:d4.
Nov 17 11:16:13 wlc1: *nim_t: Nov 17 11:16:12.869: #SIM-3-PORT_UP: sim.c:13485 Physical port 2 is up!.
Nov 17 11:16:13 wlc1: *nim_t: Nov 17 11:16:12.869: #SIM-3-PORT_UP: sim.c:13485 Physical port 3 is up!.
Nov 17 11:16:13 wlc1: *nim_t: Nov 17 11:16:12.870: #SIM-3-PORT_UP: sim.c:13485 Physical port 4 is up!.

How can I check the issue for the "software reset"?
In the GUI I've checked for Management - Tech Support - Controller Crash,
but got no messages (***** End of Crash File dump *****)

any tip to figure out what happened?