topic Re: Issue with 5520 Wireless Controller Software Crash in Wireless

Issue with 5520 Wireless Controller Software Crash

tanner.zaitt — Mon, 05 Jul 2021 19:37:03 GMT

Hello, I am in trouble with WLC Controller 5520.

The issue is related to reading the system info information from the ssh console.

For example, on Tuesday the WLC Controller is rebooted of a software crash reason.
When we log in a few minutes later and we check the uptime of the WLC with "show sysinfo" command
and there the uptime was okay and we ignore the mail notification:
System Up Time................................... 19 days 21 hrs 4 mins 46 secs for TESTWLC01.

But two days later we check it again and the output is different!

show sysinfo
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.10.121.0
RTOS Version..................................... 8.10.121.0
Bootloader Version............................... 8.3.15.177
Emergency Image Version.......................... 8.3.141.0

OUI File Last Update Time........................ Tue Feb 06 10:44:07 UTC 2018

Build Type....................................... DATA + WPS

System Name...................................... TESTWLC01
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.2170
Redundancy Mode.................................. SSO
IP Address....................................... 192.168.1.20
IPv6 Address..................................... ::
System Up Time................................... 1 days 19 hrs 36 mins 6 secs
System Timezone Location......................... (GMT +1:00) Amsterdam, Berlin, Rome, Vienna
System Stats Realtime Interval................... 5

--More-- or (q)uit
System Stats Normal Interval..................... 180

Configured Country............................... Multiple Countries : BG
Operating Environment............................ Commercial (10 to 35 C)
Internal Temp Alarm Limits....................... 10 to 38 C
Internal Temperature............................. +24 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 4
Number of Active Clients......................... 85

OUI Classification Failure Count................. 141798

Memory Current Usage............................. 11
Memory Average Usage............................. 11
CPU Current Usage................................ 0
CPU Average Usage................................ 0

Flash Type....................................... Compact Flash Card
Flash Size....................................... 1073741824

The second check is not like the first about uptime information.
An additional check with "show logging last-reset" show us real information about real software crash event 🙂

The output is :

show logging last-reset

(Cisco Controller) >show logging last-reset

!!!Message and Trap Logs from Previous Reset

--------------------------------------------- Last Reboot MsgLog & Traplog ---------------------------------------------
Sys Name: TESTWLC01
Model: AIR-CT5520-K9
Version: 8.10.121.0
Primary Boot Image: 8.10.121.0 (default) (active)
Backup Boot Image: 8.5.131.0
LastReset Reason: Software Crash
Timestamp: Tue Oct 6 14:35:36 2020
SystemUpTime: 27 days 22 hrs 52 mins 45 secs

And now I am confused ...

Where I can see really the true uptime of device after Software crash?

Is it possible to see the incorrect output of uptime in these two commands:
"show sysinfo" and "show logging last-reset" ?

And one last question is it possible to see logs only about software crash and reasons for these type issues?
And if a reboot is initiated from a specific user is possible to figure out the right user?

Thank you in advance!

Best regards!

Re: Issue with 5520 Wireless Controller Software Crash

Leo Laohoo — Thu, 08 Oct 2020 10:08:19 GMT

sh coredump summary

Re: Issue with 5520 Wireless Controller Software Crash

tanner.zaitt — Thu, 08 Oct 2020 11:29:41 GMT

Hi. thank you, could you tell me where right we should check in this coredump files?
How is possible to read the files?
I downloaded them and extracted them in Windows machine.

Re: Issue with 5520 Wireless Controller Software Crash

tanner.zaitt — Thu, 08 Oct 2020 11:38:02 GMT

The output is :

(Cisco Controller) >show coredump summary

Core Dump upload is disabled

Core Dump file on flash:

SW Version....................................... 8.10.121.0
Time Stamp....................................... Tue Oct 6 14:35:36 2020
File Size........................................ 68229995
File Name Suffix................................. _8.10.121.0_WLCTEST01_1601987736.gz

How is possible to read files in _8.10.121.0_WLCTEST01_1601987736.gz?

Re: Issue with 5520 Wireless Controller Software Crash

Leo Laohoo — Thu, 08 Oct 2020 11:40:18 GMT

To minimize the time for TAC to be waiting for outputs, please prepare the following:

Controller Bundle
sh tech
sh config
Core dump file

Send those to TAC so they can analyze the cause of the crash.

Re: Issue with 5520 Wireless Controller Software Crash

tanner.zaitt — Thu, 08 Oct 2020 12:02:47 GMT

We couldn't analyze the situation alone?

Re: Issue with 5520 Wireless Controller Software Crash

Mark Elsen — Thu, 08 Oct 2020 12:06:36 GMT

https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-0/configuration-guide/b_cg80/b_cg80_chapter_0110110.html#upload-logs

Re: Issue with 5520 Wireless Controller Software Crash

tanner.zaitt — Thu, 08 Oct 2020 12:40:18 GMT

I get the files, but I can't read them 🙂

Re: Issue with 5520 Wireless Controller Software Crash

Mark Elsen — Thu, 08 Oct 2020 12:50:01 GMT

- You mean unpacking .gz files ? Winzip can do that, if you are on linux use % gzip -d filename.

Re: Issue with 5520 Wireless Controller Software Crash

tanner.zaitt — Thu, 08 Oct 2020 13:16:17 GMT

I tried with Windows 10 machine and 7 zip and the files are numbers from 1 to 670.
With Notepad ++ and they are not readable.

With show tech I find only :

Reset reason : reset due to switch-driver crash ,(Version: 8.10.121.0)
Reset time : Tue Oct 6 14:41:31 2020
Trace back : 0x8f0454 0x1e06dea 0x1e07f46 0x8e8c85 0xf170 0x1366c4 0x129ef26 0x129efc2 0x129f43d 0x113b51d 0x113be29 0xf23bcc 0x236ef32 0x1e0fb97 0x7dff 0xe898d
Number of files present is: 9

Re: Issue with 5520 Wireless Controller Software Crash

Mark Elsen — Thu, 08 Oct 2020 15:12:47 GMT

- In any case is the info from the crash-files destined for specialized analysis by cisco tac.

Re: Issue with 5520 Wireless Controller Software Crash

tanner.zaitt — Thu, 08 Oct 2020 15:56:28 GMT

Thank you but it is a little complicated to connect to Cisco TAC team, the WLC is under the support of Cisco, but I am not part of the local infrastructure, just I am part of a different support team with monitoring functions and I can't initiate a case in TAC.

Thanks a lot for your help!
I appreciate it.
Best regards!

Re: Issue with 5520 Wireless Controller Software Crash

patoberli — Thu, 08 Oct 2020 20:37:54 GMT

You are probably hitting a bug which is solved in the newer release 8.10.130.0.

Release notes:

https://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn810mr3.html#resolved-caveats

Bug you probably hit:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvu23186

So I suggest upgrading to the newer release which should fix this bug.

Re: Issue with 5520 Wireless Controller Software Crash

tanner.zaitt — Fri, 09 Oct 2020 08:44:53 GMT

Hi, thank you!

Yes, this will solve the issue!

The answer to my question is right CSCvu23186.

Best regards!

Re: Issue with 5520 Wireless Controller Software Crash

tanner.zaitt — Tue, 13 Oct 2020 09:16:28 GMT

Hi,

We found and new bug:

CSCvs49779
https://quickview.cloudapps.cisco.com/quickview/bug/CSCvs49779

Could you share information about the upgrade procedure of 3x5520 WLC controllers configured with High Availability (SSO)?