WLC C9800-L-F memory of bootflash runs full with RP system reports

[Shiden]

Hello all,

I recently configured two physical WLC C9800-L-F. Both WLCs are redundant and configured with HA SSO Switchover. The HA SSO is working perfectly so far.

Here are some details about the WLC that could help:

Model: C9800-L-F-K9

Firmware: Bengaluru-17.6.4

The problem I'm facing is the following. The memory of the core directory in the bootflash filesystem often run full of some system reports for a reason I still ignore (see screenshot). The most weird thing is that the system reports are not that big (in my case they don't go over 4Gb). But when I delete the biggest of them (3.6Gb) suddenly 24Gb are free again.

Does anyone have an idea what can cause those systems report, why they take so much space, and is it possible to disable them or to fix this issue?

 Here is the screenshot of the flash:core directory.

I already say thanks to the people who will take time to answer this post.

Best regards

[balaji.bandi]

check any logs enabled :

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/213949-wireless-debugging-and-log-collection-on.html

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/215523-quick-start-guide-on-what-logs-and-debug.html

[Shiden]

Hi @balaji.bandi,

Thanks for your answer. Unfortunately, I only will have access to the WLCs next week. They already have been deployed at the customer's site. I never enabled any logs functionalities. Are they enabled by default?

[Rich R]

System reports are usually caused by crashes.  Suggest to upgrade IOS to resolve crashes which already have fixes:
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-6/release-notes/rn-17-6-9800.html#resolved-caveats-1765
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-9/release-notes/rn-17-9-9800.html#resolved-caveats-for-cisco-ios-xe-cupertino-17.9.3
If you still see crashes after upgrade then open a TAC case to get the system reports decoded to understand the cause and possible workarounds/resolution.

[marce1000]

 - Adding to other reply if crashes are involved or suspected issue the command (CLI) : show version | inc reload
                                                                                                                              (Look for reload reason reported) , 

 M.

[Shiden]

Hi @balaji.bandi @Rich R @marce1000  ,

Thank you for your response! Logs are not enabled on the WLCs, and we cannot update them further as they have already been updated to the suggested version by Cisco. After conducting some tests, we discovered that the report is generated when the RP link is unplugged. This causes the active WLC to freeze and generate a large report, while the standby WLC becomes active. The biggest problem is that the previously active WLC that froze cannot be accessed even over serial, which can be a major issue when no one is on site. We had to unplug and replug the power supply of the WLC in order to access it again. Here is a screenshot of the report that was generated after the crash and the amount of free space available after deleting it.

Best regards

[marce1000]

  - As stated use this command too : show version | inc reload   , when that happens, 

 M.

[Shiden]

Hi @marce1000 ,
Sorry I forgot to mention it on my last post. Because of the freeze, the WLC has been unplug and replug for this reason the reload reason is the folowing:

WLC#show version | inc reload
Last reload reason: PowerOn

[marce1000]

            - Ok, in this case that seems logic , but remember the command for future use and or crashes too , 

M.

[Scott Fella]

As long as you can reproduce the issue, this is a good time to have TAC investigate.  This way they can determine if this is a bug, which it looks like since your controller locked up, but It is also give you good insight to have a PDU in which you can cycle the power to devices.  

[Shiden · Accepted solution]

Hi all,

I found the solution to that problem.

Here is the bug that was responsible for the described issue:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwd17349

I upgraded the WLCs from version 17.6.4 to 17.6.5 and the HA SSO worked fine. Also, no more from those huge system reports were created.

Thank you all for your answers

Best regards

[Rich R]

Thanks for the update @Shiden - good to know.