Cisco 9115 CRASH Log Analysis

[abhilash.vijayanand]

Hi Everyone,
Here, I'm looking forward with help for analyzing the crash log file that i downloaded from 9800 controller for 9115AX access point.
We have frequent issues with these APs from the time we have upgraded our controller to 17.06.01 version.

For upgrading to next version, i will need to have a valid reason of bug or anything that shows the valid reason of this version is a buggy (Which i feel).

Here, I'm attaching the crash file for the AP I'm talking about.

TIA

[Haydn Andrews]

recommend putting the WLC Analyser to ensure that there isn't an issue with the config issue

https://cway.cisco.com/wireless-config-analyzer/ 

That being said the AP crashed due to Kernel panic

Unable to handle kernel NULL pointer dereference at virtual address 00000000

There are a number of public bugs for these lines but none related to your code version. Recommend engaging TAC to review the logs and provide root cause

The current TAC recommended version is 17.6.2 so if it is a bug TAC more than likely going to recommend this version.

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html 

[abhilash.vijayanand]

Hello Haydn Andrews,

Thanks for checking on this for me. Appreciate your time in responding to my post.

I have tried uploading the config file to WLC Analyzer but its not showing any activity. Is there any specific command that i need to run on Controller and put that output in notepad to check the results?

[Leo Laohoo]

---

@abhilash.vijayanand wrote:

Kernel panic - not syncing: softlockup: hung tasks

---

Raise a TAC Case and get them to confirm if this is CSCvw00282.

[abhilash.vijayanand]

Hello Leo Laohoo,

Thanks for checking on this for me. Appreciate your time in responding to my post.

I see that the bug which you provided is for 1800 Series Aps. (Understand that the logs as similar to mine).

I don't see any post in community related to 9115 Aps.
Thanks.

[Leo Laohoo]

---

@abhilash.vijayanand wrote:

I see that the bug which you provided is for 1800 Series Aps. (Understand that the logs as similar to mine).

---

Do not look at the model of the APs.  It might say 1800 but underneath the "hood", the bug affects APs running Cheetah OS which is the OS from 1800/2800/3800/4800 and up to the Catalyst 9k APs.  

[Scott Fella]

Well, you upgraded and then you started having these issues.  This should have been a flag for you to revert back.  Then what you should do next is validate in your lab to determine what other version works and what other version doesn't.  You will not fix any crash issues without upgrading or downgrading.  Even if you open a TAC case, they probably would tell you to do the same.  My recommendation is for you to revert back, then figure out in your lab what you next steps will be.

[abhilash.vijayanand]

Hello @Scott Fella,

Thanks for checking on this for me. Appreciate your time in responding to my post.

1st of all, We started having these issues after a week of upgrade and we didn't see any of such issues in our lab before upgrading.

We can't go back now due to dependencies. The only thing i can think of now is to upgrade to next golden version but i will need to prove my client with some proof that this is a bug. So, was requesting the same through this post.

Thanks & Regards,

Abhilash

[Scott Fella]

What I have seen in our environment is that it takes time before users start complaining and raising the issue up.  So a week after, is normal, I have even seen a month after.  So your next steps is to open a TAC case so that everything is documented which allows you to upgrade or downgrade.  You are fighting a battle of an upgrade that was pushed and impacting user experience.  The faster you resolve that the better for you.  Work with TAC on this.

[Rich R]

> I see that the bug which you provided is for 1800 Series Aps. (Understand that the logs as similar to mine).

> I don't see any post in community related to 9115 Aps.

Both APs run the AP COS operating system so any platform independent code in COS could affect both APs similarly.