topic Re: Cisco 9115 CRASH Log Analysis in Wireless

Cisco 9115 CRASH Log Analysis

abhilash.vijayanand — Wed, 02 Mar 2022 19:53:10 GMT

Hi Everyone,
Here, I'm looking forward with help for analyzing the crash log file that i downloaded from 9800 controller for 9115AX access point.
We have frequent issues with these APs from the time we have upgraded our controller to 17.06.01 version.

For upgrading to next version, i will need to have a valid reason of bug or anything that shows the valid reason of this version is a buggy (Which i feel).

Here, I'm attaching the crash file for the AP I'm talking about.

TIA

Re: Cisco 9115 CRASH Log Analysis

Haydn Andrews — Wed, 02 Mar 2022 21:42:04 GMT

recommend putting the WLC Analyser to ensure that there isn't an issue with the config issue

https://cway.cisco.com/wireless-config-analyzer/

That being said the AP crashed due to Kernel panic

Unable to handle kernel NULL pointer dereference at virtual address 00000000

There are a number of public bugs for these lines but none related to your code version. Recommend engaging TAC to review the logs and provide root cause

The current TAC recommended version is 17.6.2 so if it is a bug TAC more than likely going to recommend this version.

https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html

Re: Cisco 9115 CRASH Log Analysis

Leo Laohoo — Wed, 02 Mar 2022 23:43:58 GMT

@abhilash.vijayanand wrote:
Kernel panic - not syncing: softlockup: hung tasks

Raise a TAC Case and get them to confirm if this is CSCvw00282.

Re: Cisco 9115 CRASH Log Analysis

abhilash.vijayanand — Thu, 03 Mar 2022 13:16:58 GMT

Hello Haydn Andrews,

Thanks for checking on this for me. Appreciate your time in responding to my post.

I have tried uploading the config file to WLC Analyzer but its not showing any activity. Is there any specific command that i need to run on Controller and put that output in notepad to check the results?

Re: Cisco 9115 CRASH Log Analysis

abhilash.vijayanand — Thu, 03 Mar 2022 13:19:24 GMT

Hello Leo Laohoo,

Thanks for checking on this for me. Appreciate your time in responding to my post.

I see that the bug which you provided is for 1800 Series Aps. (Understand that the logs as similar to mine).

I don't see any post in community related to 9115 Aps.
Thanks.

Re: Cisco 9115 CRASH Log Analysis

Scott Fella — Thu, 03 Mar 2022 14:54:18 GMT

Well, you upgraded and then you started having these issues. This should have been a flag for you to revert back. Then what you should do next is validate in your lab to determine what other version works and what other version doesn't. You will not fix any crash issues without upgrading or downgrading. Even if you open a TAC case, they probably would tell you to do the same. My recommendation is for you to revert back, then figure out in your lab what you next steps will be.

Re: Cisco 9115 CRASH Log Analysis

abhilash.vijayanand — Thu, 03 Mar 2022 17:16:26 GMT

Hello @Scott Fella,

Thanks for checking on this for me. Appreciate your time in responding to my post.

1st of all, We started having these issues after a week of upgrade and we didn't see any of such issues in our lab before upgrading.

We can't go back now due to dependencies. The only thing i can think of now is to upgrade to next golden version but i will need to prove my client with some proof that this is a bug. So, was requesting the same through this post.

Thanks & Regards,

Abhilash

Re: Cisco 9115 CRASH Log Analysis

Scott Fella — Thu, 03 Mar 2022 17:54:35 GMT

What I have seen in our environment is that it takes time before users start complaining and raising the issue up. So a week after, is normal, I have even seen a month after. So your next steps is to open a TAC case so that everything is documented which allows you to upgrade or downgrade. You are fighting a battle of an upgrade that was pushed and impacting user experience. The faster you resolve that the better for you. Work with TAC on this.

Re: Cisco 9115 CRASH Log Analysis

Rich R — Thu, 03 Mar 2022 22:38:59 GMT

> I see that the bug which you provided is for 1800 Series Aps. (Understand that the logs as similar to mine).

> I don't see any post in community related to 9115 Aps.

Both APs run the AP COS operating system so any platform independent code in COS could affect both APs similarly.

Re: Cisco 9115 CRASH Log Analysis

Leo Laohoo — Thu, 03 Mar 2022 23:27:50 GMT

@abhilash.vijayanand wrote:

I see that the bug which you provided is for 1800 Series Aps. (Understand that the logs as similar to mine).

Do not look at the model of the APs. It might say 1800 but underneath the "hood", the bug affects APs running Cheetah OS which is the OS from 1800/2800/3800/4800 and up to the Catalyst 9k APs.