topic RE: C9115AXI Reboot issue in Wireless

C9115AXI Reboot issue

JeongSeok_Kim — Sat, 21 Jan 2023 05:14:56 GMT

4 out of 6 wireless APs connected to a specific switch have an unknown reboot problem.

WLC Model: C9800-L-F-K9

WLC OS: 17.6.3

Uplink Type: 1G UTP

AP Model: C9115AXI

AP OS: 17.6.3

Uplink Speed: 1G UTP

Power: AIR-PWRINJ6

Switch Model: C2960L-24TS

Switch OS: 15.2(6)E

The Max Retransmission or Heartbeat Timeout on the WLC identifies the reason for disconnection, and the reboot reason is Unknown on the wireless AP itself. This phenomenon occurs repeatedly for as short as 20 minutes and as long as 4 hours.

There is no clear cause analysis result in Cisco TAC, and the action method for me to solve it myself is as follows.

1. Downgrade WLC OS 17.6.4 -> 17.6.3 : Unresolved

2. Replacing all 4 problematic wireless APs : Only 1 wireless AP problem is resolved

3. Wireless AP power injector and switch port replacement : Unresolved

4. Syslog setting in wireless AP: Wireless AP reboots without sending any specific log to syslog

5. The demolished wireless AP, which had an existing problem, was connected to the computer room for testing purposes : the reboot problem was not reproduced.

Has anyone experienced and resolved these symptoms?

Re: C9115AXI Reboot issue

Rich R — Sat, 18 Feb 2023 12:30:53 GMT

No, not seen any particular problem like that with 17.6.4 - pretty stable for us.

It could be particular types of traffic which trigger the AP crash. That traffic might be present in your remote location but not in your computer room. Also were you using the same power supply in the computer room (unstable power supply can also trigger reload)?

I certainly would not recommend downgrade to 17.6.3 - don't know why TAC told you to do that. Upgrade to 17.6.4 solved a number of problems for us. 17.6.5 is out now and includes a number of AP crash fixes so try that?
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-6/release-notes/rn-17-6-9800.html#resolved-caveats-1765

Re: C9115AXI Reboot issue

Flavio Miranda — Sat, 18 Feb 2023 19:11:32 GMT

This can be some kind of incompatibility. If we look at the Power injector data sheet it states:

"The single-port PoE injector provides up to 30 Watts and is compliant with IEEE 802.3af (15.4 W) and
IEEE 802.3at (30 W) systems. This power injector is for use with the Cisco Aironet 2800 series and 3800
series access points."

And looking into Access Point Datasheet it states:

Cisco power injector, AIR-PWRINJ5= (Note: This injector supports only 802.3af)

https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9100ax-access-points/datasheet-c78-741988.html

Re: C9115AXI Reboot issue

JeongSeok_Kim — Sun, 19 Feb 2023 02:47:47 GMT

Downgrading to 17.6.3 was my decision and my final action was to replace the L2 switch.

I didn't find anything unusual in the show tech of the L2 switch, but I couldn't get support because there was no Cisco service, and it was eventually resolved after replacing the L2 switch hardware.

Thank you for posting.

Re: C9115AXI Reboot issue

JeongSeok_Kim — Sun, 19 Feb 2023 02:48:12 GMT

Downgrading to 17.6.3 was my decision and my final action was to replace the L2 switch.

I didn't find anything unusual in the show tech of the L2 switch, but I couldn't get support because there was no Cisco service, and it was eventually resolved after replacing the L2 switch hardware.

Thank you for posting

Re: C9115AXI Reboot issue

Rich R — Sun, 19 Feb 2023 11:44:26 GMT

Any AP which works with AIR-PWRINJ5 should also work with AIR-PWRINJ6 but the converse does not apply (so APs which require AIR-PWRINJ6 may not work with AIR-PWRINJ5).

Re: C9115AXI Reboot issue

Polestar — Tue, 08 Aug 2023 12:07:08 GMT

We are running 17.3.5.42 on C9115AXI and our upstream switches are all C9000 series. I am experiencing the APs connected to the upstream switch reloading without any warning error msgs - anything between 4 mins to 4 hours ...

Re: C9115AXI Reboot issue

JeongSeok_Kim — Wed, 09 Aug 2023 01:17:40 GMT

Hi Polestar

1. Does the reboot occur unconditionally within 4 hours?
2. Is there a reboot without outputting anything in the console log of the wireless AP?
3. Is the wireless AP power connection supply method PoE? If it is PoE, is the switch power module redundant?
4. If the PoE switch power module is redundant, can you repeat the test by disconnecting and connecting the power cable of the PS-1 power module?
5. Does the wireless AP reboot after doing step 4?

Re: C9115AXI Reboot issue

Polestar — Wed, 09 Aug 2023 08:13:37 GMT

Hi Kim

the answers to your questions...

1. Yes - reboots unconditionally

2. No logs -only notification of AP connected ( up-stream switch ) interface up/down

At this moment i am on my way to site to obtain the AP console logs as i cannot obtain them remotely. As the reboot occurring at random times , its a "wait game"

RE: C9115AXI Reboot issue

TAESU Kim — Wed, 09 Aug 2023 10:05:39 GMT

지금도 무작위로 재부팅 되는 증상이 있으신가여?

저도 같은 증상을 격고 있어서 문의 드립니다.

WLC - C9800-L-F-K9

AP - C9115AXI-K

Poe - C9300L-24P-4G
특이점은 천정에 매달았을경우 발생합니다
책상위에 올려놓고 테스트를 하면 문제가 없습니다

별도의 파워 인젝터를 사용해도 문제가 없습니다

Poe 스위치를 C3560G-24PS 로 교체한 경우 문제가 없습니다.

Re: C9115AXI Reboot issue

Rich R — Wed, 09 Aug 2023 10:43:43 GMT

Upgrade to 17.9.3 + APSPs or 17.9.4 (see TAC recommended below) and make sure your C9000 switches are alos running up to date code.

RE: C9115AXI Reboot issue

Rich R — Wed, 09 Aug 2023 10:47:47 GMT

@TAESU Kim what version of IOS-XE code is the WLC running?
Have you updated the PHY firmware on the WLC? (see TAC recommended doc below)

What version of IOS-XE code is the C9300 running? (that should also be up to date)

Most crashes (and reloads) are caused by software bugs (less frequently by hardware problems) so the first thing to do is make sure all the software is up to date.

Re: C9115AXI Reboot issue

JeongSeok_Kim — Wed, 09 Aug 2023 12:33:05 GMT

Hi, the reason I asked for the reason is that the irregular symptoms disappear when the wireless AP hanging from the bracket on the ceiling is removed. We narrowed down the cause to a leakage current issue and proceeded with insulation work.Hi, the reason I asked for the reason is that the irregular symptoms disappear when the wireless AP hanging from the bracket on the ceiling is removed. We narrowed down the cause to a leakage current issue and proceeded with insulation work.

RE: C9115AXI Reboot issue

JeongSeok_Kim — Wed, 09 Aug 2023 12:39:35 GMT

Hello Taesoo Kim

Does the hanging from the ceiling mean that you have installed the wireless AP on the bracket?

If you fixed the bracket with a piece rather than a rail, you can try to fix it by wrapping insulating tape around the piece.

Did you attach a bracket to the wireless AP when testing by placing it on the desk?

The fact that the symptoms disappear when the switch is replaced with an old model was the first thing I learned about that I had not tested in the environment.

RE: C9115AXI Reboot issue

Polestar — Thu, 10 Aug 2023 09:09:28 GMT

Just a quick update on this wierd scenario:

I removed the AP from its ceiling mounting bracket, so it was hanging by the interface cable ( connecting to the upstream switch ). I connected to the console port - that was yesterday and since then the AP has been UP ( no reloads no issues ) for iver 21 hours now.
What I have done now is I have remounted the AP to the ceiling bracket and waiting and watching the console msgs ( which seems to show no msgs at all )... Lets see if it crashes/reloads now

RE: C9115AXI Reboot issue

Polestar — Thu, 10 Aug 2023 09:39:36 GMT

AP is currently :

cisco C9115AXI-E ARMv8 Processor rev 0 (v8l) with 1971660/1185988K bytes of memory.
Processor board ID FCW2614YF7H
AP Running Image : 17.3.5.42
Primary Boot Image : 17.3.5.42
Backup Boot Image : 8.10.130.0

1 Multigigabit Ethernet interfaces
2 802.11 Radios
Radio Driver version : Broadcom BCA: 17.18 RC2.0
Radio FW version : 24203.122.r44027 44027
NSS FW version : NA

C9300L stack :

Version 16.12.4, RELEASE SOFTWARE (fc5)

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 17.8.1r[FC2], RELEASE SOFTWARE (P)

RE: C9115AXI Reboot issue

JeongSeok_Kim — Fri, 11 Aug 2023 01:15:08 GMT

Hi Polestar

TAC did not provide any cause analysis, and it is suspected to be an electrical problem when judged by itself.
If possible, it would be nice if you could open the TAC case separately and proceed with EFA for switches and wireless APs.

RE: C9115AXI Reboot issue

Rich R — Tue, 15 Aug 2023 14:55:16 GMT

Seriously consider getting the WLC and switch software updated (WLC is oldish, switch is antique).

The problem with running old versions like that is that even if they identify a bug you're not going to get a fix unless you migrate to a new version anyway and moving to new version resolves hundreds of existing bugs - one of which might be causing you problems. It's much easier for TAC to engage BU on problems if the software is up to date to start with and at least you know you've eliminated those known bugs which have already been fixed.

RE: C9115AXI Reboot issue

Polestar — Thu, 30 Nov 2023 15:31:51 GMT

Just a quick update on this story:
We upgraded our WLC to 17.9.3 which seems to have fixed the issue for many of our C9115-AXI-E APs.

There is still 1 ( only 1 ) that seems to reload at random times and i cannot see anything / reason cause in the console.

Power levels ok / no other upstream issues although i have to say the upstream switch C9300L is running on 16.12.4

All other sites / remote locations are running fine. Any ideas why the console output could be suppressed ? Other than last reload " unkown"

Re: C9115AXI Reboot issue

stuart.pannell — Tue, 12 Mar 2024 17:36:46 GMT

Hi all, we have been banging our heads over this issue for some time, we have opened 3 TAC cases and upgraded to the latest software at least 3 times. We are currently running 17.9.4a code on our WLC9800-40, 17.9.4a code on our 9200L switches and latest APSP 17.9.4.201. Yet we still have APs on multiple sites randomly rebooting with the error MAX RETRIES. If we move the AP off the ceiling mount then they appear to be stable and not reboot. Could I please ask how you are insulating these APs and if that is working for you?