Cisco 2702i on 9800-80-WLC Restarting radio due to reason code 27

[Gehrig_W]

Hello Cisco-WLAN-experts,

recently we have migrated a several-floor-building in this huge university hospital
covered by 2702i-APs to a 9800-80-WLC running on 17.9.3-code.

In many of these 2702i-APs I can see many messages dealing with this error-messages for Dot11Radio1 und Dot11Radio0

 %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 27

And always a block of following messages is logged in the AP log.

Jul  3 06:26:33.330: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 27
Jul  3 06:26:33.334: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down
Jul  3 06:26:33.338: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 10
Jul  3 06:26:33.338: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up

In many cases this block of messages appear in a regular time distance like every 15 minutes.

Jul  3 00:11:33.975: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
Jul  3 00:26:33.947: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
Jul  3 00:41:33.916: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
Jul  3 00:56:33.881: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
Jul  3 01:11:33.861: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
Jul  3 01:26:33.837: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
Jul  3 01:41:33.822: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
Jul  3 01:56:33.799: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down

What might be the reason for this ongoing administrative radio-resets on both antennas ?

I compared with different time deltas in the WLC RRM but found no 15 minutes delta time.

DCA-Interval: 10 Minutes

TPC-Interval: 600 Seconds

FRA-Interval: 60 Minutes

Neither our good old Prime nor our AI-based brandnew DNA-Center has a clue, what might be wrong or point out the reason for these administrative resets.

How can I cure this on the 2702i-APs or the 9800-80-WLC?

Thank You for any good advice from a human brain and human WLAN expert.

Greeeting from Wuerzburg

Wini

[marce1000]

  - Probably a  bug , go for controller release :
            https://software.cisco.com/download/home/286321396/type/282046477/release/Dublin-17.12.3
   which has recently become an advisory release too  , and check if that can help  , 

 M.

[Gehrig_W]

Hello marce1000, thank You for Your advise.

Unfotunately our ISE is still on V2.4 which is no longer supported according to Release-Notes V17.12.3

Kind  regards

Wini

[marce1000]

         >....But Version 17.12.3 is not supporting 2702i-APs anymore.
   - That's incorrect , look at https://www.cisco.com/c/en/us/td/docs/wireless/compatibility/matrix/compatibility-matrix.html 
      and look for 2700 , now then you will see it with (I/E) appended for 17.12.3  - It's time for Cisco to get a clean up on that page
      and or start looking for automating it  , like with the Optics Compatibility matrix.

     As far as Release Notes is concerned , Cisco has the habit of posting that a little later , and for good customer service it should be the reverse!

 M.

[marce1000]

      >...Unfotunately our ISE is still on V2.4 which is no longer supported according to Release-Notes V17.12.3
    Yes but that is worse , I have seen that for other businesses too , this is a bad showstopper ,and should get resolved on
    a  reasonably short timeframe , to follow for wireless standards. For instance the wifi7 APs will only work with 17.15.x

 M.

[marce1000]

    - Adding , they do have : https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214855-ios-xe-wireless-feature-list-per-release.html
           But there is a gap for....

 M.

[marce1000]

  - Seems that the Release Notes for 17.12.x are not available for all models on the download page :
          https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-12/release-notes/rn-17-12-9800.html

 M.

[Rich R]

Hi @Gehrig_W your problem is not that 9800 17.12 does not support ISE 2.4 - it's that ISE 2.4 is END OF SUPPORT!  So really you're in exactly the same position with 17.9 now.
https://www.cisco.com/c/en/us/products/collateral/security/identity-services-engine/bulletin-c25-743964.html
Because 2.4 is EOS it is not tested with 17.12 and therefore not listed as compatible but there's no reason why it shouldn't keep working.  What ISE functions/features are you using?  Have those changed between 2.4 and current supported releases?

What I'm saying is that you are already running at risk by using end of life ISE software and upgrading to 17.12 WLC software will not change that.  As long as you test the features you're using to confirm they all still work correctly there should not be a problem.

But you need to get that ISE updated asap.

Regarding your original question on the 2702 radio resets - I have no idea what is causing those.  You could try enable some debugs on the AP to see whether that gives you more info.  Unfortunately those APs are now end of support too so TAC are unlikely to be very helpful.  So your options are limited and upgrading software is probably the only other thing you can try.  Cisco extended the IOS AP support in 17.9 and 17.12 to give people more time to migrate but that time is now - they need to be replaced with currently supported AP models.

Usually when IOS resets radios it saves event.r0 and event.r1 files on flash which are like crashinfo files.  Look for those (and if the timestamps match the radio resets) and check the contents - they might give some idea to the reason for the resets.

[Gehrig_W]

Hello Rich,

thank You very much for Your information.

Regarding the 2702i Interface resets, I cannot find Your mentioned event.r0 and event.r1 files on flash.

Here ist the Directory-Info from a 2702i-Flash:

I startet to use debug on this 2702i-AP, which obviously created the files event.r0 and event.r1 you mentioned.

For example:

A15-AP80#debug dot11 d0 radio-reset fail or normal.    

Do You have a clue about these SPAM-messages:

ul 8 07:39:19.161: %EVT-5-NTC: CAPWAP Received SPAM_VENDOR_ID_PAYLOAD
Jul 8 07:39:19.161: %EVT-5-NTC: Vendor subtype AP_MFP_KEY_SUBTYPE

Could this be the problem for interface-resets ?

Greetings

Wini