topic Re: Understanding Various AP-IOS Flash Corruption Issues in Wireless

Understanding Various AP-IOS Flash Corruption Issues

panayiotiscy — Mon, 05 Jul 2021 17:14:33 GMT

Hello Team,

So we are about to upgrade a wlc pair for our customer and as a pre-requisite we are following the instructions of this document:https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-lan-wlan/213317-understanding-various-ap-ios-flash-corru.html#anc22

The python/pip are successfully installed, edited the config.ini to match our information and gave it a shot.

After a while, the results are printed but It seems that we are facing a fundamental error for all APS:

<snip>

Errors
============================================================
AP connection error: Telnet instance has no attribute 'get_transport': 92

AP connection error: timed out : 15
AP connection error: telnet connection closed: 2
============================================================

Has anyone faced the same error? Any tips/ideas ?

To note, we are using windows server 2012-64bit instead of the Win10 purely because this is the only available server to access the management network of the WLC. Also, telnet client is enabled on the Windows Server.

Thank you

Panayiotis

Re: Understanding Various AP-IOS Flash Corruption Issues

Ric Beeching — Thu, 11 Apr 2019 13:55:15 GMT

Hi,

1) Can the windows server telnet to the AP from command prompt? Just to "telnet <ip>" and see if it can get there. Or use putty to test.
2) Are the APs configured globally to accept telnet connections?
3) Can you telnet from another device to the AP and login with the same credentials in the config.ini?
4) Is a firewall potentially blocking the connection either locally on the win server or on between the server and firewall?

Ric

Re: Understanding Various AP-IOS Flash Corruption Issues

Leo Laohoo — Fri, 12 Apr 2019 06:46:08 GMT

This bug is fixed in 8.5.140.0. What firmware is the controller running on?

Re: Understanding Various AP-IOS Flash Corruption Issues

panayiotiscy — Fri, 12 Apr 2019 06:51:01 GMT

Hello Ric,

Yes, thats doable..something else must be causing this behavior but i cant tell for sure...

thanks

Re: Understanding Various AP-IOS Flash Corruption Issues

panayiotiscy — Fri, 12 Apr 2019 06:52:49 GMT

Hello Leo,

Its actually an old one, under 8.2.x, thats why we would like to run the script and then proceed with the upgrade.

Does the script support the 8.2.x releases?

Thanks

Re: Understanding Various AP-IOS Flash Corruption Issues

patoberli — Fri, 12 Apr 2019 09:05:42 GMT

Yes it should. Please note, the script tries to login via ssh or telnet (I think you must configure this first) to each AP and then run some tests. After that you have to restart the script again and it runs some other tests. I can't remember if I run it a third time in total, but I think running it twice is enough.
The testing server should be in the same VLAN as the AP management interface. Make sure telnet/ssh is enabled on the APs (by default it's disabled) and also make sure that the user/pass you configured in the script is identical to the one you have set for the AP management.

And most importantly, use Python 2.x and not version 3. Scripts written for 2.x aren't compatible with 3.x.

Re: Understanding Various AP-IOS Flash Corruption Issues

Leo Laohoo — Fri, 12 Apr 2019 09:16:45 GMT

Yes it does but you'll need to run the script regularly, as in DAILY.

Re: Understanding Various AP-IOS Flash Corruption Issues

panayiotiscy — Fri, 12 Apr 2019 09:17:33 GMT

Hello Patobeli,

The server is indeed in the same vlan as the management vlan of the WLC, not the APs though...

I dont think its a telnet/ssh problem, the credentials are correct.

Please see the log output for a specific ap (all the other APs have the same log )

<snip>

2019-04-08 21:37:26,867 wlanpoller INFO: WLC processing complete; waiting for AP data collection to complete
2019-04-08 21:37:29,253 wlanpoller INFO: KXXXX : AIR-CAP1602I-E-K9 - Connected - IP addr: a.b.c.d
2019-04-08 21:37:29,253 wlanpoller INFO: KXXXX : AIR-CAP1602I-E-K9 - Collecting poller output
..
..
..
2019-04-08 21:38:18,674 wlanpoller CRITICAL: KXXXX : AIR-CAP1602I-E-K9 - Connection error (telnet://a.b.c.d) Telnet instance has no attribute 'get_transport'

<snip>

What do you think?

Thanks

Re: Understanding Various AP-IOS Flash Corruption Issues

panayiotiscy — Tue, 16 Apr 2019 08:18:43 GMT

Hi Leo,

Is there any way to enable additional debugging information for the script?

The failure reason doesnt make any sense and i dont really know how to troubleshoot this further.

Thanks

Re: Understanding Various AP-IOS Flash Corruption Issues

Leo Laohoo — Tue, 16 Apr 2019 08:51:43 GMT

How many APs do you have? And have you seen behaviours where the APs boot up with corrupted IOS?

Re: Understanding Various AP-IOS Flash Corruption Issues

panayiotiscy — Tue, 16 Apr 2019 09:12:31 GMT

Hello Leo,

We have about 500 Access points in total.

Access points failures are not frequent and of course not sure even if the low number of failures is associated with the flash corruption issue.

We are just eager to run the script, identify the candidate Access Points to fail and proceed with the WLC upgrade.

But, the script fails so we are in the dark.

Re: Understanding Various AP-IOS Flash Corruption Issues

patoberli — Tue, 16 Apr 2019 09:54:00 GMT

Can you try, from your script client, to telnet into every single AP by hand (or at least some) and see if the connection and enable works?

That must work, only then the script will also work.

Re: Understanding Various AP-IOS Flash Corruption Issues

panayiotiscy — Tue, 16 Apr 2019 12:36:23 GMT

Hello Patoberli,

checks performed:

from server to wlc:

the script logs-in and retrieves the AP list successfully

from server to APs:

using native telnet on Windows server we can telnet to the AP(s) and log in using the credentials defined in config.ini

I think that this is also depicted below but it doesnt make sense because the script initially reports that its connecting to the AP(s) but then it reports error in connecting using telnet:

snip>

<snip>

The way i interpret the Critical error is that the script cant allocate the necessary telnet resource as such it fails, but this is only my understanding...

Thanks

Re: Understanding Various AP-IOS Flash Corruption Issues

patoberli — Tue, 16 Apr 2019 13:21:03 GMT

Which version of Python is installed and is it configured properly in the PATH variable?

Re: Understanding Various AP-IOS Flash Corruption Issues

Leo Laohoo — Wed, 17 Apr 2019 09:32:52 GMT

If you've seen this behaviour with your APs, what model of APs are most affected?

Re: Understanding Various AP-IOS Flash Corruption Issues

panayiotiscy — Wed, 17 Apr 2019 13:59:45 GMT

Hello Patoberli,

i am attaching the relevant output for pip/ path details.

Re: Understanding Various AP-IOS Flash Corruption Issues

panayiotiscy — Wed, 17 Apr 2019 14:01:09 GMT

hello leo,

it fails for all of them regardless of AP model

Re: Understanding Various AP-IOS Flash Corruption Issues

patoberli — Wed, 17 Apr 2019 14:40:25 GMT

Looks correct.

Then I'm out of ideas sorry.

Honestly, I haven't yet seen people complaining about broken images on their AP here on the forum.

Thus my suggestion is:

Upgrade to the latest WLC version, 8.3.150.0 was released today, it includes the flash corruption fix. Same as 8.8.120.0 and 8.5.140.0. Then, before rebooting the WLC, pre-load the new image to all APs. If that shows as successful, the chances are very good that they will come up as soon as you reboot the WLC correctly.

Re: Understanding Various AP-IOS Flash Corruption Issues

Leo Laohoo — Thu, 18 Apr 2019 07:47:10 GMT

Here is one way:
1. Make sure each AP still have the RCV sub-directory.
2. Go to each AP and erase the CAPWAP image.
When the bug hits, the AP will reboot and will be forced to load the RCV image. The AP will then join the WLC and download the AP.
Monitor the WLC and make sure to always erase the CAPWAP image.