Understanding Various AP-IOS Flash Corruption Issues

[panayiotiscy]

Hello Team,

So we are about to upgrade a wlc pair for our customer and as a pre-requisite we are following the instructions of this document:https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-lan-wlan/213317-understanding-various-ap-ios-flash-corru.html#anc22

The python/pip are successfully installed, edited the config.ini to match our information and gave it a shot.

After a while, the results are printed but It seems that we are facing a fundamental error for all APS:

<snip>

Errors
============================================================
AP connection error: Telnet instance has no attribute 'get_transport': 92

AP connection error: timed out : 15
AP connection error: telnet connection closed: 2
============================================================

Has anyone faced the same error? Any tips/ideas ?

To note, we are using windows server 2012-64bit instead of the Win10 purely because this is the only available server to access the management network of the WLC. Also, telnet client is enabled on the Windows Server.

Thank you

Panayiotis

[Ric Beeching]

Hi,

1) Can the windows server telnet to the AP from command prompt? Just to "telnet <ip>" and see if it can get there. Or use putty to test.
2) Are the APs configured globally to accept telnet connections?
3) Can you telnet from another device to the AP and login with the same credentials in the config.ini?
4) Is a firewall potentially blocking the connection either locally on the win server or on between the server and firewall?

Ric

[panayiotiscy]

Hello Ric,

Yes, thats doable..something else must be causing this behavior but i cant tell for sure...

thanks

[Leo Laohoo]

This bug is fixed in 8.5.140.0. What firmware is the controller running on?

[panayiotiscy]

Hello Leo,

Its actually an old one, under 8.2.x, thats why we would like to run the script and then proceed with the upgrade.

Does the script support the 8.2.x releases?

Thanks

[patoberli]

Yes it should. Please note, the script tries to login via ssh or telnet (I think you must configure this first) to each AP and then run some tests. After that you have to restart the script again and it runs some other tests. I can't remember if I run it a third time in total, but I think running it twice is enough.
The testing server should be in the same VLAN as the AP management interface. Make sure telnet/ssh is enabled on the APs (by default it's disabled) and also make sure that the user/pass you configured in the script is identical to the one you have set for the AP management.

And most importantly, use Python 2.x and not version 3. Scripts written for 2.x aren't compatible with 3.x.

[panayiotiscy]

Hello Patobeli,

The server is indeed in the same vlan as the management vlan of the WLC, not the APs though...

I dont think its a telnet/ssh problem, the credentials are correct.

Please see the log output for a specific ap (all the other APs have the same log )

<snip>

2019-04-08 21:37:26,867 wlanpoller INFO: WLC processing complete; waiting for AP data collection to complete
2019-04-08 21:37:29,253 wlanpoller INFO: KXXXX : AIR-CAP1602I-E-K9 - Connected - IP addr: a.b.c.d
2019-04-08 21:37:29,253 wlanpoller INFO: KXXXX : AIR-CAP1602I-E-K9 - Collecting poller output
..
..
..
2019-04-08 21:38:18,674 wlanpoller CRITICAL: KXXXX : AIR-CAP1602I-E-K9 - Connection error (telnet://a.b.c.d) Telnet instance has no attribute 'get_transport'

<snip>

What do you think?

Thanks

[Leo Laohoo]

Yes it does but you'll need to run the script regularly, as in DAILY.

[panayiotiscy]

Hi Leo,

Is there any way to enable additional debugging information for the script?

The failure reason doesnt make any sense and i dont really know how to troubleshoot this further.

Thanks

[Leo Laohoo]

How many APs do you have? And have you seen behaviours where the APs boot up with corrupted IOS?

[panayiotiscy]

Hello Leo,

We have  about 500 Access points in total.

Access points failures are not frequent and of course not sure even if the low number of failures is associated with the flash corruption issue.

We are just eager to run the script, identify the candidate Access Points to fail  and proceed with the WLC upgrade.

But, the script fails so we are in the dark.

[patoberli]

Can you try, from your script client, to telnet into every single AP by hand (or at least some) and see if the connection and enable works?

That must work, only then the script will also work.

[panayiotiscy]

Hello Patoberli,

checks performed:

from server to wlc:

the script logs-in and retrieves the AP list successfully

from server to APs:

using native telnet on Windows server we can telnet to the AP(s) and log in using the credentials defined in config.ini

I think that this is also depicted below but it doesnt make sense because the script initially reports that its connecting to the AP(s) but then it reports error in connecting using telnet:

snip>

2019-04-08 21:37:26,867 wlanpoller INFO: WLC processing complete; waiting for AP data collection to complete
2019-04-08 21:37:29,253 wlanpoller INFO: KXXXX : AIR-CAP1602I-E-K9 - Connected - IP addr: a.b.c.d
2019-04-08 21:37:29,253 wlanpoller INFO: KXXXX : AIR-CAP1602I-E-K9 - Collecting poller output
..
..
..
2019-04-08 21:38:18,674 wlanpoller CRITICAL: KXXXX : AIR-CAP1602I-E-K9 - Connection error (telnet://a.b.c.d) Telnet instance has no attribute 'get_transport'

<snip>

The way i interpret the Critical error is that the script cant allocate the necessary telnet resource as such it fails, but this is only my understanding...

Thanks 

[patoberli]

Which version of Python is installed and is it configured properly in the PATH variable?

[panayiotiscy]

Hello Patoberli,

i am attaching the relevant output for pip/ path details.