Ap won't accept backup or primary image

[theophile]

Hi,

Trying to predownload firmware 7.6.130.0 on APs, I've a bunch of AP on which it doesn't work (neither to the backup image nor the primary one). The status is failed. The wlc is running 7.4.130 as the active image and 7.6.130.0 when it will be rebooted . AP model is 2600. When I look at the ap's log, here is what I see:

*Oct  1 13:39:57.343: %CAPWAP-3-ERRORLOG: not receive read response(0)
*Oct  1 14:06:47.971: %CAPWAP-3-ERRORLOG: AC rejected image data request
*Oct  1 14:08:07.967: %CAPWAP-3-ERRORLOG: not receive read response(0)
*Oct  1 14:12:25.971: %CAPWAP-3-ERRORLOG: AC rejected image data request
*Oct  1 14:13:45.967: %CAPWAP-3-ERRORLOG: not receive read response(0)
*Oct  1 14:23:37.971: %CAPWAP-3-ERRORLOG: AC rejected image data request
*Oct  1 14:24:57.971: %CAPWAP-3-ERRORLOG: not receive read response(0)
*Oct  1 14:34:34.975: %CAPWAP-3-ERRORLOG: AC rejected image data request
*Oct  1 14:35:54.971: %CAPWAP-3-ERRORLOG: not receive read response(0)
*Oct  1 14:39:03.975: %CAPWAP-3-ERRORLOG: AC rejected image data request
*Oct  1 14:40:23.947: %CAPWAP-3-ERRORLOG: not receive read response(0)

Any advices will be really appreciated!

Theo

[Blazej Pawlak]

Enter AP via console, probably your AP flash is full, clear everything* from flash (predwonlaod images, (not the working one), unesscasy files, etc). Reboot controller, AP should download version working on WLC and reimage itself. If this wont start apart from predownload .tar file from controller clear flash again then use command archive to unpack the image. 

[theophile]

Hi BBP634940, thanks for your reply.

I thought at first it was a flash size's problem but after checking it is fine. i've also performed a 'fsck' with no errors detected. To be sure, I also use a brand new AP, right out of the box (2600i). Same problem with 1040/1600...

This morning, I've done some debug on the wlc's side and here is the result:

*spamApTask7: Oct 02 10:19:49.757: 24:e9:b3:42:31:d0 Refusing Discovery Request from AP 24:e9:b3:42:31:d0 - limit for maximum AP downloads 800 reached

What's strange is that the licence installed on the wlc is for 800 APs and there are only 754 connected to it, so we're not overrunning...

Cheers

[Blazej Pawlak]

Clear AP join table.

What is yours HA deployment? (With this amount I believe you have one implemented)

Can you run show sysinfo on your WLC? Those yours WLC have latest FUS image installed?

[theophile]

ok I just clear the join table

Yes I've a HA deployment and I hope the standbye wlc gets its image from the active one but no way to verify...!

Here is the sysinfo:

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.4.121.0
RTOS Version..................................... 7.4.121.0
Bootloader Version............................... 7.3.101.0
Emergency Image Version.......................... 7.3.101.0

Build Type....................................... DATA + WPS

System Name...................................... ***********
System Location.................................. ***********
System Contact................................... ***********
System ObjectID.................................. 1.3.6.1.4.1.9.1.1615
Redundancy Mode.................................. AP SSO
IP Address....................................... ***********
System Up Time................................... 163 days 7 hrs 24 mins 32 secs
System Timezone Location.........................
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... FR  - France

--More-- or (q)uit
Operating Environment............................ Commercial (10 to 35 C)
Internal Temp Alarm Limits....................... 10 to 38 C
Internal Temperature............................. +26 C
Fan Status....................................... OK
RAID Volume Status............................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 11
Number of Active Clients......................... 3756

Memory Current Usage............................. Unknown
Memory Average Usage............................. Unknown
CPU Current Usage................................ Unknown
CPU Average Usage................................ Unknown

Burned-in MAC Address............................ ***********
Power Supply 1................................... Present, OK
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 800

[Blazej Pawlak]

For redundancy summary go with

show redundancy summary 

as it goes for license count go with CLI command
 

show license summary

Clear ap Join table Stats 

clear ap join stats

This will clear statistic so you can make sure that only controller by you APs are joining controller

Go with this Guide for HA Image Update

https://supportforums.cisco.com/document/136826/wlc-high-availability-ap-sso-guidelines

You can always get to secondary WLC via SSH

Also regarding predownloading here are some hints 

http://www.my80211.com/home/2011/2/20/wlc-predownload-the-image-to-the-access-points-from-the-cont.html

You have probably seen this message when Other APs has downloaded image.

[theophile]

Thanks Blazej,

I know for the hints concerning the predownloading. In my case all downloads are over, so I'm not sure my problem is related to that.

I've done the clear join command

But I still have the problem, every APs I try to attach to the wlc got the same error message (*spamApTask7: Oct 02 10:19:49.757: 24:e9:b3:42:31:d0 Refusing Discovery Request from AP 24:e9:b3:42:31:d0 - limit for maximum AP downloads 800 reached)

Really strange!

[Blazej Pawlak]

Hello Theophile

It could be some unknown bug on WLC software that is related to count of downloaded AP. Try to restart the controller this should clear some memory on WLC. 

[theophile]

I've planned to reboot the wlc tomorrow. I will then let you know the result

[theophile]

We've rebooted the wlc yesterday and since then, all new APs are now able to join it. But this works only when we boot on the 7.4 firmware. No APs are able to join if we boot the wlc using the 7.6 firmware, really strange.

I'm now searching for a solution to fix this 7.6 problem.

Cheers

Theo

[holger.moersfelder@global.ntt]

Hi Theo,

we are hitting a simulare issue. Upgrading from 7.4.110.0 to 7.6.130.0 is intended. The pre-image download failed for 50% of the 3600 APs.

HA-Mode is actively used.

I found out that APs failed where the mini IOS is on SW 7.3.1.73.

Other APs with Mini IOS 7.3.1.53 passed image predownloading.

We will try to find out the reason of this synthome before the whole system will be reloaded.

Could you please check the mini ios version of your APs for verification ?

Regards Holger
 

[theophile]

Hi Holger,

our 3600s are running Mini IOS Version 7.3.1.53

From what I found until today, the predownload failed because of insuffisant flash size on the AP to keep both running and backup firmware...

Regards

[Leo Laohoo]

What regulatory domain is your AP purchased for? 

[theophile]

Hello Leo, the regulatory domain is Europe. These APs were correctly attached to the wlc before the pre-download job. This job has failed and since then no way to re-attach them to the same wlc!

[Scott Fella]

Just to add... Pre-download doesn't work with the 2602's, but the other AP's should be fine. Are the AP's joining the WLC and getting the image for v7.4.121.0?  I'm assuming they are since you are trying to issue the pre-download. Are you also globally issuing the pre-download or individual access points?

Scott