There is a manufacturing calibration issue on AP 3500 and AP 1260 (VID V01). It is addressed in manufacturing via VID V02.
The calibration problem can cause random memory corruption, which in turn leads to a variety of symptoms.
This is Field Notice FN63537.
This problem may affect any 1260 and 3500 series AP for which both of the following conditions are true:
A 1260 or 3500 series access point may exhibit one or more of the following symptoms:
Unexpected exception to CPUvector 700, PC = 0x5F784 , LR = 0x1A5524
-Traceback= 0x5F784 0x1A5524 0x1A56F0 0x177AE4 0x276F88 0x1A5A70
Unexpected exception to CPUvector 200, PC = 0x237CB0 , LR = 0x237C94
-Traceback= 0x237CB0 0x237D18 0x1F2824 0x51CC30 0x51B898 0x508E5C 0x511794
0x511B7C 0x503B44 0x51C49C 0x51CB30 0x526378 0x530184 0x52FD0C 0x5DA6AC 0x584C4C
reset_interrupt_level: reverse reset level current level 4 new level 1174028828
Unexpected exception to CPUvector 700, PC = 0x1A19C8 , LR = 0x1A1918
-Traceback= 0x1A19C8 0x1A1918 0x5DEB84 0x5DEEA4 0x5DEF54 0x5DF0CC 0x1A5A70
In a majority of instances when an AP is down with a white LED and there is no console access, physically rebooting the AP by toggling the switch port does recover it.
A new AP IOS image will be provided with new bootloader/uboot code. When IOS boots, it will check the bootloader version that is installed - if it does not have the fix (i.e. either an AP which has not undergone this process or new APs purchased before June 2012, prior to the bootloader change in manufacturing), then the following will happen:
The bootloader/uboot/calibration upgrade procedure will run only the first time the AP reboots after loading the new IOS. This upgrade is a one-time process, which will take about 7 minutes. The new DDR values shall remain even if the AP is upgraded/downgraded to any different code version.
For autonomous APs, the fix will be provided in an normal IOS tar image ("tarball"). For lightweight APs, the fix will be provided in AP IOS, bundled in wireless LAN controller images.
The fixed autonomous IOS and wireless LAN controller images are now available on CCO:
For unified deployments, there are three options:
After the download, the primary image will be the newly loaded fixed image, and the original image will be the backup (as seen with "show boot".) Use the command "config boot backup" to make sure that the original image is the active one:
(2106-1) >config boot backup
(2106-1) >show boot
Primary Boot Image............................... 184.108.40.206
Backup Boot Image................................ 220.127.116.11 (default) (active)
Issue "config ap image predownload primary all" to push the special IOS image to all APs.
Verify via "show ap image all" that all APs now have both the original and the fixed IOS image. This may take from several minutes to several hours depending on the number of APs.
Issue "config ap image swap all" to set the AP's BOOT variable to the special IOS image
Now reboot the 3500/1260 APs via any desired method.
The APs will boot the special IOS image, which will upgrade the bootloader, run the DDR calibration, boot the fixed IOS image again, join the controller. The controller and the AP figure out that the right (original) image is on the AP flash, so the AP sets its BOOT variable to the original IOS image, reboots with the original IOS image, and joins the controller.
For upgrading autonomous IOS deployments:
The new AP tarball contains the following three files:
1. The present BS: bootloader that was created at FCS of AP3500/1260 will call a new RAM-based bootloader.
2. LED will glow AMBER. The RAM-based “secondary bootloader” contains all of the upgrade logic, and will do the one time copy of u-boot to BB:, and the one time copy of the new BS: bootloader to BS:.
3. LED will turn TEAL. This denotes the completion of step (2) and the AP will reset.
4. LED will glow PINK. The new BS: bootloader will jump to u-boot after the reset. This will calibrate the DDR and reset the AP.
5. LED will turn GREEN, as it usually does during the normal boot up operation. The BS: bootloader will use the new DDR values, and boot the RAM-based bootloader. The RAM-based bootloader will boot IOS and continue its normal operation after joining the controller.
6. If the system is ever downgraded or upgraded and the RAM-based bootloader disappears, the ROM-based BS: bootloader will boot IOS. The new DDR values are still used.
To summarize, the sequence of LEDs during the upgrade process shall be - AMBER- TEAL- <AP reset> - PINK- <AP reset> - GREEN.
Sample console logs during the entire operation can be found here: https://supportforums.cisco.com/docs/DOC-25461
The original AP3500/1260 Cisco Bootloader is overwritten. This step takes 5-10 seconds. It is important power is not removed during this operation. This is the only time the AP is at risk of becoming a “brick”.
Sanity Check - The easiest way to verify is to check the bootloader version on the APs via "show version" once they have been upgraded.
What is the increase in size of the AP tarball due to the addition of these files?
About 800kb on top of the normal IOS tarball.
How much time does the entire process take?
The entire process after the AP's have downloaded the new tarball should take under 10 minutes.
As described above it involves the following steps:- Copy bootloader - Copy u-boot - Reset - Perform Calibration - Reset - Boot IOS and join the controller via CAPWAP.
What if the controller or IOS image is downgraded or upgraded in the future?
Since the current bootloader is replaced with the new bootloader, the fix shall persist no matter what run time image is used. This is just an one time calibration process.
What happens to APs which have already been calibrated either via the new image or new bootloader from manufacturing?
There is a check at the very beginning for both those conditions and hence the entire new upgrade and calibration logic will be skipped. The AP will incur no additional delay during its bootup if it already has the new calibration.
What is the probability of APs becoming "brick" during the upgrade process?
The probability is low, since this can only occur when the new bootloader is being copied to flash. If at that time power to the AP is lost, it will not recover. It is just a 5 - 10 second window when this can occur if there any anomaly in the PoE source.
Apart from copying of the new bootloader, what happens if any other part of the new calibration/upgrade is skipped or fails?
If any other part of the calibration/upgrade is skipped or fails, the AP will resume its normal operation and continue to use the old calibration value. It won't be stuck in a loop of trying to re-calibrate itself.
How does a boot-looping access-point get the new boot image?
Unless the flash has a problem, AP should be able to stay up long enough to load the image. If the AP is bricked or continuously looping, it will need to be RMA.