cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
6024
Views
5
Helpful
5
Replies

APs show IP 0.0.0.0 and cannot be edited

mario.jost
Level 3
Level 3

Dear community

As we approach the number of 250 devices, we have some strange phenomena. Some APs are shown with IP 0.0.0.0 on the WLC.

09-08-_2018_15-21-52.jpg

Lets have a closer look onto apPWF05. It is reachable via ping, can be resolved via DNS as any other access points. CDP neighbor on the switch confirms the same ip:

Device ID: apPWF05
Entry address(es): 
  IP address: 172.16.229.32
  IPv6 address: FE80::AA9D:21FF:FE80:48B3  (link-local)
Platform: cisco AIR-CAP2602I-E-K9,  Capabilities: Trans-Bridge Source-Route-Bridge IGMP 
Interface: GigabitEthernet1/0/4,  Port ID (outgoing port): GigabitEthernet0
Holdtime : 167 sec

Version :
Cisco IOS Software, C2600 Software (AP3G2-K9W8-M), Version 15.3(3)JD4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Fri 17-Mar-17 12:36 by prod_rel_team

advertisement version: 2
Duplex: full
Power drawn: 15.400 Watts
Power request id: 33182, Power management id: 2
Power request levels are:15400 13000 0 0 0 
Management address(es): 
  IP address: 172.16.229.32

I can even connect to the AP via SSH and login. If i reboot the access point, nothing changes with this situation. I had the same with the AP below that one. I went ahead and did a factory reset by reloading without saving the configuration. These 2 access points now seem to be hopping on and off the WLC. If i reload the Wireless -> All APs list, the count constantly switches between 248, 249 and 250. If i wanna click on one of the APs, sometimes it gives me following message:

AP doesn't exist anymore on the system. 

And if i manage to get into the settings of these APs, after changing the name or any other setting for that matter, i get following message when clicking on save:

The AP name cannot exceed 32 characters.
Error in setting AP Location
Error in setting AP Statistics Timer
Primary Controller : Unable to set controller name.
Secondary Controller : Unable to set controller name.
Tertiary Controller : Unable to set controller name.
Failed to update Link Latency
Request Failed. Disable AP before configuring country

So this just underlines the "flapping" mentioned above. We use a vWLC and are running on 8.5.120.0 since around 3 months now. We had this issue on one AP since a few weeks already, but didnt have a closer look. Yesterday, i installed a new access point in another branch office, now we have 2 of these 0.0.0.0 IP APs. It seems like there is a limitation of APs on the WLC. We installed the large image of the vWLC that should be able to handle 3000 APs. I read somewhere in the forum, that we can increase the number of APs with config ap max-count <number> command. Sadly, this command is not recognized by the WLC. But i found the command show ap max and that gives me this:

(Cisco Controller) >show ap max
Max APs Supported................................ 3000
Max AP Groups Supported.......................... 3000
Max AP join limit................................ 300

(Cisco Controller) >show license capacity
Licensed Feature    Max Count         Current Count     Remaining Count     Remaining Capacity
----------------------------------------------------------------------------------------------
AP Count            300               249               51                  2751/3000

Just adding the license situation in case this question comes up. I couldnt find any limitation of 250 in the open caveats on this firmware verison. Anyone having any ideas of what i can try to resolve this situation?

 

Any help is greatly appreciated. Thanks.

5 Replies 5

mario.jost
Level 3
Level 3

I managed to access the logs of one access point and it seems that the flash of the device is defective:

*Aug  9 04:26:51.891: %CAPWAP-6-AP_IMG_DWNLD: Required image not found on AP. Downloading image from Controller.
*Aug  9 04:26:51.891: capwap_image_proc: encounter flash problem, retry here
Write of the Private File nvram:/lwapp_ap.cfg Failed
*Aug  9 04:29:01.303: 
Write of the Private File nvram:/lwapp_ap.cfg Failed
*Aug  9 04:29:01.303: 
Write of the Private File nvram:/lwapp_ap.cfg Failed
*Aug  9 04:29:01.303: %LWAPP-3-CLIENTERRORLOG: Save LWAPP Config: error saving config file

apPWF05#show flash:
Directory of flash:/
%Error opening flash:/ (Invalid argument)
No space information available

apPWF05#show file systems
File Systems:

       Size(b)       Free(b)      Type  Flags  Prefixes
*            -             -     flash     rw   flash:
             -             -    opaque     rw   arch:
      11999232       9580544     flash     rw   ram:
             -             -    opaque     rw   bs:
             -             -   unknown     rw   zflash:
             -             -    opaque     rw   archive:
             -             -    opaque     rw   system:
         32768          5068     nvram     rw   nvram:
             -             -    opaque     rw   tmpsys:
             -             -   network     rw   tftp:
             -             -    opaque     rw   null:
             -             -   network     ro   capwap:
             -             -   network     ro   lwapp:
             -             -   network     rw   rcp:
             -             -   network     rw   http:
             -             -   network     rw   scp:
             -             -   network     rw   ftp:
             -             -    opaque     ro   tar:
             -             -   network     rw   https:

I testwise added another AP to check if a 3rd AP is having the 0.0.0.0 issue, but this didnt happen. So i was just put off by the timing of the second AP having this problem at the time i added another AP to the system. I will replace these APs now.

I am very familiar with what is going on with the APs.  

What firmware is the controller running on?  I've seen these behaviour (there are several of them) since the introduction of 8.X.X.X.  This issue is due to the corruption of a few files (mostly the power table) which causes the APs to go into a boot-crash-loop.   Here are some of the issues I've observed: 

 

  • APs with default AP name (and location) but have AP group details. 
  • APs with the primary firmware different to the firmware of the controller. 
  • APs showing with 0.0.0.0 as an IP address for hours.  
  • APs in boot-crash-loop would only show up on the switch with "Ieee" with the command of "sh power inline".  TDR will show all cable pairs as good and interface traffic will show only one-way traffic (output traffic from the switch port to the AP but no return traffic).

These are known issues and I don't have Bug IDs. 

The workaround is to force the AP to re-download the CAPWAP image.  This can be done by console or remote into the AP and forcefully remove the CAPWAP directory and force the AP to load the "RCV" file.  

If you have the APs in front of you, I'd recommend formating the flash and then upload the RCV file.  

Permanent fix is to upgrade the firmware of the controller.

Totally agree with Leo - there are a number of bugs which cause this flash corruption in the 8.x code. They're all supposed to be fixed in the latest releases though 8.0 still has some minor issues. Upgrade to latest release on WLC to pick up those fixes and fix the APs as Leo suggests. Just be warned that the very latest release of code has introduced https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvj32563 which prevents upload of anything (code, banner etc) to the WLC while any AP is trying to join and it can get stuck in that state requiring to disable AP management (disconnect all APs) in order to start any upload to the WLC. So if you upgrade to an affected release your next attempt at upgrade becomes a nightmare!
Rich

Thanks for your answers. As written on my first post, we are running on 8.5.120.0... I will plan to upgrade to 8.5.135 next week. Or is there any other firmware you would suggest at the current time? Remember, we use virtual WLC. We are running at an 8.x release since around 2 years and never had mentioned issue. So maybe this is something that only shows after long term use.


@mario.jost wrote:

Thanks for your answers. As written on my first post, we are running on 8.5.120.0... I will plan to upgrade to 8.5.135 next week. 


I haven't tried 8.5.135.0, yet.  


@mario.jost wrote:

So maybe this is something that only shows after long term use.


Depends on the firmware.  Some shows up within 24 hours and some weeks and weeks later.

One of the ways I find out when the power table is corrupt is (either in ROMmon or IOS) list the file directory of the IOS.  If there is a file with "0" size, then I can guarantee the bug is present.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Review Cisco Networking products for a $25 gift card