New 3802i unable to join WLC 2504, green and red flashing light.

[todorn123]

Hello,

Attached is complete console log from AP. WLC 2504 Software version is 8.5.171.0. I suspect issue is AP unable to get DHCP but there is no apparent reason for it and I see no DHCP logs in the console output. I reset the APs to defaults multiple times without any luck. Don't see any issues with POE on switch ports. I have other 15APs connected and joined without any issues a month or so ago, but models are 2702s and 3702s. All APs are on vlan1, no ip helpers, cisco 2960-X switches and Windows DHCP server. 

[*06/08/2018 00:53:27.1017]
[*06/08/2018 00:53:32.1021] waiting for POE negotiation to complete
[*06/08/2018 00:53:32.1021]
[*06/08/2018 00:53:37.1026] waiting for POE negotiation to complete
[*06/08/2018 00:53:37.1026]
[*06/08/2018 00:53:42.1031] waiting for POE negotiation to complete
[*06/08/2018 00:53:42.1031]
[*06/08/2018 00:53:47.1035] waiting for POE negotiation to complete
[*06/08/2018 00:53:47.1035]
[*06/08/2018 00:53:52.1040] waiting for POE negotiation to complete
[*06/08/2018 00:53:52.1040]
[*06/08/2018 00:53:57.1044] waiting for POE negotiation to complete
[*06/08/2018 00:53:57.1044]
[*06/08/2018 00:54:01.7434] SYS-COND: AP is in fair condition
[*06/08/2018 00:54:02.1051] Waiting for uplink IPv4/IPv6 configuration
[*06/08/2018 00:54:07.1059] Waiting for uplink IPv4/IPv6 configuration
[*06/08/2018 00:54:12.1068] Waiting for uplink IPv4/IPv6 configuration
[*06/08/2018 00:54:17.1075] Resetting wired0, if[06/08/2018 00:54:17.1400] wired0: stopped
AP6CB2.AE88.88A8#[06/08/2018 00:55:25.7500] wired0: stopped
[06/08/2018 00:55:27.8400] wired0: started
[06/08/2018 00:55:42.9000] wired0: stopped
[06/08/2018 00:55:44.9900] wired0: started
[06/08/2018 00:56:00.0800] wired0: stopped

[ammahend]

are all 15 APs connected to same switch ? share show power inline, show inventory from switch.

show sysinfo from 2504

[todorn123]

Below are outputs of the commands requested. One 3802 AP with the issue is connected to Gi2/0/27, however another one is just fine on Gi2/0/25 joined a week or so ago. 

RICCSSWI015#show power inline

Module Available Used Remaining
(Watts) (Watts) (Watts)
------ --------- -------- ---------
1 740.0 168.0 572.0
2 740.0 50.3 689.7
3 740.0 15.4 724.6
Interface Admin Oper Power Device Class Max
(Watts)
--------- ------ ---------- ------- ------------------- ----- ----
Gi1/0/1 auto on 16.8 AIR-CAP3702I-A-K9 4 30.0
Gi1/0/2 auto off 0.0 n/a n/a 30.0
Gi1/0/3 auto on 16.8 AIR-CAP2702I-B-K9 4 30.0
Gi1/0/4 auto off 0.0 n/a n/a 30.0
Gi1/0/5 auto on 16.8 AIR-CAP2702I-B-K9 4 30.0
Gi1/0/6 auto off 0.0 n/a n/a 30.0
Gi1/0/7 auto on 16.8 AIR-CAP2702I-B-K9 4 30.0
Gi1/0/8 auto off 0.0 n/a n/a 30.0
Gi1/0/9 auto on 16.8 AIR-CAP3702I-A-K9 4 30.0
Gi1/0/10 auto off 0.0 n/a n/a 30.0
Gi1/0/11 auto on 16.8 AIR-CAP3702I-A-K9 4 30.0
Gi1/0/12 auto off 0.0 n/a n/a 30.0
Gi1/0/13 auto on 16.8 AIR-CAP3702I-A-K9 4 30.0
Gi1/0/14 auto off 0.0 n/a n/a 30.0
Gi1/0/15 auto on 16.8 AIR-CAP3702I-A-K9 4 30.0
Gi1/0/16 auto off 0.0 n/a n/a 30.0
Gi1/0/17 auto on 16.8 AIR-CAP3702I-A-K9 4 30.0
Gi1/0/18 auto off 0.0 n/a n/a 30.0
Gi1/0/19 auto on 16.8 AIR-CAP3702I-A-K9 4 30.0
Gi1/0/20 auto off 0.0 n/a n/a 30.0
Gi1/0/21 auto off 0.0 n/a n/a 30.0
Gi1/0/22 auto off 0.0 n/a n/a 30.0
Gi1/0/23 auto off 0.0 n/a n/a 30.0
Gi1/0/24 auto off 0.0 n/a n/a 30.0
Gi1/0/25 auto off 0.0 n/a n/a 30.0
Gi1/0/26 auto off 0.0 n/a n/a 30.0
Gi1/0/27 auto off 0.0 n/a n/a 30.0
Gi1/0/28 auto off 0.0 n/a n/a 30.0
Gi1/0/29 auto off 0.0 n/a n/a 30.0
Gi1/0/30 auto off 0.0 n/a n/a 30.0
Gi1/0/31 auto off 0.0 n/a n/a 30.0
Gi1/0/32 auto off 0.0 n/a n/a 30.0
Gi1/0/33 auto off 0.0 n/a n/a 30.0
Gi1/0/34 auto off 0.0 n/a n/a 30.0
Gi1/0/35 auto off 0.0 n/a n/a 30.0
Gi1/0/36 auto off 0.0 n/a n/a 30.0
Gi1/0/37 auto off 0.0 n/a n/a 30.0
Gi1/0/38 auto off 0.0 n/a n/a 30.0
Gi1/0/39 auto off 0.0 n/a n/a 30.0
Gi1/0/40 auto off 0.0 n/a n/a 30.0
Gi1/0/41 auto off 0.0 n/a n/a 30.0
Gi1/0/42 auto off 0.0 n/a n/a 30.0
Gi1/0/43 auto off 0.0 n/a n/a 30.0
Gi1/0/44 auto off 0.0 n/a n/a 30.0
Gi1/0/45 auto off 0.0 n/a n/a 30.0
Gi1/0/46 auto off 0.0 n/a n/a 30.0
Gi1/0/47 auto off 0.0 n/a n/a 30.0
Gi1/0/48 auto off 0.0 n/a n/a 30.0
Gi2/0/1 auto off 0.0 n/a n/a 30.0
Gi2/0/2 auto off 0.0 n/a n/a 30.0
Interface Admin Oper Power Device Class Max
(Watts)
--------- ------ ---------- ------- ------------------- ----- ----
Gi2/0/3 auto off 0.0 n/a n/a 30.0
Gi2/0/4 auto off 0.0 n/a n/a 30.0
Gi2/0/5 auto off 0.0 n/a n/a 30.0
Gi2/0/6 auto on 5.0 Polycom VVX 411 3 30.0
Gi2/0/7 auto off 0.0 n/a n/a 30.0
Gi2/0/8 auto off 0.0 n/a n/a 30.0
Gi2/0/9 auto off 0.0 n/a n/a 30.0
Gi2/0/10 auto off 0.0 n/a n/a 30.0
Gi2/0/11 auto off 0.0 n/a n/a 30.0
Gi2/0/12 auto off 0.0 n/a n/a 30.0
Gi2/0/13 auto off 0.0 n/a n/a 30.0
Gi2/0/14 auto off 0.0 n/a n/a 30.0
Gi2/0/15 auto off 0.0 n/a n/a 30.0
Gi2/0/16 auto off 0.0 n/a n/a 30.0
Gi2/0/17 auto off 0.0 n/a n/a 30.0
Gi2/0/18 auto off 0.0 n/a n/a 30.0
Gi2/0/19 auto off 0.0 n/a n/a 30.0
Gi2/0/20 auto off 0.0 n/a n/a 30.0
Gi2/0/21 auto off 0.0 n/a n/a 30.0
Gi2/0/22 auto off 0.0 n/a n/a 30.0
Gi2/0/23 auto off 0.0 n/a n/a 30.0
Gi2/0/24 auto off 0.0 n/a n/a 30.0
Gi2/0/25 auto on 29.9 AIR-AP3802I-B-K9 4 30.0
Gi2/0/26 auto off 0.0 n/a n/a 30.0
Gi2/0/27 auto on 15.4 Ieee PD 4 30.0
Gi2/0/28 auto off 0.0 n/a n/a 30.0
Gi2/0/29 auto off 0.0 n/a n/a 30.0
Gi2/0/30 auto off 0.0 n/a n/a 30.0
Gi2/0/31 auto off 0.0 n/a n/a 30.0
Gi2/0/32 auto off 0.0 n/a n/a 30.0
Gi2/0/33 auto off 0.0 n/a n/a 30.0
Gi2/0/34 auto off 0.0 n/a n/a 30.0
Gi2/0/35 auto off 0.0 n/a n/a 30.0
Gi2/0/36 auto off 0.0 n/a n/a 30.0
Gi2/0/37 auto off 0.0 n/a n/a 30.0
Gi2/0/38 auto off 0.0 n/a n/a 30.0
Gi2/0/39 auto off 0.0 n/a n/a 30.0
Gi2/0/40 auto off 0.0 n/a n/a 30.0
Gi2/0/41 auto off 0.0 n/a n/a 30.0
Gi2/0/42 auto off 0.0 n/a n/a 30.0
Gi2/0/43 auto off 0.0 n/a n/a 30.0
Gi2/0/44 auto off 0.0 n/a n/a 30.0
Gi2/0/45 auto off 0.0 n/a n/a 30.0
Gi2/0/46 auto off 0.0 n/a n/a 30.0
Gi2/0/47 auto off 0.0 n/a n/a 30.0
Gi2/0/48 auto off 0.0 n/a n/a 30.0
Gi3/0/1 auto off 0.0 n/a n/a 30.0
Gi3/0/2 auto off 0.0 n/a n/a 30.0
Gi3/0/3 auto off 0.0 n/a n/a 30.0
Gi3/0/4 auto off 0.0 n/a n/a 30.0
Gi3/0/5 auto off 0.0 n/a n/a 30.0
Gi3/0/6 auto on 15.4 Ieee PD 0 30.0
Gi3/0/7 auto off 0.0 n/a n/a 30.0
Gi3/0/8 auto off 0.0 n/a n/a 30.0
Gi3/0/9 auto off 0.0 n/a n/a 30.0
Gi3/0/10 auto off 0.0 n/a n/a 30.0
Gi3/0/11 auto off 0.0 n/a n/a 30.0
Gi3/0/12 auto off 0.0 n/a n/a 30.0
Gi3/0/13 auto off 0.0 n/a n/a 30.0
Gi3/0/14 auto off 0.0 n/a n/a 30.0
Interface Admin Oper Power Device Class Max
(Watts)
--------- ------ ---------- ------- ------------------- ----- ----
Gi3/0/15 auto off 0.0 n/a n/a 30.0
Gi3/0/16 auto off 0.0 n/a n/a 30.0
Gi3/0/17 auto off 0.0 n/a n/a 30.0
Gi3/0/18 auto off 0.0 n/a n/a 30.0
Gi3/0/19 auto off 0.0 n/a n/a 30.0
Gi3/0/20 auto off 0.0 n/a n/a 30.0
Gi3/0/21 auto off 0.0 n/a n/a 30.0
Gi3/0/22 auto off 0.0 n/a n/a 30.0
Gi3/0/23 auto off 0.0 n/a n/a 30.0
Gi3/0/24 auto off 0.0 n/a n/a 30.0
Gi3/0/25 auto off 0.0 n/a n/a 30.0
Gi3/0/26 auto off 0.0 n/a n/a 30.0
Gi3/0/27 auto off 0.0 n/a n/a 30.0
Gi3/0/28 auto off 0.0 n/a n/a 30.0
Gi3/0/29 auto off 0.0 n/a n/a 30.0
Gi3/0/30 auto off 0.0 n/a n/a 30.0
Gi3/0/31 auto off 0.0 n/a n/a 30.0
Gi3/0/32 auto off 0.0 n/a n/a 30.0
Gi3/0/33 auto off 0.0 n/a n/a 30.0
Gi3/0/34 auto off 0.0 n/a n/a 30.0
Gi3/0/35 auto off 0.0 n/a n/a 30.0
Gi3/0/36 auto off 0.0 n/a n/a 30.0
Gi3/0/37 auto off 0.0 n/a n/a 30.0
Gi3/0/38 auto off 0.0 n/a n/a 30.0
Gi3/0/39 auto off 0.0 n/a n/a 30.0
Gi3/0/40 auto off 0.0 n/a n/a 30.0
Gi3/0/41 auto off 0.0 n/a n/a 30.0
Gi3/0/42 auto off 0.0 n/a n/a 30.0
Gi3/0/43 auto off 0.0 n/a n/a 30.0
Gi3/0/44 auto off 0.0 n/a n/a 30.0
Gi3/0/45 auto off 0.0 n/a n/a 30.0
Gi3/0/46 auto off 0.0 n/a n/a 30.0
Gi3/0/47 auto off 0.0 n/a n/a 30.0
Gi3/0/48 auto off 0.0 n/a n/a 30.0

RICCSSWI015#show inventory
NAME: "1", DESCR: "WS-C2960X-48FPD-L"
PID: WS-C2960X-48FPD-L , VID: V01 , SN: FCW1841A83G

NAME: "Switch 1 - FlexStackPlus Module", DESCR: "Stacking Module"
PID: C2960X-STACK , VID: V01 , SN: FOC17404ZEV

NAME: "TenGigabitEthernet1/0/1", DESCR: "SFP-10GBase-SR"
PID: SFP-10G-SR , VID: V03 , SN: AGD1542465A

NAME: "2", DESCR: "WS-C2960X-48FPD-L"
PID: WS-C2960X-48FPD-L , VID: V01 , SN: FCW1838A4TQ

NAME: "Switch 2 - FlexStackPlus Module", DESCR: "Stacking Module"
PID: C2960X-STACK , VID: V01 , SN: FOC201753CW

NAME: "3", DESCR: "WS-C2960X-48FPD-L"
PID: WS-C2960X-48FPD-L , VID: V01 , SN: FCW1846B065

NAME: "Switch 3 - FlexStackPlus Module", DESCR: "Stacking Module"
PID: C2960X-STACK , VID: V01 , SN: FOC21072HGH

NAME: "TenGigabitEthernet3/0/1", DESCR: "SFP-10GBase-SR"
PID: SFP-10G-SR , VID: V03 , SN: FNS1719191T

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.5.171.0
Bootloader Version............................... 1.0.20
Field Recovery Image Version..................... 7.6.101.1
Firmware Version................................. PIC 20.0

OUI File Last Update Time........................ Sun Sep 07 10:44:07 IST 2014

Build Type....................................... DATA + WPS

System Name...................................... RICCWCON01
System Location.................................. Richmond
System Contact................................... Todor Nedyalkov
System ObjectID.................................. 1.3.6.1.4.1.9.1.1279
IP Address....................................... 10.48.150.5
IPv6 Address..................................... ::
Last Reset....................................... Software reset
System Up Time................................... 8 days 20 hrs 58 mins 18 secs
System Timezone Location.........................

--More-- or (q)uit
System Stats Realtime Interval................... 5
System Stats Normal Interval..................... 180

Configured Country............................... US - United States
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +28 C
External Temperature............................. +33 C
Fan Status....................................... 3800 rpm

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 4
Number of Active Clients......................... 28

OUI Classification Failure Count................. 8638

Memory Current Usage............................. 40
Memory Average Usage............................. 40
CPU Current Usage................................ 1
CPU Average Usage................................ 1

Flash Type....................................... Compact Flash Card

--More-- or (q)uit
Flash Size....................................... 1073741824

Burned-in MAC Address............................ BC:C4:93:16:EA:80
Maximum number of APs supported.................. 75
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1/SHA2

[marce1000]

 - (Possibly) : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi14641

 M.

[todorn123]

Thanks for the suggestion. Switch ports for the APs are set on auto full-duplex, showing as connected on 1000M. Unfortunately, I don't have a power injector to try as a workaround.  I checked the serial number from one of the APs and it showed as ''not affected". 

https://www.cisco.com/c/en/us/support/docs/field-notices/701/fn70143.html

[Rich R]

Is CDP enabled on the port?  (the port should have CDP or LLDP enabled but not both)

show run int from Gi2/0/25 and Gi2/0/27 ?

Have you tried swapping the APs on those ports?  Does the problem stay with the port or follow the AP?
Is the AP connected directly to the switch or via patch panels/cables?  In which case try connecting directly to the switch with a known working cable?

Even though the serial number doesn't match I'd still be suspicious that your AP might be affected.  

[todorn123]

Hi,

Yes. APs are connected directly to Cisco 2960-X with brand new cat6 cables 5ft long. Problem stays with the problematic APs.
If I connect a working AP to same port 25 or 27 it boots up just fine and connects to WLC. Also tried different patch cables without any luck. CDP is enabled by default, no LLDP. I try assign a static IP address but nothing happens. I am also very suspicious two APs are affected by this bug. I can't update software on them with TFTP since I can't assign IP addresses to the APs, USB key software is not supported. I feel like my only other option to try buy a power injector and if that doesn't work return to seller. 

AP6CB2.AE88.88A8#capwap ap ip 10.48.20.99 255.255.255.0 10.48.20.1
AP6CB2.AE88.88A8#
AP6CB2.AE88.88A8#show capwap ip config
Sytem is not ready, waiting for uplink IP address.
AP6CB2.AE88.88A8#[06/09/2018 01:38:55.9000] wired0: stopped
[06/09/2018 01:38:57.9900] wired0: started
[06/09/2018 01:39:13.0800] wired0: stopped
[06/09/2018 01:39:15.1500] wired0: started

Switch command outputs below: 

RICCSSWI015#show run interface gi2/0/25
Building configuration...

Current configuration : 133 bytes
!
interface GigabitEthernet2/0/25
description ***Regional Office AP***
switchport trunk native vlan 20
switchport mode trunk
end

RICCSSWI015#show cdp interface gi2/0/25
GigabitEthernet2/0/25 is up, line protocol is up
Encapsulation ARPA
Sending CDP packets every 60 seconds
Holdtime is 180 seconds

RICCSSWI015#show lldp
% LLDP is not enabled

RICCSSWI015#show interfaces gi2/0/25 switchport
Name: Gi2/0/25
Switchport: Enabled
Administrative Mode: trunk
Operational Mode: trunk
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: dot1q
Negotiation of Trunking: On
Access Mode VLAN: 1 (default)
Trunking Native Mode VLAN: 20 (Printer_and_AP )
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL
Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Appliance trust: none

[Arshad Safrulla]

Hi,

Your initial post mentions that all AP's are in VLAN1. But you have configured Native VLAN under the AP connecting switchport? Is the AP is expected to be in Local Mode or FLex connect mode?

If local mode you can configure the port as access port under the AP management VLAN (VLAN 1 or VLAN20), if flex connect current configuration is correct.

You have to make sure that you have a working DHCP server under the same VLAN (VLAN1 or VLAN20) if not you need to configure ip helper under the SVI for the gateway. Please refer the below document.

Configure DHCP OPTION 43 for Lightweight Access Points - Cisco

[todorn123]

Hi,

Yes. All APs are in FlexConnect mode and option 43 is configured. I moved the working APs off of vlan1 to vlan20 yesterday successfully. When I originally posted the question all were on vlan1. It needed to be done anyways and also to see if maybe something strange on valn1 was preventing these two new APs from getting an IP from DHCP server. However, I don't think the issue is DHCP because the AP will not take a static IP address when I assign it with the command below.  Also, attached screenshot show wired0 interface which is the POE/Mgig interface directly connected to switch as down with protocol up? And log messages repeat wired0:started then wired0:stopped...  

AP6CB2.AE88.88A8#capwap ap ip 10.48.20.99 255.255.255.0 10.48.20.1
AP6CB2.AE88.88A8#
AP6CB2.AE88.88A8#show capwap ip config
Sytem is not ready, waiting for uplink IP address.

[Rich R · Accepted solution]

So assuming you've addressed the points @Arshad Safrulla raised then yes it is sounding more like a faulty AP problem.

[todorn123]

I've contacted supplier for replacements of these two APs. Will let you know how it goes when I get new ones. Hopefully faulty APs is the problem.