Cisco AP 9120 in deep green mode, no IP from DHCP

[tomand]

Hi! We have several Cisco AP 9120 AXI which is not joining WLC and the obvious reason is no IP. When connecting to the AP via console you will see what is shown below.

Other Aps on the same switch is working okay and all are new. I saw the is shipped with ver 8.10.x

We will run them on 17.6.2 on WLC 9800-CL

Have found a bug case with now solution https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvm96943

We have tried the old trick with IP-helper and IP forward protocol udp 5246 with no better result.

When we set the IP manually with wlc IP it works and register /upgrades software. If you then delete the static IP you are back in green mode again..

Anyone seen this and hopefully found a solution?

[*03/22/2022 14:32:30.3730] <=== Deactivate Deep Green Mode
[*03/22/2022 14:32:30.3730] bcmswlpbk0 (Ext switch port: (Logical Port: Virtual link UP
[*03/22/2022 14:32:45.7260] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link DOWN.
[*03/22/2022 14:32:45.7260] ===> Activate Deep Green Mode
[*03/22/2022 14:32:45.7260] bcmswlpbk0 (Ext switch port: (Logical Port: Virtual link DOWN
[*03/22/2022 14:32:47.8330] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link Up at 1000 mbps full duplex
[*03/22/2022 14:32:47.8330] <=== Deactivate Deep Green Mode
[*03/22/2022 14:32:47.8330] bcmswlpbk0 (Ext switch port: (Logical Port: Virtual link UP
[*03/22/2022 14:33:02.0820] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link DOWN.
[*03/22/2022 14:33:02.0820] ===> Activate Deep Green Mode
[*03/22/2022 14:33:02.0820] bcmswlpbk0 (Ext switch port: (Logical Port: Virtual link DOWN
[*03/22/2022 14:33:04.1890] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link Up at 1000 mbps full duplex
[*03/22/2022 14:33:04.1890] <=== Deactivate Deep Green Mode
[*03/22/2022 14:33:04.1890] bcmswlpbk0 (Ext switch port: (Logical Port: Virtual link UP

*03/23/2022 08:44:01.2970] ERROR: return from vap_amsdu_rx_max was -45
[*03/23/2022 08:44:01.6800] DOT11_DRV[1]: set_channel Channel set to 36/20
[*03/23/2022 08:44:02.3790] wl: Unsupported
[*03/23/2022 08:44:02.3790] ERROR: return from vap_amsdu_rx_max was -45
[*03/23/2022 08:44:02.7870] DOT11_DRV[1]: set_channel Channel set to 36/20
[*03/23/2022 08:44:02.9900] DOT11_DRV[1]: set_channel Channel set to 36/20
[*03/23/2022 08:44:03.6900] wl: Unsupported
[*03/23/2022 08:44:03.6910] ERROR: return from vap_amsdu_rx_max was -45
[*03/23/2022 08:44:04.0220] wl: Unsupported
[*03/23/2022 08:44:04.0220] ERROR: return from vap_amsdu_rx_max was -45
[*03/23/2022 08:44:04.6170] DOT11_DRV[1]: set_channel Channel set to 36/20
[*03/23/2022 08:44:07.7050] IOT Bootloader Flash Tool 1.4
[*03/23/2022 08:44:07.7050] Firmware upgrade initiated by ble_init
[*03/23/2022 08:44:07.7050] bundled_ver_split ['2', '7', '20']
[ OK ] Started BLE init service.
[ OK ] Reached target Multi-User System.
[*03/23/2022 08:44:26.1310] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link DOWN.
[*03/23/2022 08:44:26.1310] ===> Activate Deep Green Mode
[*03/23/2022 08:44:26.1310] bcmswlpbk0 (Ext switch port: (Logical Port: Virtual link DOWN
[*03/23/2022 08:44:28.2370] <=== Deactivate Deep Green Mode
[*03/23/2022 08:44:28.2370] bcmswlpbk0 (Ext switch port: (Logical Port: Virtual link UP
[*03/23/2022 08:44:28.2370] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link Up at 1000 mbps full duplex
[*03/23/2022 08:44:42.4870] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link DOWN.
[*03/23/2022 08:44:42.4870] ===> Activate Deep Green Mode
[*03/23/2022 08:44:42.4870] bcmswlpbk0 (Ext switch port: (Logical Port: Virtual link DOWN
[*03/23/2022 08:44:44.5930] <=== Deactivate Deep Green Mode
[*03/23/2022 08:44:44.5930] bcmswlpbk0 (Ext switch port: (Logical Port: Virtual link UP
[*03/23/2022 08:44:44.5930] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link Up at 1000 mbps full duplex

[ammahend]

Just for test, try configuring dhcp scope local on the switch or router at AP gateway.

If it works then check dhcp server “bad address” entries, if there are any then clear them first. If possible do a capture on dhcp server and see if it even get the Unicast from up helper source ?

connect PC on the port where AP is connected see if you get an IP (remove forwarding protocol, just keep ip helper).

It seems like really you have to troubleshoot dhcp, since you said it works with static IP. 

[Flavio Miranda]

There´s no such thing "We have tried the old trick with IP-helper and IP forward protocol udp 5246 with no better result."

Either you need ip-helper address or not. If your DHCP server sits in a different network, you will have a Layer 3 device between DHCP server and your Access Point and you will need to use DHCP helper for a simple reason. DHCP request is a broadcast traffic and will reach only the vlan where the Access Point is installed.

 Then, you need dhcp-helper address in order to encapsulate this broadcast request into a unicast request in order to get to the DHCP server.

You can validade this simply by puting a notebook on the Access Point port and see if this get ip address. If not, the problem is not the AP. Make sure you dont have dot1x on the switch port or some kind of security like port-security, for example.

[Wes Schochet]

I think the 5426 ip helper trick is designed to forward the CAPWAP discovery packets when the controllers are not responding to discovery broadcasts.  IP address assignment is a whole different thing.

[Rich R]

CSCvm96943 is Terminated - would have been nice if TAC had updated the description to explain why it was terminated but whatever the reason this means devs concluded this is not a bug/does not need fixing, so was probably some kind of config/cable/switch problem.  You could ask TAC for info if you want more detail.

As @Flavio Miranda has already mentioned DHCP relay is an absolute requirement if the server is not on the local subnet (not a trick).

So you need to go back to basics with layer 1 (cable, ports, hardware), layer 2 (ethernet config), layer 3 (IP config) and DHCP (relay and server).  Use packet captures and debugs as appropriate to isolate the problem.

[guoqing73]

I use AP9115 and the same issue occurred. 

I checked configuration of switch port connecting to AP, and found no dot1q configured.

After configuring dot1q, AP gets IP.

[patrick-ryan]

I'm having the same issue, from everything I can tell the AP is getting proper PoE, vlan is correct, interface vlan is correct, helper addresses to DHCP servers all correct because both my tester and laptop get IP no problem, there was port security rules on the port, which I have removed for trouble shooting, still don't get an IP. I don't understand why you'd need to turn on dot1q for the wap, we run newer switches that don't even accept the encapsulation dot1q command.

[joshhunter]

Hello, did you manage to fix or find a resolution?

Thxs

[sameh.makram1]

if you change vlan ap cisco work correctly  sure 

[KBMTech]

I have the same issue, but it is not with every IP. We employ over 10000 of these 9120AXI' throughout all our locations and this issue occurs in about 1%.  I have tried some of the fixes found here for boot loops, and that seems to work in about a fourth of the defective units (removing and recreating the active boot partition to force boot from the other partition, and also restoring the image bundle via TFTP directly to the AP). When restoring the image via TFTP, I usually connect it and leave alone for about an hour and when I come back it is either joined to the WLC or still on Deep green mode. Don't sit there and wait, there's better paint to watch dry elsewhere. If after an hour it still on Deep Green Mode, It's an automatic RMA request to Cisco.

[joshhunter]

Hello, did you manage to fix or find a resolution? I too have a growing number of APs suffering corrupted/bad boot partitions and an even increasing number of APs stuck in DARK GREEN Mode. 

Thxs

[joshhunter]

Hello, did you manage to fix or find a resolution? I too have a growing number of APs suffering corrupted/bad boot partitions and an even increasing number of APs stuck in DARK GREEN Mode. 

Thxs

[Leo Laohoo]

What APs are these?

[Gaurav Kansal]

Hi,
Can you please share show version output of access points.

Regards
Gaurav Kansal

[sameh.makram1]

I try changes vlan and success

[Gaurav Kansal]

Hi,

Can you please describe that whats the issue and how you resolve this.