topic Re: Cisco AP 9120 in deep green mode, no IP from DHCP in Wireless

Cisco AP 9120 in deep green mode, no IP from DHCP

tomand — Wed, 23 Mar 2022 09:56:17 GMT

Hi! We have several Cisco AP 9120 AXI which is not joining WLC and the obvious reason is no IP. When connecting to the AP via console you will see what is shown below.

Other Aps on the same switch is working okay and all are new. I saw the is shipped with ver 8.10.x

We will run them on 17.6.2 on WLC 9800-CL

Have found a bug case with now solution https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvm96943

We have tried the old trick with IP-helper and IP forward protocol udp 5246 with no better result.

When we set the IP manually with wlc IP it works and register /upgrades software. If you then delete the static IP you are back in green mode again..

Anyone seen this and hopefully found a solution?

[*03/22/2022 14:32:30.3730] <=== Deactivate Deep Green Mode
[*03/22/2022 14:32:30.3730] bcmswlpbk0 (Ext switch port: 😎 (Logical Port: 😎 Virtual link UP
[*03/22/2022 14:32:45.7260] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link DOWN.
[*03/22/2022 14:32:45.7260] ===> Activate Deep Green Mode
[*03/22/2022 14:32:45.7260] bcmswlpbk0 (Ext switch port: 😎 (Logical Port: 😎 Virtual link DOWN
[*03/22/2022 14:32:47.8330] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link Up at 1000 mbps full duplex
[*03/22/2022 14:32:47.8330] <=== Deactivate Deep Green Mode
[*03/22/2022 14:32:47.8330] bcmswlpbk0 (Ext switch port: 😎 (Logical Port: 😎 Virtual link UP
[*03/22/2022 14:33:02.0820] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link DOWN.
[*03/22/2022 14:33:02.0820] ===> Activate Deep Green Mode
[*03/22/2022 14:33:02.0820] bcmswlpbk0 (Ext switch port: 😎 (Logical Port: 😎 Virtual link DOWN
[*03/22/2022 14:33:04.1890] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link Up at 1000 mbps full duplex
[*03/22/2022 14:33:04.1890] <=== Deactivate Deep Green Mode
[*03/22/2022 14:33:04.1890] bcmswlpbk0 (Ext switch port: 😎 (Logical Port: 😎 Virtual link UP

*03/23/2022 08:44:01.2970] ERROR: return from vap_amsdu_rx_max was -45
[*03/23/2022 08:44:01.6800] DOT11_DRV[1]: set_channel Channel set to 36/20
[*03/23/2022 08:44:02.3790] wl: Unsupported
[*03/23/2022 08:44:02.3790] ERROR: return from vap_amsdu_rx_max was -45
[*03/23/2022 08:44:02.7870] DOT11_DRV[1]: set_channel Channel set to 36/20
[*03/23/2022 08:44:02.9900] DOT11_DRV[1]: set_channel Channel set to 36/20
[*03/23/2022 08:44:03.6900] wl: Unsupported
[*03/23/2022 08:44:03.6910] ERROR: return from vap_amsdu_rx_max was -45
[*03/23/2022 08:44:04.0220] wl: Unsupported
[*03/23/2022 08:44:04.0220] ERROR: return from vap_amsdu_rx_max was -45
[*03/23/2022 08:44:04.6170] DOT11_DRV[1]: set_channel Channel set to 36/20
[*03/23/2022 08:44:07.7050] IOT Bootloader Flash Tool 1.4
[*03/23/2022 08:44:07.7050] Firmware upgrade initiated by ble_init
[*03/23/2022 08:44:07.7050] bundled_ver_split ['2', '7', '20']
[ OK ] Started BLE init service.
[ OK ] Reached target Multi-User System.
[*03/23/2022 08:44:26.1310] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link DOWN.
[*03/23/2022 08:44:26.1310] ===> Activate Deep Green Mode
[*03/23/2022 08:44:26.1310] bcmswlpbk0 (Ext switch port: 😎 (Logical Port: 😎 Virtual link DOWN
[*03/23/2022 08:44:28.2370] <=== Deactivate Deep Green Mode
[*03/23/2022 08:44:28.2370] bcmswlpbk0 (Ext switch port: 😎 (Logical Port: 😎 Virtual link UP
[*03/23/2022 08:44:28.2370] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link Up at 1000 mbps full duplex
[*03/23/2022 08:44:42.4870] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link DOWN.
[*03/23/2022 08:44:42.4870] ===> Activate Deep Green Mode
[*03/23/2022 08:44:42.4870] bcmswlpbk0 (Ext switch port: 😎 (Logical Port: 😎 Virtual link DOWN
[*03/23/2022 08:44:44.5930] <=== Deactivate Deep Green Mode
[*03/23/2022 08:44:44.5930] bcmswlpbk0 (Ext switch port: 😎 (Logical Port: 😎 Virtual link UP
[*03/23/2022 08:44:44.5930] wired0 (Ext switch port: 7) (Logical Port: 15) (phyId: 1f) Link Up at 1000 mbps full duplex

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

Ambuj M — Wed, 23 Mar 2022 10:55:00 GMT

Just for test, try configuring dhcp scope local on the switch or router at AP gateway.

If it works then check dhcp server “bad address” entries, if there are any then clear them first. If possible do a capture on dhcp server and see if it even get the Unicast from up helper source ?

connect PC on the port where AP is connected see if you get an IP (remove forwarding protocol, just keep ip helper).

It seems like really you have to troubleshoot dhcp, since you said it works with static IP.

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

Flavio Miranda — Wed, 23 Mar 2022 11:42:15 GMT

There´s no such thing "We have tried the old trick with IP-helper and IP forward protocol udp 5246 with no better result."

Either you need ip-helper address or not. If your DHCP server sits in a different network, you will have a Layer 3 device between DHCP server and your Access Point and you will need to use DHCP helper for a simple reason. DHCP request is a broadcast traffic and will reach only the vlan where the Access Point is installed.

Then, you need dhcp-helper address in order to encapsulate this broadcast request into a unicast request in order to get to the DHCP server.

You can validade this simply by puting a notebook on the Access Point port and see if this get ip address. If not, the problem is not the AP. Make sure you dont have dot1x on the switch port or some kind of security like port-security, for example.

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

Rich R — Wed, 23 Mar 2022 14:31:29 GMT

CSCvm96943 is Terminated - would have been nice if TAC had updated the description to explain why it was terminated but whatever the reason this means devs concluded this is not a bug/does not need fixing, so was probably some kind of config/cable/switch problem. You could ask TAC for info if you want more detail.

As @Flavio Miranda has already mentioned DHCP relay is an absolute requirement if the server is not on the local subnet (not a trick).

So you need to go back to basics with layer 1 (cable, ports, hardware), layer 2 (ethernet config), layer 3 (IP config) and DHCP (relay and server). Use packet captures and debugs as appropriate to isolate the problem.

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

guoqing73 — Tue, 12 Jul 2022 07:41:06 GMT

I use AP9115 and the same issue occurred.

I checked configuration of switch port connecting to AP, and found no dot1q configured.

After configuring dot1q, AP gets IP.

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

sameh.makram1 — Wed, 10 Aug 2022 08:19:19 GMT

if you change vlan ap cisco work correctly sure

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

patrick-ryan — Mon, 24 Jun 2024 17:34:15 GMT

I'm having the same issue, from everything I can tell the AP is getting proper PoE, vlan is correct, interface vlan is correct, helper addresses to DHCP servers all correct because both my tester and laptop get IP no problem, there was port security rules on the port, which I have removed for trouble shooting, still don't get an IP. I don't understand why you'd need to turn on dot1q for the wap, we run newer switches that don't even accept the encapsulation dot1q command.

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

KBMTech — Thu, 12 Dec 2024 20:40:23 GMT

I have the same issue, but it is not with every IP. We employ over 10000 of these 9120AXI' throughout all our locations and this issue occurs in about 1%. I have tried some of the fixes found here for boot loops, and that seems to work in about a fourth of the defective units (removing and recreating the active boot partition to force boot from the other partition, and also restoring the image bundle via TFTP directly to the AP). When restoring the image via TFTP, I usually connect it and leave alone for about an hour and when I come back it is either joined to the WLC or still on Deep green mode. Don't sit there and wait, there's better paint to watch dry elsewhere. If after an hour it still on Deep Green Mode, It's an automatic RMA request to Cisco.

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

Gaurav Kansal — Mon, 16 Dec 2024 12:01:33 GMT

Hi,
Can you please share show version output of access points.

Regards
Gaurav Kansal

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

sameh.makram1 — Mon, 16 Dec 2024 14:24:25 GMT

I try changes vlan and success

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

Wes Schochet — Tue, 17 Dec 2024 12:43:45 GMT

I think the 5426 ip helper trick is designed to forward the CAPWAP discovery packets when the controllers are not responding to discovery broadcasts. IP address assignment is a whole different thing.

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

Gaurav Kansal — Thu, 19 Dec 2024 09:21:49 GMT

Hi,

Can you please describe that whats the issue and how you resolve this.

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

joshhunter — Thu, 19 Dec 2024 16:02:59 GMT

Hello All,

I am also having the same issue with Cisco 9115AXI brand new out of the box.

Most are working fine, but many refuse to obtain IP Addresses and sit looping in DARK GREEN MODE.

If I swap a 'bad' AP for a new AP out of the box, it will work fine. This seems to rule out port configuration, structured cabling, DHCP server etc.

I can get it to connect if I set a static IP Address and define the CAPWAP server address. If I remove the static configuration, the AP gets stuck again in the DHCP loop.

Additionally, I can see the DHCP Server offering a lease to the Access Point. I cannot make any sense of it. Also, had a few with corrupted boot image/partition; the known bug and work-around of deleting/re-creating the boot partition fixed those.

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

joshhunter — Thu, 19 Dec 2024 16:08:24 GMT

Hello, did you manage to fix or find a resolution?

Thxs

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

joshhunter — Thu, 19 Dec 2024 16:10:34 GMT

Hello, did you manage to fix or find a resolution? I too have a growing number of APs suffering corrupted/bad boot partitions and an even increasing number of APs stuck in DARK GREEN Mode.

Thxs

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

Leo Laohoo — Thu, 19 Dec 2024 22:35:59 GMT

What APs are these?

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

Rich R — Wed, 25 Dec 2024 13:40:14 GMT

What is your switch port configuration?
Depending on switch port config some APs will work and some won't - element of luck.
The config we came up with from working with TAC which works for all our APs (about 23,000 APs):
interface <interface name>
switchport trunk native vlan XXX
switchport mode trunk
spanning-tree portfast trunk
spanning-tree portfast edge trunk
spanning-tree bpdufilter enable
spanning-tree bpduguard disable
speed auto
duplex auto
cdp enable
end
portfast command variant depends on switch model/IOS version so both are there.
Obviously if you're running Local mode AP on access mode switchport then adjust the config accordingly.
The key points are portfast and the bpdu commands because if the port doesn't come up quickly enough or goes into blocking due to rogue bpdu's sent by the AP then the AP can't start.
Obviously you may also have other port config as required.

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

joshhunter — Tue, 14 Jan 2025 13:17:08 GMT

Hello, after further investigation, the Microsoft Server is marking entries with BAD ADDRESS, we only see this issue with new Access Points out of the box. From then onwards, after connecting to the Wireless Controller and upgrading their AP image we have not seen any DHCP Issue. So, the issue is only seen on new site roll outs where we are deploying new SD-WAN, New Switches and 20+ Access Points.

We suspect, this could possibly be the issue we are hitting. Either this, or due to the number of access points all requesting a DHCP Address at the same time (new project site install).

Cisco Bug: CSCvj14517
9800 support for "ap dhcp release-override" feature - interop with Microsoft DHCP servers

Symptom:
An access point sends three DHCP RELEASE packets, before attempting to get a new lease. This can happen when, for example, the AP is unable to join its controller for an extended period.

The DHCP server marks the address as "BAD_ADDRESS" ... after some time, as a result, the DHCP scope may run out of addresses.

Conditions:
This problem is seen with the following:

* controller is a 9800 (not AireOS)
* all AP models.
* DHCP server is Microsoft Windows 2008 or 2012 Server

https://quickview.cloudapps.cisco.com/quickview/bug/CSCwc32360

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

Rich R — Thu, 16 Jan 2025 13:44:59 GMT

Hi @joshhunter you've listed 2 completely different bugs there.

CSCvj14517 is a well known issue with Cisco APs and Microsoft DHCP server which Cisco have decided not to fix on 9800. So you either make sure you have a large enough pool to not run out or have regular maintenance to clear those "bad addresses" from the server.

CSCwc32360 affects wireless clients not APs so shouldn't be relevant to your AP problem. Again this is not being fixed - it's just a reminder to make sure you have enough IPs in the client DHCP pool. And remember to use "ip overlap" in flex profile if you intend to re-use IP addresses across sites (local switching)
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/technical-reference/c9800-best-practices.html#FlexConnectsitetag

Re: Cisco AP 9120 in deep green mode, no IP from DHCP

joshhunter — Thu, 16 Jan 2025 14:12:49 GMT

Hello @Rich R , Thank you for your reply.

My bad, I was referring only to CSCvj14517. I'm sure there were enough IP Addresses in the scope since there were only around 20-30APs and the scope had 115 addresses available, with 57 in Server Pool and 58 in partner pool, available. Deleting the BAD IPs did not seem to help recover the APs only recreating the scope when we tried.

We will continue investigating, and, on the next site roll out will double check what you have suggested.

As for the other bug I listed, my mistake, CSCwc32360, that was something else I was looking into and accidently pasted. You are correct, that Software update and the flex connect setting - IP Overlap was the fix for that issue.