CAPWAP - Failed to decode discovery response

[Heitkamp]

Hi Guys!

I did a WLC (WLC 2504) Update to version 8.3.150.0 and after the update 2 of our 30 access points (AIR-AP1852I-E-K9) cannot connect to the WLC. I connected via console to the access points and get following error message:

[*05/03/2020 15:39:37.4246] Discovery Response from 192.168.219.130
[*05/03/2020 15:39:37.4246]
[*05/03/2020 15:39:37.4246] This AP model is supported by controller version newer than 7.0.0.0!
[*05/03/2020 15:39:37.4246]
[*05/03/2020 15:39:37.4246] Discovery response from MWAR ''running version 0.0.0.0 is rejected.
[*05/03/2020 15:39:37.4246] Failed to decode discovery response.
[*05/03/2020 15:39:37.4246] CAPWAP SM handler: Failed to process message type 2 state 2.
[*05/03/2020 15:39:37.4246] Failed to handle capwap control message from controller - status 4
[*05/03/2020 15:39:37.4246] Failed to process unencrypted capwap packet 0x1803000 from 192.168.219.130
[*05/03/2020 15:39:37.4246] Failed to send capwap message 0 to the state machine. Packet already freed.
[*05/03/2020 15:39:37.4246] IPv4 wtpProcessPacketFromSocket returned 4
[*05/03/2020 15:39:37.4246] Discovery Response from 192.168.219.130
[*05/03/2020 15:39:37.4246]
[*05/03/2020 15:39:37.4246] This AP model is supported by controller version newer than 7.0.0.0!
[*05/03/2020 15:39:37.4246]
[*05/03/2020 15:39:37.4246] Discovery response from MWAR ''running version 0.0.0.0 is rejected.
[*05/03/2020 15:39:37.4246] Failed to decode discovery response.
[*05/03/2020 15:39:37.4246] CAPWAP SM handler: Failed to process message type 2 state 2.
[*05/03/2020 15:39:37.4246] Failed to handle capwap control message from controller - status 4
[*05/03/2020 15:39:37.4246] Failed to process unencrypted capwap packet 0x1801000 from 192.168.219.130
[*05/03/2020 15:39:37.4246] Failed to send capwap message 0 to the state machine. Packet already freed.
[*05/03/2020 15:39:37.4246] IPv4 wtpProcessPacketFromSocket returned 4

I manually updated the version of one access point to 8.3.150.0 because I already found this bug page .

sh version
cisco AIR-AP1852I-E-K9 ARMv7 Processor rev 0 (v7l) with 967420/800844K bytes of memory.
Processor board ID XXXXXXXXXX
AP Running Image : 8.3.150.0
Primary Boot Image : 8.3.150.0
Backup Boot Image : 8.3.143.0

What do I need to do?

Kind regards

Heitkamp

[Rafael E]

After upgrading AP manually did it work? 

[Heitkamp]

No. That is why i opened this discussion.

[Rafael E]

whats is the WLC controller story?

i would run: 

WLC 

debug capwap events enable

debug capwap errors enable 

debug capwap dtls-keepalive enable

AP 

debug capwap client events

debug capwap client errors

[Heitkamp]

WLC output

*spamApTask0: May 13 16:58:45.107: c4:b9:cd:fb:e6:20 Join Priority Processing status = 0, Incoming Ap's Priority 0, MaxLrads = 55, MaxLicense=30 joined Aps =28
*spamApTask0: May 13 16:58:45.107: c4:b9:cd:fb:e6:20 apType = 71 apModel: 
*spamApTask0: May 13 16:58:45.108: c4:b9:cd:fb:e6:20 apType = 71 apModel: 
*spamApTask0: May 13 16:58:45.108: c4:b9:cd:fb:e6:20 Discovery Response sent to 192.168.219.167:5248
*spamApTask0: May 13 16:58:45.114: c4:b9:cd:fb:e6:20 Discovery Request from 192.168.219.167:5248

AP output

Starting Discovery.
[*05/03/2020 16:14:49.8438]
[*05/03/2020 16:14:49.8438] CAPWAP State: Discovery
[*05/03/2020 16:14:49.8438] Discovery Request sent to 192.168.219.130, discovery type STATIC_CONFIG(1)
[*05/03/2020 16:14:49.8538] Did not get log server settings from DHCP.
[*05/03/2020 16:14:49.8538] Could Not resolve CISCO-CAPWAP-CONTROLLER
[*05/03/2020 16:14:49.8538]
[*05/03/2020 16:14:49.8538] Send Discovery requests to dcb capwapCfg.mwarIpAddr[0]=192.168.219.130
[*05/03/2020 16:14:49.8538] Discovery Request sent to 192.168.219.130, discovery type STATIC_CONFIG(1)
[*05/03/2020 16:14:49.8538]
[*05/03/2020 16:14:49.8538] SENDING DISCOVERY REQUEST: WLC name wlc-2504, addr 192.168.219.130
[*05/03/2020 16:14:49.8538]
[*05/03/2020 16:14:49.8538] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*05/03/2020 16:14:49.8538] AP in local mode switch to standalone
[*05/03/2020 16:14:49.8538] Discovery Response from 192.168.219.130
[*05/03/2020 16:14:49.8538]
[*05/03/2020 16:14:49.8538] This AP model is supported by controller version newer than 7.0.0.0!
[*05/03/2020 16:14:49.8538]
[*05/03/2020 16:14:49.8538] Discovery response from MWAR ''running version 0.0.0.0 is rejected.
[*05/03/2020 16:14:49.8538] Failed to decode discovery response.
[*05/03/2020 16:14:49.8538] CAPWAP SM handler: Failed to process message type 2 state 2.
[*05/03/2020 16:14:49.8538] Failed to handle capwap control message from controller - status 4
[*05/03/2020 16:14:49.8538] Failed to process unencrypted capwap packet 0x1dd0000 from 192.168.219.130
[*05/03/2020 16:14:49.8538] Failed to send capwap message 0 to the state machine. Packet already freed.
[*05/03/2020 16:14:49.8538] IPv4 wtpProcessPacketFromSocket returned 4
[*05/03/2020 16:14:49.8538] Discovery Response from 192.168.219.130
[*05/03/2020 16:14:49.8538]
[*05/03/2020 16:14:49.8538] This AP model is supported by controller version newer than 7.0.0.0!

[Rafael E]

I think your issue is related to: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvu20765

There isn't any fix as of today. BUG is on progress. 

[Heitkamp]

Sorry but I don't have the permissions to see this bug.

[Rafael E]

Sorry i did not realize it was internal only... you should open a SR with TAC so we can make that BUG visible and once fix is ready you get the image. 

[Rich R]

you could also be hitting https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvh85082 which is not fixed in 8.3, in which case your only option is to upgrade to 8.5.161.0.
8.3 is end of life https://www.cisco.com/c/en/us/products/collateral/wireless/8500-series-wireless-controllers/bulletin-c25-742339.html
But before you upgrade make sure all your APs are supported on 8.5 - see https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc11 and https://www.cisco.com/c/en/us/td/docs/wireless/compatibility/matrix/compatibility-matrix.html#pgfId-590411 and read the 8.5 release notes.

[Heitkamp]

Sadly we have some older access points, which need 8.3 to work reasonable.
But thank you for this bug search!

[Heitkamp]

I upgraded the access point to the version 8.5.161.0 but I still get this message.

[*05/03/2020 16:23:59.8326] CAPWAP State: Discovery
[*05/03/2020 16:23:59.8326] Discovery Request sent to 192.168.219.130, discovery type STATIC_CONFIG(1)
[*05/03/2020 16:23:59.8326] Discovery Request sent to 192.168.219.130, discovery type STATIC_CONFIG(1)
[*05/03/2020 16:23:59.8326] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*05/03/2020 16:23:59.8326] Discovery Response from 192.168.219.130
[*05/03/2020 16:23:59.8326] Discovery response from MWAR ''running version 0.0.0.0 is rejected.
[*05/03/2020 16:23:59.8326] Failed to decode discovery response(status = 4).
[*05/03/2020 16:23:59.8326] CAPWAP SM handler: Failed to process message type 2 state 2.
[*05/03/2020 16:23:59.8326] Failed to handle capwap control message from controller - status 4
[*05/03/2020 16:23:59.8326] Failed to process unencrypted capwap packet 0x1dfe000 from 192.168.219.130
[*05/03/2020 16:23:59.8326] Failed to send message to CAPWAP state machine, msgId 0
[*05/03/2020 16:23:59.8326] Failed to send capwap message 0 to the state machine. Packet already freed.
[*05/03/2020 16:23:59.8326] IPv4 wtpProcessPacketFromSocket returned 4
[*05/03/2020 16:23:59.8326] Discovery Response from 192.168.219.130
[*05/03/2020 16:23:59.8326] Discovery response from MWAR ''running version 0.0.0.0 is rejected.
[*05/03/2020 16:23:59.8326] Failed to decode discovery response(status = 4).
[*05/03/2020 16:23:59.8326] CAPWAP SM handler: Failed to process message type 2 state 2.
[*05/03/2020 16:23:59.8326] Failed to handle capwap control message from controller - status 4
[*05/03/2020 16:23:59.8326] Failed to process unencrypted capwap packet 0x1df8000 from 192.168.219.130
[*05/03/2020 16:23:59.8326] Failed to send message to CAPWAP state machine, msgId 0
[*05/03/2020 16:23:59.8326] Failed to send capwap message 0 to the state machine. Packet already freed.
[*05/03/2020 16:23:59.8326] IPv4 wtpProcessPacketFromSocket returned 4
[*05/03/2020 16:23:59.8326] Discovery Response from 192.168.219.130
[*05/03/2020 16:23:59.8326] Discovery response from MWAR ''running version 0.0.0.0 is rejected.
[*05/03/2020 16:23:59.8326] Failed to decode discovery response(status = 4).
[*05/03/2020 16:23:59.8326] CAPWAP SM handler: Failed to process message type 2 state 2.
[*05/03/2020 16:23:59.8326] Failed to handle capwap control message from controller - status 4
[*05/03/2020 16:23:59.8326] Failed to process unencrypted capwap packet 0x1df4000 from 192.168.219.130
[*05/03/2020 16:23:59.8326] Failed to send message to CAPWAP state machine, msgId 0
[*05/03/2020 16:23:59.8326] Failed to send capwap message 0 to the state machine. Packet already freed.
[*05/03/2020 16:23:59.8326] IPv4 wtpProcessPacketFromSocket returned 4

[patoberli]

As it's still not working on the latest 8.5.161.0 release, I suggest opening a TAC.
You do have enough free licenses on the WLC?

[MaxFisher925]

Was a solution or explanation reached with your issue? 

We have a similar issue with a 2504 and 1832i.

[Heitkamp]

Sadly, no. A few months later we upgraded to Meraki. During the transition period, we downgraded to the working version.

[Rich R]

@MaxFisher925  Well first of all - are you running latest version of software - which in your case would be 8.5.182.7? (refer to TAC recommended releases below which get updated regularly)

Many more bugs are fixed in 8.10 but you can't use that on 2504.

If it's just a single 1832 showing this problem it may also be hardware failure in which case RMA.

Any more detailed help will require what software version you're using and full console log from the AP from power on.

<moved to in-line reply from original>