CAPWAP AP Keeps Flipping (UP/DOWN) in EWC AP Environment

[BenXue2009]

Hello Cisco Community,

I’m facing an issue with a network setup that includes  EWC and  CAPWAP APs,they are all connect Catalyst 9200L,same trunk allow vlan,same native vlan(vlan 17,ip add 172.16.7.X/24). The problem is that CAPWAP APs keep going UP/DOWN repeatedly. I would appreciate any guidance or suggestions.

• Devices:

  • EWC Model: Cisco 9115AXI-E-EWC(version:16.12.4a)

  • CAPWAP APs: Cisco 9115AXI-E(version:17.9.4)

  • Switches: Cisco 9200L-48P switches.

• Symptoms:

  • CAPWAP APs fail to stabilize and keep flipping (UP/DOWN).

  • Logs indicate: Heartbeat timer expiry for AP. Close CAPWAP DTLS session.

  • Observed errors:.Last Disconnect Reason: Heartbeat timeout

•  Known Issues:

  • EWC is running an older version (16.12.4a), and CAPWAP APs are on 17.9.4.

  • Attempted EWC upgrade to 17.9.6 via TFTP but encountered upgrade failed.

  • Checked cables, Switch ports configurations, and CAPWAP settings—all appear correct.

  • Switch ports config as below:

    interface GigabitEthernet1/0/3
    switchport trunk native vlan 17
    switchport trunk allowed vlan 11,13,17
    switchport mode trunk
    end

    interface GigabitEthernet2/0/5
    switchport trunk native vlan 17
    switchport trunk allowed vlan 11,13,17
    switchport mode trunk
    end

    interface GigabitEthernet2/0/6
    switchport trunk native vlan 17
    switchport trunk allowed vlan 11,13,17
    switchport mode trunk
    end

• EWC version:     AP-1-9115AXI-E-EWC#show version
  Cisco IOS XE Software, Version 16.12.4a
  Cisco IOS Software [Gibraltar], C9800-AP Software (C9800-AP-K9_IOSXE-UNIVERSALK9-M), Version 16.12.4a 

• AP-1-9115AXI-E-EWC#show wireless stats ap join summary
  Number of APs: 4

  Base MAC Ethernet MAC AP Name IP Address Status Last Failure Phase Last Disconnect Reason
  -----------------------------------------------------------------------------------------------------------------------------
  20cc.2732.45XX 70bc.482c.08XX AP70BC.482C.08XX 172.16.7.11 Not Joined Image-Download Heart beat
  20cc.2732.55XX 70bc.482c.0aXX AP70BC.482C.0AXX 172.16.7.4 
  20cc.2732.63XX 70bc.482c.0cXX AP70BC.482C.0CXX 172.16.7.9 Not Joined Image-Download Heart beat
  e437.9fde.d6XX  e437.9fd6.5aXX APE437.9FD6.5AXX 172.16.7.2

• CAPWAP setting
• AP-1-9115AXI-E-EWC#show wireless management trustpoint
  Trustpoint Name : CISCO_IDEVID_SUDI
  Certificate Info : Available
  Certificate Type : MIC
  Private key Info : Available
  FIPS suitability : Not Applicable  

• Configuration Checks: Verified that switch configurations, VLANs, and connectivity are correctly configured.

•  Questions for the Community:

  1. How can I successfully upgrade EWC to 17.9.6 to match CAPWAP AP firmware?

  2. Is there a way to resolve the issue without upgrading?Heartbeat timer expiry

  3. Are there any other configurations or best practices I might have missed?

     Any guidance or recommendations would be greatly appreciated!

     Thank you in advance for your help.

[MHM Cisco World]

Check in config the below line 

Wireless management trustpoint CISCO_IDEVID_SUDI

The 9800 must have trustpoint to allow AP to join 

MHM

[BenXue2009]

Thx man ,but it is default,already have.
#show wireless management trustpoint
Trustpoint Name : CISCO_IDEVID_SUDI
Certificate Info : Available
Certificate Type : MIC
Private key Info : Available
FIPS suitability : Not Applicable

[marce1000]

 - The software version on the EWC AP  is too  old , go for at least 17.9.5 or above , 

 M.

[BenXue2009]

Thx man,That is what I thought. and I was download 17.9.6 and copy it in EWC AP bootflash.

and "boot system flash:C9800-AP-iosxe-wlc.bin"
but upgrade didn't work as below.

#dir
Directory of bootflash:/
662 drwx 400 Nov 27 2024 12:03:36 +01:00 core
576 -rw- 2 Nov 27 2024 12:05:50 +01:00 recovery_mode
1042 -rw- 272 Nov 28 2024 13:44:06 +01:00 btdecode_err
663 drwx 504 Nov 27 2024 12:04:42 +01:00 dc_profile_dir
658 drwx 400 Nov 27 2024 12:04:22 +01:00 .installer
2432 -rw- 2 Nov 27 2024 15:10:38 +01:00 debugTrace_172.16.7.11.txt

2006 -rw- 302822428 Nov 27 2024 11:35:52 +01:00 C9800-AP-iosxe-wlc.bin

#boot system bootflash:C9800-AP-iosxe-wlc.bin
Command Executed Successfully

#show boot
BOOT variable does not exist
CONFIG_FILE variable does not exist
Configuration register is 0x2102
Standby BOOT variable does not exist
Standby CONFIG_FILE variable does not exist
Standby BOOTLDR variable does not exist
Standby Configuration register is 0x2102

[marce1000]

 - Looks like you did not do a correct upgrade ; start from https://software.cisco.com/download/home/286321056/type/286323077/release/17.9.6

 M.

[BenXue2009]

Yes,that is what I download,but upgrade failed.

And now AP no space to update ap1g7,I will delete some debug docments in AP,but cannot remote do like that.

[marce1000]

  - In general you can't keep the images for client AP's on the EWC itself ; they need to be provisioned trough a separate tftp or sftp server ; for info's look at : https://www.cisco.com/c/en/us/td/docs/wireless/controller/ewc/17-1/config-guide/ewc_cg_17_11/image_download.html#id_128466
                                        https://www.cisco.com/c/en/us/td/docs/wireless/controller/ewc/17-1/config-guide/ewc_cg_17_11/image_download.html#id_128465

   Then remove those images for client-APs from the EWC and you will have sufficient space for it's own image:
   (C9800-AP-iosxe-wlc.bin)

 M.

[BenXue2009]

Yes,yesterday I was already upload image from WEB GUI---Software Management---SMU finished

I should upload from Software Management---software upgrade,there have AP image,but there is no space in EWC AP now,and I am back home from customer already,I think I will be there Jan 2025, and try to fix it with CLI.

[marce1000]

 - The first screenshot is wrong and not applicable for upgrading the EWC image on the AP ; as far as the second  is concerned ; as stated , remove the client images from the EWC AP , setup the tftp server and follow the guidelines from my previous reply to have the EWC redirect the clients to where to find their images, and best is to do all with the CLI for more direct interaction and overview of what is going on , 

 M.

[Rich R]

I suggest you practise this yourself *before* you go to the customer site again!

Like Marce said you must not copy files to the AP flash directly - there is not enough space and you're not supposed to do it that way - it just won't work.

My advice at this point:
- Backup the EWC config
- Convert the EWC back to CAPWAP mode
- Install the latest CAPWAP image as per TAC recommended link below.  17.9 is already approaching end of life so I recommend 17.12.x
- Convert back to EWC using standard conversion procedure from the EWC config guide using CLI or GUI (not by copying files to the AP flash!)
- Restore the EWC config but note that the trustpoint has changed to CISCO_IDEVID_CMCA3_SUDI so if your existing config sets the trustpoint manually you might need to remove that config because EWC will use the new cert by default (no config required).  The output should look like this:
C9120AXI-WLC#show wireless management trustpoint
Trustpoint Name : CISCO_IDEVID_CMCA3_SUDI
Certificate Info : Available
Certificate Type : MIC
Certificate Hash : 7bc0db793f78bf6d16db381e5a0ae451bb6da9c8
Private key Info : Available
FIPS suitability : Not Applicable
https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-15/config-guide/b_wl_17_15_cg/sudi99-certificate-support.html
- Make sure there is a TFTP server for the rest of the APs to download the software from.  You can use a switch or router for that.

[Haydn Andrews]

The EWC is running 16.12.x

You need to upgrade to 17.3.5 or 17.6.x first before you can go to 17.9.x

Other option convert one of the APs on 17.9 to an EWC and then revert this EWC to CAPWAP