Intermittent register on AP Catalyst 9130

[Erick Calzada]

Hi there, once again hoping you're healthy.

We have a WLAN with 6 APs model Catalyst 9130 working by Embedded Wireless Controller. The software version of this controller is 17.7.1.

The problem is that every Access Point in the network is continuously losing connection with the Controller. This happens randomly and in a intermittent way. However, the Access Points never get reloaded.

Checking the system logs, I found these ones:

ACE_MESSAGE-5-CAPWAPIMGDWNLD_EWC_AP_AP_LIST_EVENTS: Chassis 1 R0/0: wncd: List Event: AP with wtp_mac b811.4b3d.d500for Image Type ap1g6a is removed from Master AP list
Jan 31 13:37:21.777: %CAPWAP_IMGDWNLD_TRACE_MESSAGE-5-CAPWAPIMGDWNLD_EWC_AP_AP_LIST_EVENTS: Chassis 1 R0/0: wncd: List Event: AP with wtp_mac b811.4b3d.d2e0for Image Type ap1g6a is added to Master AP list
Jan 31 13:37:21.777: %CAPWAP_IMGDWNLD_TRACE_MESSAGE-5-CAPWAPIMGDWNLD_EWC_AP_AP_LIST_EVENTS: Chassis 1 R0/0: wncd: List Event: Re-election of Master AP is successful for Image Type ap1g6a
Jan 31 13:37:21.778: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP3' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:37:24.342: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP5' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:37:26.200: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP5' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:38:50.054: %CAPWAP_IMGDWNLD_TRACE_MESSAGE-5-CAPWAPIMGDWNLD_EWC_AP_AP_LIST_EVENTS: Chassis 1 R0/0: wncd: List Event: AP with wtp_mac b811.4b3d.d2e0for Image Type ap1g6a is removed from Master AP list
Jan 31 13:38:50.054: %CAPWAP_IMGDWNLD_TRACE_MESSAGE-5-CAPWAPIMGDWNLD_EWC_AP_AP_LIST_EVENTS: Chassis 1 R0/0: wncd: List Event: AP with wtp_mac b811.4b3d.d480for Image Type ap1g6a is added to Master AP list
Jan 31 13:38:50.054: %CAPWAP_IMGDWNLD_TRACE_MESSAGE-5-CAPWAPIMGDWNLD_EWC_AP_AP_LIST_EVENTS: Chassis 1 R0/0: wncd: List Event: Re-election of Master AP is successful for Image Type ap1g6a
Jan 31 13:38:50.066: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP4 Mac: b811.4b66.c860 Session-IP: 192.168.100.51[5264] 192.168.100.23[5246] Ethernet MAC: a488.73d6.5b9c Joined
Jan 31 13:38:51.192: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP4' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:38:52.096: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP6' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:38:53.056: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP4' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:39:05.550: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP6' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:39:22.932: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP3 Mac: b811.4b3d.d500 Session-IP: 192.168.100.50[5272] 192.168.100.23[5246] Ethernet MAC: a488.73d6.49fc Joined
Jan 31 13:39:23.897: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP3' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:39:25.660: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP3' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:41:16.168: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP3 Mac: b811.4b3d.d500 Session-IP: 192.168.100.50[5272] 192.168.100.23[5246] CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
Jan 31 13:41:16.176: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP3' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:41:49.303: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP3 Mac: b811.4b3d.d500 Session-IP: 192.168.100.50[5272] 192.168.100.23[5246] Ethernet MAC: a488.73d6.49fc Joined
Jan 31 13:41:50.293: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP3' is no longer a peer (internal AP, selection ENABLED)
Jan 31 13:41:52.299: %PEER_SELECTION-5-EWC_PEER_SELECTION_REMOVE_EV: Chassis 1 R0/0: wncd: REMOVE event: AP 'AP3' is no longer a peer (internal AP, selection ENABLED)

I wish you could help me with this question.

Best regards and have a nice day!

[balaji.bandi]

Where is these AP connected, the same switch? what is the switch Logs, and can you post the switch port config?

[Erick Calzada]

Hi BB, thanks for your quick response!

That's right, all the APs are connected to the same Switch. The logs from the Switch don't show anything related because the APs are not disconnected from the Switch, but only from the EWC.

Also, the config of the ports is as follows: we have two VLANS on the Switch; however, we use only one of them for Administration and users. The ports for Access Points are in trunk mode.

Best regards!

[Scott Fella]

Keep in mind, if the ap doesn't stay joined, its not intermittent.  Intermittent is like if the ap is joined and once a week or a few times a day, the AP looses connection to the controller.  Knowing if the ap can join successfully for at least a few hours is different from an ap trying to join and failing.  The blogs and videos that are out there are very good resources to follow as its typically something that is missed.

These are not plug and play solutions.  Once you get it properly setup, then its plug and play.

[Scott Fella]

Let's get some more info.

Is this a new install and was it working before?

Is the EWC controller running fine and the AP within the EWC, is that joined?

Did you follow the guide step by step?

NTP and country code configured

Cisco Embedded Wireless Controller on Catalyst Access Points Configuration Guide, IOS XE Cupertino 17.7.x - Overview of Cisco Embedded Wireless Controller on Catalyst Access Points [Cisco Embedded Wireless Controller on Catalyst Access Points] - Cisc...

[Erick Calzada]

Hi Scott, thanks for your response.

Is this a new install and was it working before?

• With this solution, this is a new installation. The WLAN used to work with a Physical Controller model 2500 series, and the Access Points were some 3700 series, so it worked fine. When we migrated to this new solution, we started having problems.

Is the EWC controller running fine and the AP within the EWC, is that joined?

• Well, sometimes. See: the EWC has the same problem than the Access Points, sometimes we´re able to see the Controller, but not the AP that is selected as preferred master.

Did you follow the guide step by step?

• Yes, we've read and followed the guide you're sharing me.

NTP and country code configured

• Yes, they are. In fact, we're able to see the Controller and Access Points by periods of time.

Best regards!

[Scott Fella]

AireOS and IOS-XE are different so we won't compare that.  Did you go through the day 0 setup?  With EWC, accessing the controller is the easiest part, but many struggle getting the ap to join itself.  Like what is mentioned on all the guides, the ap needs to be connected to a trunk port with a native vlan that is the wireless management.  You need to make sure the ap's get an ip address and also make sure you defined the country code on the controller or else the ap's will not join.

Here are a few more links to review.

Embedded WLC – CRAZY NETWORKING (defaultgateway.org)

Page 32: BRKEWN-2010 (ciscolive.com)

[Arshad Safrulla]

Is all the AP's are EWC capable? If yes please set the preferred master. To do this you need to go to Configuration>>Access Points>>Select the Master AP>>Advanced>>Check the preferred master and make controller.

Make sure that preferred master is selected only on AP's which EWC capable. Recommendation is you do this on 2 AP's as this will enable you to have HA. All the other AP's must have this option not selected and Lightweight image. 

Also why 17.7.1? It is very new short term support release and prone to bugs and and unknown issues. II I were you, I would consider latest release from 17.3.X or 17.6.X.

[Erick Calzada]

Hi Arshadsaf, thanks a lot for your response.

Yes, we only have one Access Point as EWC capable, in fact, in this moment we don't have HA, so the rest of Access Points are in CAPWAP mode.

We tried with older software versions before, but it used to do the same thing, so that's why we decided to upload to a newer version.

In addition, the EWC and the other Access Points get lost in the same way, but they never lose energy.

Best regards!

[Arshad Safrulla]

is the EWC and the AP's in the same broadcast domain/VLAN? It is by design mandatory that the management of the EWC and all the AP's to be in the same VLAN.

Also make sure that you don't have any other Cisco WLC/ EWC in the same management VLAN. @Scott Fella mentioned please make sure your upstream switch ports are configured properly. It has to be trunk with, management vlan as the trunk and allow only wireless VLAN's over that trunk.