Cisco 2602 Light Weigth AP's randomly disjoining controller

[ALIAOF_]

I basically need a confirmation on a particular setup I'm seeing:

1. Cisco 2504 controller
2. Cisco 2602 AP's
3. AP's randomly will leave the controller and re join it.  AP log will show all that plus all the radio interfaces resetting during this time.
4. Also WLC shows the following under "stats" Layer 3 discovery request not received on management VLAN
5. So the WLC's management interface is in VLAN1, but the switch port some of these AP's are connected to is in a different VLAN

So I'm thinking that is the main issue, but why they still end up joining the controller if they are in a different VLAN?  I need to move all the AP's in VLAN1

Here is the log:

*May  8 03:50:56.835: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST
., 2)
*May  8 03:50:56.835: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE
*May  8 03:50:56.835: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.141.80.35:5246
*May  8 03:50:56.915: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*May  8 03:50:56.915: %LWAPP-3-CLIENTERRORLOG: LWAPP LED Init: incorrect led state 255
*May  8 03:50:56.927: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down
*May  8 03:50:56.927: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down
*May  8 03:50:56.939: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*May  8 03:50:56.955: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*May  8 03:50:57.531: %CLEANAIR-6-STATE: Slot 0 down
*May  8 03:50:57.531: %CLEANAIR-6-STATE: Slot 1 down
*May  8 03:50:57.927: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*May  8 03:50:57.967: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down
*May  8 03:50:57.975: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*May  8 03:50:58.959: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*May  8 03:50:58.967: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*May  8 03:50:58.995: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*May  8 03:50:59.003: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*May  8 03:50:59.011: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*May  8 03:50:59.995: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*May  8 03:51:00.003: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*May  8 03:51:00.031: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*May  8 03:51:01.031: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*May  8 03:51:06.955: %CAPWAP-3-ERRORLOG: Selected MWAR 'WLC-01'(index 0).
*May  8 03:51:06.955: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*May  8 03:50:57.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.141.80.35 peer_port: 5246
*May  8 03:50:57.447: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.141.80.35 peer_port: 5246
*May  8 03:50:57.447: %CAPWAP-5-SENDJOIN: sending Join Request to 10.141.80.35
*May  8 03:50:57.451: %CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.
*May  8 03:50:57.451: %CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state 5.
*May  8 03:50:57.451: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller
*May  8 03:50:57.451: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from 10.141.80.35
*May  8 03:50:57.911: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*May  8 03:50:57.979: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*May  8 03:50:58.051: ac_first_hop_mac - IP:10.141.84.1 Hop IP:10.141.84.1 IDB:BVI1
*May  8 03:50:58.051: Setting AC first hop MAC: 0000.0c07.ac28

*May  8 03:50:58.051: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*May  8 03:50:58.083: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller WLC-01
*May  8 03:50:58.231: %WIDS-6-ENABLED: IDS Signature is loaded and enabled
*May  8 03:50:58.911: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*May  8 03:50:58.955: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to down
*May  8 03:50:58.963: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*May  8 03:50:59.051: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*May  8 03:50:59.943: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*May  8 03:50:59.987: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*May  8 03:50:59.995: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*May  8 03:51:00.003: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*May  8 03:51:00.987: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*May  8 03:51:00.995: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*May  8 03:51:01.027: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*May  8 03:51:02.027: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*May  8 03:51:14.411: %CLEANAIR-6-STATE: Slot 0 enabled
*May  8 03:51:16.467: %CLEANAIR-6-STATE: Slot 1 enabled

[Rasika Nayanajith]

AP & WLC management can be in different subnet (infact that is one of a best practice).

What is the WLC software version you running ? If it is below 7.4.x, go for at least 7.4.121.0 code with FUS 1.9.0.0

HTH

Rasika

**** Pls rate all useful responses ****

[ALIAOF_]

If they are in a different subnet won't I have the same issue with AP's not joining?  I believe I'll need to specify DHCP options on the router so that I don't encounter this issue with AP's not joining. Currently I have 7.4.100 with FUS of 1.7.0.0.

Working on a schedule to upgrade all the controllers to 7.4.121 but I was thinking about 1.8.0.0 any specific reason you mentioned 1.9.0.0?

Also no the best practice part, is there a specific reason this is a best practice?

[Scott Fella · Accepted solution]

For smaller deployments, putting AP's and the WLC on the same subnet is okay.  Large deployments, I would seperate them.  Once the AP joins, then you can move them to any subnet as they already know of the WLC ip.  Option 43, DNS, boradcast forwarding is only really needs for discovery of the WLC for new access points.  v7.4.121.1 is a stable code but I would also upgrade the FUS to 1.9.0.0 if possible.  This does take around 35-45 munutes.  

If your ap's that are havinf issues are specific to a location or maybe a switch, then that might be the issue.... connectivy somehow is breaking.  Uptime shows the power up time of the AP, so there is no reboot.  Join time, shows you how long it has been joined.  I have customers with the same setup and no issues.

Please rate helpful post and Cisco Support Community will donate to Kiva

Scotty

[ALIAOF_]

Thank you Scott yes as you mentioned AP's were already joined to the controller so the DHCP options were not needed any more.  Yes I have a huge deployment coming up and going to need a /23 just for the AP's.

On the 7.4.121 code found out that the one we are running now is no longer the Cisco's suggested IOS so need to schedule these upgrades.  Yes FUS upgrades are definitely long and very annoying lol.

[Scott Fella]

You can use multiple /24 subnets if you want.  That might be better than just creating one large subnet that might grow again. 

Please rate helpful post and Cisco Support Community will donate to Kiva

Scotty

[jmeachum]

It is not recommended to have more then 60-80 access points on a vlan.  If you do then I would strongly recommend configuring a unicast address for syslog messages from the access points.

From the WLC CLI console:

config ap syslog host global a.b.c.d

I recommend that a.b.c.d is routed to a null 0 or to a real syslog server.  If you use something like 1.2.3.4 then you could be sending your syslog messages to the internet.

[ALIAOF_]

This is news to me and not something I have ever heard before.  Also why would I want to send syslog messages from the AP going to an IP that is routed to null 0?

[jmeachum]

Just to give you control over where the packets get dropped.

[jmeachum]

I guess it would also be helpful if I pointed out that if you do not configure the global unicast syslog address that all of the syslog messages will go out as broadcast messages.

[Leo Laohoo]

*May  8 03:50:58.083: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller WLC-01

This is the only time I can see the AP join the WLC.  Tell us, what is the "up time" of the AP once joined with the WLC?  

[ALIAOF_]

Up time:

23 d, 20 h 08 m 59 s