Solved: Re: Trouble getting Cisco 2600 Series AP to stay joined to WLC 5

gregalink · ‎12-14-2013

Hi,

I have recently been tasked with upgrading our old Autonomous APs to LWAPs. We have a 5508 WLC at our Virtual Co-Lo and I am using Flexconnect to accomadate local switching and dhcp at our sites. I have upgraded over 50 APs and joined them to the controller. These include only 1130AG and 1240AG models. However they are working flawlessly and staying connected to the controller. The issue I'm having is with a new batch of 2600 series APs staying connected to the controller. I have attempted to do research into what may be causing the disconnects but have yet to find a solution. I am using DNS to resolve the CAPWAP & LWAPP queries from the APs to the controller accross our WAN. In reading other posts I thought it may be an issue with packets getting dropped but have had our Vendor who manages Sonicwalls at both ends of the WAN confirm for me there is no packet loss. Below are logs I gathered using puttty from the AP & WLC. Any help would be greatly appreciated.

AP I'm doing the testing on:

NAME: "AP2600", DESCR: "Cisco Aironet 2600 Series (IEEE 802.11n) Access Point"

PID: AIR-CAP2602I-A-K9 , VID: V01, SN: FTX1740J8V1

WLC in question:

Manufacturer's Name.............................. Cisco Systems Inc.

Product Name..................................... Cisco Controller

Product Version.................................. 7.3.112.0

Bootloader Version............................... 1.0.1

Field Recovery Image Version..................... 6.0.182.0

Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27

Build Type....................................... DATA + WPS

System Name...................................... wificontroller

System Location.................................. Corp

System Contact................................... Net Engineer

System ObjectID.................................. 1.3.6.1.4.1.9.1.1069

Redundancy Mode.................................. Disabled

IP Address....................................... 10.250.32.8

Last Reset....................................... Software reset

System Up Time................................... 190 days 3 hrs 34 mins 24 secs

System Timezone Location......................... (GMT -5:00) Eastern Time (US and Canada)

Configured Country............................... US - United States

Operating Environment............................ Commercial (0 to 40 C)

Internal Temp Alarm Limits....................... 0 to 65 C

--More-- or (q)uit

Internal Temperature............................. +38 C

External Temperature............................. +20 C

Fan Status....................................... OK

State of 802.11b Network......................... Enabled

State of 802.11a Network......................... Enabled

Number of WLANs.................................. 14

Number of Active Clients......................... 71

Burned-in MAC Address............................ C8:9C:1D:8C:52:E0

Power Supply 1................................... Present, OK

Power Supply 2................................... Absent

Maximum number of APs supported.................. 100

Here is the output that keeps on occuring as the AP joins the WLC for a brief time and then changes to standalone mode

WT-4thFlr-AP3#

*Dec 14 15:42:04.419: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST

., 3)

*Dec 14 15:42:11.443: %EVT-4-WRN: Write of flash:/event.capwap done

*Dec 14 15:42:11.483: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode

*Dec 14 15:42:11.487: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE

*Dec 14 15:42:11.487: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.250.32.8:5246

*Dec 14 15:42:11.571: %WIDS-6-DISABLED: IDS Signature is removed and disabled.

*Dec 14 15:42:21.575: %CAPWAP-3-ERRORLOG: Selected MWAR 'wificontroller'(index 0).

*Dec 14 15:42:21.575: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Dec 14 15:42:12.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.250.32.8 peer_port: 5246

*Dec 14 15:42:14.303: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.250.32.8 peer_port: 5246

*Dec 14 15:42:14.303: %CAPWAP-5-SENDJOIN: sending Join Request to 10.250.32.8

*Dec 14 15:42:15.127: Starting Ethernet promiscuous mode

*Dec 14 15:42:15.535: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash

*Dec 14 15:42:15.667: ac_first_hop_mac - IP:10.1.2.250 Hop IP:10.1.2.250 IDB:BVI1

*Dec 14 15:42:15.667: Setting AC first hop MAC: 0017.c575.a23c

*Dec 14 15:42:15.855: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller wificontroller

*Dec 14 15:42:15.911: %LWAPP-4-CLIENTEVENTLOG: No Flex ACL map configuration file to load. Connect to controller to get configuration file

*Dec 14 15:42:15.911: %LWAPP-4-CLIENTEVENTLOG: No LS Flex ACL map configuration file to load. Connect to controller to get configuration file

*Dec 14 15:42:15.915: %LWAPP-4-CLIENTEVENTLOG: No Central Dhcp map configuration file to load. Connect to controller to get configuration file

*Dec 14 15:42:15.915: %LWAPP-3-CLIENTERRORLOG: Switching to Connected mode

*Dec 14 15:42:23.639: %WIDS-6-ENABLED: IDS Signature is loaded and enabled

*Dec 14 15:42:34.615: %CLEANAIR-6-STATE: Slot 0 disabled

*Dec 14 15:42:34.615: %CLEANAIR-6-STATE: Slot 1 disabled

*Dec 14 15:45:43.783: %CAPWAP-3-ERRORLOG: Retransmission count for packet exceeded max(CAPWAP_WTP_EVENT_REQUEST

., 11)

*Dec 14 15:45:43.787: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode

*Dec 14 15:45:43.787: %CAPWAP-3-ERRORLOG: GOING BACK TO DISCOVER MODE

*Dec 14 15:45:43.787: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.250.32.8:5246

*Dec 14 15:45:43.867: %WIDS-6-DISABLED: IDS Signature is removed and disabled.

*Dec 14 15:45:53.867: %CAPWAP-3-ERRORLOG: Selected MWAR 'wificontroller'(index 0).

*Dec 14 15:45:53.867: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Dec 14 15:45:44.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.250.32.8 peer_port: 5246

*Dec 14 15:45:46.315: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 10.250.32.8 peer_port: 5246

*Dec 14 15:45:46.315: %CAPWAP-5-SENDJOIN: sending Join Request to 10.250.32.8

*Dec 14 15:45:46.487: Starting Ethernet promiscuous mode

*Dec 14 15:45:49.903: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash

*Dec 14 15:45:50.031: ac_first_hop_mac - IP:10.1.2.250 Hop IP:10.1.2.250 IDB:BVI1

*Dec 14 15:45:50.031: Setting AC first hop MAC: 0017.c575.a23c

Here are the results of debug capwap client event on the AP:

WT-4thFlr-AP3#debug capwap client event

CAPWAP Client EVENT display debugging is on

WT-4thFlr-AP3#

*Dec 14 15:54:58.335: %CAPWAP-3-EVENTLOG: Echo Interval Expired.

*Dec 14 15:54:58.335: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Dec 14 15:54:58.335: %CAPWAP-3-EVENTLOG: Echo Request sent to 10.250.32.8

*Dec 14 15:54:58.343: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0

*Dec 14 15:54:58.343: %CAPWAP-3-EVENTLOG: Queue Empty.

*Dec 14 15:54:58.343: %CAPWAP-3-EVENTLOG: Echo Response from 10.250.32.8

*Dec 14 15:55:08.000: %CAPWAP-3-EVENTLOG: Setting time to 15:55:08 UTC Dec 14 2013

*Dec 14 15:55:25.579: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Dec 14 15:55:25.587: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0

*Dec 14 15:55:25.587: %CAPWAP-3-EVENTLOG: Queue Empty.

*Dec 14 15:55:25.587: %CAPWAP-3-EVENTLOG: Wtp Event Response from 10.250.32.8

*Dec 14 15:55:25.827: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Dec 14 15:55:25.835: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0

*Dec 14 15:55:25.835: %CAPWAP-3-EVENTLOG: Queue Empty.

*Dec 14 15:55:25.835: %CAPWAP-3-EVENTLOG: Wtp Event Response from 10.250.32.8

*Dec 14 15:55:55.835: %CAPWAP-3-EVENTLOG: Echo Interval Expired.

*Dec 14 15:55:55.835: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Dec 14 15:55:55.835: %CAPWAP-3-EVENTLOG: Echo Request sent to 10.250.32.8

*Dec 14 15:55:55.843: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0

*Dec 14 15:55:55.843: %CAPWAP-3-EVENTLOG: Queue Empty.

*Dec 14 15:55:55.843: %CAPWAP-3-EVENTLOG: Echo Response from 10.250.32.8

*Dec 14 15:55:56.000: %CAPWAP-3-EVENTLOG: Setting time to 15:55:56 UTC Dec 14 2013

*Dec 14 15:56:25.735: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Dec 14 15:56:25.743: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0

*Dec 14 15:56:25.743: %CAPWAP-3-EVENTLOG: Queue Empty.

*Dec 14 15:56:25.743: %CAPWAP-3-EVENTLOG: Wtp Event Response from 10.250.32.8

*Dec 14 15:56:25.983: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Dec 14 15:56:25.991: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0

*Dec 14 15:56:25.991: %CAPWAP-3-EVENTLOG: Queue Empty.

*Dec 14 15:56:25.991: %CAPWAP-3-EVENTLOG: Wtp Event Response from 10.250.32.8

*Dec 14 15:56:55.991: %CAPWAP-3-EVENTLOG: Echo Interval Expired.

*Dec 14 15:56:55.991: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Dec 14 15:56:55.991: %CAPWAP-3-EVENTLOG: Echo Request sent to 10.250.32.8

*Dec 14 15:56:55.999: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0

*Dec 14 15:56:55.999: %CAPWAP-3-EVENTLOG: Queue Empty.

*Dec 14 15:56:55.999: %CAPWAP-3-EVENTLOG: Echo Response from 10.250.32.8

*Dec 14 15:56:56.000: %CAPWAP-3-EVENTLOG: Setting time to 15:56:56 UTC Dec 14 2013

Here are the results of debug capwap client packet detail:

WT-4thFlr-AP3#

*Dec 14 15:59:01.823: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:01.823: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246

*Dec 14 15:59:01.823: Msg Type : CAPWAP_ECHO_REQUEST

*Dec 14 15:59:01.823: Msg Length : 0

*Dec 14 15:59:01.823: Msg SeqNum : 44

*Dec 14 15:59:01.823: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:01.831: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:01.831: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246

*Dec 14 15:59:01.831: HLEN 2, Radio ID 0, WBID 1

*Dec 14 15:59:01.831: Msg Type : CAPWAP_ECHO_RESPONSE

*Dec 14 15:59:01.831: Msg Length : 15

*Dec 14 15:59:01.831: Msg SeqNum : 44

*Dec 14 15:59:01.831:

*Dec 14 15:59:01.831: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 11

*Dec 14 15:59:01.831: Vendor Identifier : 0x00409600

*Dec 14 15:59:01.831:

IE : UNKNOWN IE 151

*Dec 14 15:59:01.831: IE Length : 5

*Dec 14 15:59:01.831: Decode routine not available, Printing Hex Dump

*Dec 14 15:59:01.831:

52 AC 80 46 00

*Dec 14 15:59:01.831: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:20.931: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:20.931: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246

*Dec 14 15:59:20.931: HLEN 2, Radio ID 0, WBID 1

*Dec 14 15:59:20.931: Msg Type : CAPWAP_CONFIGURATION_UPDATE_REQUEST

*Dec 14 15:59:20.931: Msg Length : 93

*Dec 14 15:59:20.931: Msg SeqNum : 38

*Dec 14 15:59:20.931:

*Dec 14 15:59:20.931: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 89

*Dec 14 15:59:20.931: Vendor Identifier : 0x00409600

*Dec 14 15:59:20.931:

IE : RRM_NEIGHBOR_CTRL_PAYLOAD

*Dec 14 15:59:20.931: IE Length : 83

*Dec 14 15:59:20.931: Decode routine not available, Printing Hex Dump

*Dec 14 15:59:20.931:

00 0A FA 20 08 01 F4 00 07 0A FA 20 08 03 00 01

01 00 3C 00 B4 2E 06 2E E7 B4 94 51 B2 C7 79 25

22 FD BE 04 F6 00 00 00 00 00 00 00 00 4F 50 52

53 2D 57 69 46 69 00 00 00 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00 00 00 00 00 00 01 06 0B

01 01 01

*Dec 14 15:59:20.931: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:20.931: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:20.931: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246

*Dec 14 15:59:20.931: Msg Type : CAPWAP_CONFIGURATION_UPDATE_RESPONSE

*Dec 14 15:59:20.931: Msg Length : 8

*Dec 14 15:59:20.931: Msg SeqNum : 38

*Dec 14 15:59:20.931:

*Dec 14 15:59:20.931: Type : CAPWAP_MSGELE_RESULT_CODE, Length 4

*Dec 14 15:59:20.931: Result Code : CAPWAP_SUCCESS

*Dec 14 15:59:20.931: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:21.139: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:21.139: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246

*Dec 14 15:59:21.139: HLEN 2, Radio ID 0, WBID 1

*Dec 14 15:59:21.139: Msg Type : CAPWAP_CONFIGURATION_UPDATE_REQUEST

*Dec 14 15:59:21.139: Msg Length : 111

*Dec 14 15:59:21.139: Msg SeqNum : 39

*Dec 14 15:59:21.139:

*Dec 14 15:59:21.139: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 107

*Dec 14 15:59:21.139: Vendor Identifier : 0x00409600

*Dec 14 15:59:21.139:

IE : RRM_NEIGHBOR_CTRL_PAYLOAD

*Dec 14 15:59:21.139: IE Length : 101

*Dec 14 15:59:21.139: Decode routine not available, Printing Hex Dump

*Dec 14 15:59:21.143:

01 0A FA 20 08 01 F4 00 07 0A FA 20 08 0C 00 01

01 00 3C 00 B4 2E 06 2E E7 B4 94 51 B2 C7 79 25

22 FD BE 04 F6 00 00 00 00 00 00 00 00 4F 50 52

53 2D 57 69 46 69 00 00 00 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00 00 00 00 00 00 24 28 2C

30 34 38 3C 40 95 99 9D A1 01 01 01 01 01 01 01

01 01 01 01 01

*Dec 14 15:59:21.143: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:21.143: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:21.143: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246

*Dec 14 15:59:21.143: Msg Type : CAPWAP_CONFIGURATION_UPDATE_RESPONSE

*Dec 14 15:59:21.143: Msg Length : 8

*Dec 14 15:59:21.143: Msg SeqNum : 39

*Dec 14 15:59:21.143:

*Dec 14 15:59:21.143: Type : CAPWAP_MSGELE_RESULT_CODE, Length 4

*Dec 14 15:59:21.143: Result Code : CAPWAP_SUCCESS

*Dec 14 15:59:21.143: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:25.547: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:25.547: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246

*Dec 14 15:59:25.547: Msg Type : CAPWAP_WTP_EVENT_REQUEST

*Dec 14 15:59:25.547: Msg Length : 14

*Dec 14 15:59:25.547: Msg SeqNum : 45

*Dec 14 15:59:25.547:

*Dec 14 15:59:25.547: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 10

*Dec 14 15:59:25.547: Vendor Identifier : 0x00409600

*Dec 14 15:59:25.547:

IE : RRM_LOAD_DATA_PAYLOAD

*Dec 14 15:59:25.547: IE Length : 4

*Dec 14 15:59:25.547: slot 0 rxLoad 0 txLoad 0 ccaLoad 33

*Dec 14 15:59:25.547: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:25.555: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:25.555: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246

*Dec 14 15:59:25.555: HLEN 2, Radio ID 0, WBID 1

*Dec 14 15:59:25.555: Msg Type : CAPWAP_WTP_EVENT_RESPONSE

*Dec 14 15:59:25.555: Msg Length : 0

*Dec 14 15:59:25.555: Msg SeqNum : 45

*Dec 14 15:59:25.555: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:25.795: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:25.795: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246

*Dec 14 15:59:25.795: Msg Type : CAPWAP_WTP_EVENT_REQUEST

*Dec 14 15:59:25.795: Msg Length : 14

*Dec 14 15:59:25.795: Msg SeqNum : 46

*Dec 14 15:59:25.795:

*Dec 14 15:59:25.795: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 10

*Dec 14 15:59:25.795: Vendor Identifier : 0x00409600

*Dec 14 15:59:25.795:

IE : RRM_LOAD_DATA_PAYLOAD

*Dec 14 15:59:25.795: IE Length : 4

*Dec 14 15:59:25.795: slot 1 rxLoad 0 txLoad 0 ccaLoad 0

*Dec 14 15:59:25.795: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:25.803: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:25.803: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246

*Dec 14 15:59:25.803: HLEN 2, Radio ID 0, WBID 1

*Dec 14 15:59:25.803: Msg Type : CAPWAP_WTP_EVENT_RESPONSE

*Dec 14 15:59:25.803: Msg Length : 0

*Dec 14 15:59:25.803: Msg SeqNum : 46

*Dec 14 15:59:25.803: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:30.375: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:30.375: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246

*Dec 14 15:59:30.375: HLEN 2, Radio ID 0, WBID 1

*Dec 14 15:59:30.375: Msg Type : CAPWAP_CONFIGURATION_UPDATE_REQUEST

*Dec 14 15:59:30.375: Msg Length : 17

*Dec 14 15:59:30.375: Msg SeqNum : 40

*Dec 14 15:59:30.375:

*Dec 14 15:59:30.375: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 13

*Dec 14 15:59:30.375: Vendor Identifier : 0x00409600

SlotId : 0

Mobile Mac Addr : BC:52:B7:E3:17:CB

*Dec 14 15:59:30.375: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:30.375: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:30.375: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246

*Dec 14 15:59:30.375: Msg Type : CAPWAP_CONFIGURATION_UPDATE_RESPONSE

*Dec 14 15:59:30.379: Msg Length : 8

*Dec 14 15:59:30.379: Msg SeqNum : 40

*Dec 14 15:59:30.379:

*Dec 14 15:59:30.379: Type : CAPWAP_MSGELE_RESULT_CODE, Length 4

*Dec 14 15:59:30.379: Result Code : CAPWAP_SUCCESS

*Dec 14 15:59:30.379: <<<< End of CAPWAP Packet >>>>

*Dec 14 15:59:30.387: <<<< Start of CAPWAP Packet >>>>

*Dec 14 15:59:30.387: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246

*Dec 14 15:59:30.387: HLEN 2, Radio ID 0, WBID 1

*Dec 14 15:59:30.387: Msg Type : CAPWAP_WTP_EVENT_RESPONSE

*Dec 14 15:59:30.387: Msg Length : 0

*Dec 14 15:59:30.387: Msg SeqNum : 47

*Dec 14 15:59:30.387: <<<< End of CAPWAP Packet >>>>

*Dec 14 16:00:00.387: <<<< Start of CAPWAP Packet >>>>

*Dec 14 16:00:00.387: CAPWAP Control mesg Sent to 10.250.32.8, Port 5246

*Dec 14 16:00:00.387: Msg Type : CAPWAP_ECHO_REQUEST

*Dec 14 16:00:00.387: Msg Length : 0

*Dec 14 16:00:00.387: Msg SeqNum : 48

*Dec 14 16:00:00.387: <<<< End of CAPWAP Packet >>>>

*Dec 14 16:00:00.395: <<<< Start of CAPWAP Packet >>>>

*Dec 14 16:00:00.395: CAPWAP Control mesg Recd from 10.250.32.8, Port 5246

*Dec 14 16:00:00.395: HLEN 2, Radio ID 0, WBID 1

*Dec 14 16:00:00.395: Msg Type : CAPWAP_ECHO_RESPONSE

*Dec 14 16:00:00.395: Msg Length : 15

*Dec 14 16:00:00.395: Msg SeqNum : 48

*Dec 14 16:00:00.395:

*Dec 14 16:00:00.395: Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 11

*Dec 14 16:00:00.395: Vendor Identifier : 0x00409600

*Dec 14 16:00:00.395:

IE : UNKNOWN IE 151

*Dec 14 16:00:00.395: IE Length : 5

*Dec 14 16:00:00.395: Decode routine not available, Printing Hex Dump

*Dec 14 16:00:00.395:

52 AC 80 81 00

*Dec 14 16:00:00.395: <<<< End of CAPWAP Packet >>>>

Saravanan Lakshmanan · ‎12-19-2013

try reducing mtu on ap 2600.

View solution in original post

Leo Laohoo · ‎12-14-2013

Since your WLC is set up for FlexConnect, can you please check your AP Policies and see if the 2600 is in the list or not?

gregalink · ‎12-14-2013

Under my AP Policies I only have "Accept Manufactured Installed Certificate (MIC)" checked. I attempted to add the AP based on MAC Address (c0:67:af:6f:25:70) with this certificate type but still have the same issue. I then ran the following debug on my controller and this is the output I recieve regarding that MAC. I tried to cut the output short because it get's somewhat redundant but was unsure what exactly to look for in the output. Should I be selecting a different certificate type? I am somewhat new to wireless technologies but doing my best to pick things up so if this seems trivial please forgive my ignorance.

debug pm pki enable

*sshpmLscTask: Dec 14 20:42:56.450: sshpmLscTask: LSC Task received a message 4

*spamApTask6: Dec 14 20:42:58.840: sshpmGetIssuerHandles: locking ca cert table

*spamApTask6: Dec 14 20:42:58.841: sshpmGetIssuerHandles: calling x509_alloc() for user cert

*spamApTask6: Dec 14 20:42:58.841: sshpmGetIssuerHandles: calling x509_decode()

*spamApTask6: Dec 14 20:42:58.845: sshpmGetIssuerHandles: C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AP3G2-c067af6f2570, MAILTO=support@cisco.com

*spamApTask6: Dec 14 20:42:58.845: sshpmGetIssuerHandles: O=Cisco Systems, CN=Cisco Manufacturing CA

*spamApTask6: Dec 14 20:42:58.845: sshpmGetIssuerHandles: Mac Address in subject is c0:67:af:6f:25:70

*spamApTask6: Dec 14 20:42:58.845: sshpmGetIssuerHandles: Cert Name in subject is AP3G2-c067af6f2570

*spamApTask6: Dec 14 20:42:58.845: sshpmGetIssuerHandles: Cert is issued by Cisco Systems.

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: called to evaluate

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 0, CA cert >bsnOldDefaultCaCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 1, CA cert >bsnDefaultRootCaCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 2, CA cert >bsnDefaultCaCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 3, CA cert >bsnDefaultBuildCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 4, CA cert >cscoDefaultNewRootCaCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCID: comparing to row 5, CA cert >cscoDefaultMfgCaCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: called to get cert for CID 282aef7e

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 0, certname >bsnOldDefaultCaCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 1, certname >bsnDefaultRootCaCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 2, certname >bsnDefaultCaCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 3, certname >bsnDefaultBuildCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 4, certname >cscoDefaultNewRootCaCert<

*spamApTask6: Dec 14 20:42:58.845: sshpmGetCertFromCID: comparing to row 5, certname >cscoDefaultMfgCaCert<

*spamApTask6: Dec 14 20:42:58.845: ssphmUserCertVerify: calling x509_decode()

*spamApTask6: Dec 14 20:42:58.856: ssphmUserCertVerify: user cert verfied using >cscoDefaultMfgCaCert<

*spamApTask6: Dec 14 20:42:58.856: sshpmGetIssuerHandles: ValidityString (current): 2013/12/15/01:42:58

*spamApTask6: Dec 14 20:42:58.856: sshpmGetIssuerHandles: ValidityString (NotBefore): 2013/08/25/13:01:22

*spamApTask6: Dec 14 20:42:58.856: sshpmGetIssuerHandles: ValidityString (NotAfter): 2023/08/25/13:11:22

*spamApTask6: Dec 14 20:42:58.856: sshpmGetIssuerHandles: getting cisco ID cert handle...

*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: called to evaluate

*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 0, CA cert >bsnOldDefaultCaCert<

*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 1, CA cert >bsnDefaultRootCaCert<

*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 2, CA cert >bsnDefaultCaCert<

*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 3, CA cert >bsnDefaultBuildCert<

*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 4, CA cert >cscoDefaultNewRootCaCert<

*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 5, CA cert >cscoDefaultMfgCaCert<

*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 0, ID cert >bsnOldDefaultIdCert<

*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 1, ID cert >bsnDefaultIdCert<

*spamApTask6: Dec 14 20:42:58.856: sshpmGetCID: comparing to row 2, ID cert >cscoDefaultIdCert<

*spamApTask6: Dec 14 20:42:58.857: sshpmFreePublicKeyHandle: called with 0x2c5f0cb8

*spamApTask6: Dec 14 20:42:58.857: sshpmFreePublicKeyHandle: freeing public key

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: called to evaluate

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 0, CA cert >bsnOldDefaultCaCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 1, CA cert >bsnDefaultRootCaCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 2, CA cert >bsnDefaultCaCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 3, CA cert >bsnDefaultBuildCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 4, CA cert >cscoDefaultNewRootCaCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 5, CA cert >cscoDefaultMfgCaCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 0, ID cert >bsnOldDefaultIdCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 1, ID cert >bsnDefaultIdCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: comparing to row 2, ID cert >cscoDefaultIdCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: called to get cert for CID 183fd2b6

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 0, certname >bsnOldDefaultCaCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 1, certname >bsnDefaultRootCaCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 2, certname >bsnDefaultCaCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 3, certname >bsnDefaultBuildCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 4, certname >cscoDefaultNewRootCaCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 5, certname >cscoDefaultMfgCaCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 0, certname >bsnOldDefaultIdCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 1, certname >bsnDefaultIdCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCertFromCID: comparing to row 2, certname >cscoDefaultIdCert<

*spamApTask0: Dec 14 20:43:17.451: sshpmGetCID: called to evaluate

Leo Laohoo · ‎12-14-2013

*Dec 14 15:45:49.903: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash

If the AP is currently in the same location as the WLC, disable FlexConnect and switch the AP into local. See what happens.

gregalink · ‎12-14-2013

The AP is at one of our nursing facilities and the WLC is at our Co-Lo in another building connected via WAN. We only have the one controller and this AP came straight out of a box so has not been associated with any other controllers. Any other suggestions or useful debugs I could try to determine the problem?

gregalink · ‎12-15-2013

I'm not sure if this is helpful but here is the output I recieved from the following show command. The high number of link failures is what stands out to me but still can't determine what is causing this.

WT-4thFlr-AP3#term mon

WT-4thFlr-AP3#show capwap client config

configMagicMark 0xF1E2D3C4

chkSumV2 27340

chkSumV1 10268

swVer 7.3.112.0

adminState ADMIN_ENABLED(1)

name WT-4thFlr-AP3

location WT 4th Floor AP3

group name WestminsterThurber

mwarName wificontroller

mwarIPAddress 10.250.32.8

mwarName

mwarIPAddress 0.0.0.0

mwarName

mwarIPAddress 0.0.0.0

ssh status Enabled

Telnet status Disabled

numOfSlots 2

spamRebootOnAssert 1

spamStatTimer 180

randSeed 0x2082

transport SPAM_TRANSPORT_L3(2)

transportCfg SPAM_TRANSPORT_DEFAULT(0)

initialisation SPAM_PRODUCTION_DISCOVERY(1)

--More-- ApMode FlexConnect

ApSubMode Not Configured

AP Rogue Detection Mode Enabled

OfficeExtend AP [0] Disabled

OfficeExtend AP JoinMode[0] Standard

Discovery Timer 10 secs

Heart Beat Timer 30 secs

Led State Enabled 1

Primed Interval 0

AP ILP Pre-Standard Switch Support Disabled

AP Power Injector Disabled

Infrastructure MFP validation Disabled

Configured Switch 1 Addr 10.250.32.8

non-occupancy channels:

Ethernet (Duplex/Speed) auto/auto

Slot 0

adminstate ADMIN_ENABLED(1)

radioType RADIO_TYPE_80211bg

CleanAirAdminState Enabled

countryCode US

countryISOCode US

chanAutoCfg CONFIG_AUTO

--More-- channel 6

channel width 20

extension channel none

txPowerAutoCfg CONFIG_AUTO

txPowerLevel 2

diversitySelection DIVERSITY_ENABLED

htRxAntennaSelection 0F

htTxAntennaSelection 0F

beamformCfg 03

Antenna Mode ANTENNA_OMNI

antennaSelection_0 INTERNAL_ANTENNA

antennaSelection_1 INTERNAL_ANTENNA

antennaSelection_2 INTERNAL_ANTENNA

antennaSelection_3 INTERNAL_ANTENNA

twiceExtAntennaGain 0

Profile Mode CONFIG_AUTO

Load Profile

rfBusyThreshold 0

numClientsThreshold 0

bytesPerSecThreshold 0

Interference Profile

InterferenceThreshold 0

Noise Profile

--More-- NoiseThreshold 0

Coverage Profile

SNRThreshold 0

ExceptionThreshold 0

minClientsThreshold 0

11gSupport Enabled

override mode Disabled

CCX RM Mode CONFIG_AUTO

CCX RM Config

rm state 0

rm meas interval 0

rts enabled 0

rts threshold 2347

LOMM optimization 0

LOMM Number of Channels 0

channel[0] = 0, channel[1] = 0, channel[2] = 0, channel[3] = 0,

FMC HS AP Type is 0, Threshhold 0

Slot 1

adminstate ADMIN_ENABLED(1)

radioType RADIO_TYPE_80211a

CleanAirAdminState Enabled

countryCode US

countryISOCode US

--More-- chanAutoCfg CONFIG_AUTO

channel 44

channel width 20

extension channel none

txPowerAutoCfg CONFIG_AUTO

txPowerLevel 1

diversitySelection DIVERSITY_ENABLED

htRxAntennaSelection 0F

htTxAntennaSelection 0F

beamformCfg 03

Antenna Mode ANTENNA_OMNI

antennaSelection_0 INTERNAL_ANTENNA

antennaSelection_1 INTERNAL_ANTENNA

antennaSelection_2 INTERNAL_ANTENNA

antennaSelection_3 INTERNAL_ANTENNA

twiceExtAntennaGain 0

Profile Mode CONFIG_AUTO

Load Profile

rfBusyThreshold 0

numClientsThreshold 0

bytesPerSecThreshold 0

Interference Profile

InterferenceThreshold 0

--More-- Noise Profile

NoiseThreshold 0

Coverage Profile

SNRThreshold 0

ExceptionThreshold 0

minClientsThreshold 0

11gSupport Disabled

override mode Disabled

CCX RM Mode CONFIG_AUTO

CCX RM Config

rm state 0

rm meas interval 0

rts enabled 0

rts threshold 2347

LOMM optimization 0

LOMM Number of Channels 0

channel[0] = 0, channel[1] = 0, channel[2] = 0, channel[3] = 0,

FMC HS AP Type is 0, Threshhold 0

AP failure counters: LinkFailure = 277, SpamReboots = 4, ApCrashes = 0

AP join priority = 1

--More-- AP lsc enable = 0

AP lsc reboot cnt = 0

AP lsc max num of Retry = 0

Mesh AP lsc enable = 0

AP retransmit count = 255

AP retransmit timer = 255

AP vlan tag status = Disabled

SSC Controller Hash validation enabled.

Leo Laohoo · ‎12-15-2013

Get this AP back to your facility where the WLC is located. I want to determine whether or not you've got routing issue or MESH IOS is incorrectly loaded into your AP.

Put the AP in the same subnet as your WLC and see if the AP joins.

gregalink · ‎12-16-2013

Ok, umm... there is no way for me to currently move the AP to our Virtual CoLo which is not at the main office. It's in a highly secure data center and I would need to schedule time in advance and not even sure I could hook up the AP inside of the facility. However, I do have a number of this model AP still in the box and so I went ahead and deployed one this morning at our Corporate Facility which is connected accross the WAN like all of our other sites to the Colo. The AP has been up for over 50 minutes and has not disassociated with the controller even once. The few of this same model I have deployed at our other sites have only stayed connected for a couple of minutes at a time.

I have an idea of why this AP may be working here and not at our other locations. Our Corporate office is the only site where the business WLANs are VLANed off from the wired network the guest is of course VLANed everywhere. Now before you start chastising me I have plans to VLAN off the Business wireless at all our sites from the Wired network but I wanted to update all of the APs first namely to address some other wireless issues. So at all of our other sites except Corporate the Business Wireless shares from the same pool as the Business Wired which is VLAN1 by default. So when I created my interfaces on the WLC I did so on the corresponding wired networks. This is the only difference I know of between our Corporate Office and the other sites.

This was not readily apparant to me as a problem at first because the existing 1130AG & 1240AG APs didn't appear to have this issue. I do have most of the APs updated now and was planning on setting up the additoinal networks & VLANs but wasn't sure if this could indeed be causing the issue.

Just remember I inhereted most of this stuff. Luckily I have my CCNA - Routing & Switching Cert now and have some idea of how these should have been properly setup.

Your thoughts?

Leo Laohoo · ‎12-16-2013

Now before you start chastising me I have plans to VLAN off the Business wireless at all our sites from the Wired network but I wanted to update all of the APs first namely to address some other wireless issues. So at all of our other sites except Corporate the Business Wireless shares from the same pool as the Business Wired which is VLAN1 by default.

No plan of doing that.

Two scenarios are spinning right now:

1. The AP in your Co-Loc could be loaded with MESH IOS;

2. You've got a potential routing problem.

Rasika Nayanajith · ‎12-16-2013

Hi Gregory,

Your issue is this AP came with mesh image.

If you cannot get this AP registered to your WLC in mesh mode, then try to reset the AP to factory settings & try to upload correct recovery image (ap3g2-rcvk9w8-tar.152-4.JA1.tar or any previous version) using the mode button option describe in below post.

Then your AP shoud be able to register to your 5508 as a local mode AP. Then change it to FlexConnect if that is the requirement.

http://mrncciew.com/2013/12/13/ap-conversion-using-mode-button/

Give it a try & see

HTH

Rasika

**** Pls rate all useful responses ****

gregalink · ‎12-16-2013

Thanks for the advice Leo and Rasika. I actually had my first experience today using a TFTP server to downgrade from Lightweight to Autonomous on an old AP with no certificate so this shouldn't be too difficult. I'll be at one of our facilities tomorrow and will have a chance to test this out. I'll let you know what the results are.

Saravanan Lakshmanan · ‎12-19-2013

try reducing mtu on ap 2600.

gregalink · ‎12-30-2013

Hi,

I was out of town for the Holidays and enjoyed some much needed time away from technology so sorry for my delay in getting back to you all. Here is what I have done so far. Rasika, I did change the recovery image on one of those APs to ap3g2-rcvk9w8-tar.152-4.JA1.tar at which time it reloaded the recovery image then connected to the controller and updated to the following version ap3g2-rcvk9w8-tar.152-2.JA1.tar. Once the AP rebooted with the new image from the controller I then ran into the same issue. I haven't had an oppurtunity to attempt reducing the MTU's on the device yet but plan to try that next. What should I change the MTU value to?

Thanks,

Rasika Nayanajith · ‎12-30-2013

Hi Gregory,

Pls attach the AP console output this time around to see what's happening.

If you are changing the AP MTU you can do it via below command & set it tio 1363 (TCP MSS size), I think you should have AP registered first to do this

WLC>config ap tcp-adjust-mss enable 1363

You can find more detail about this in below post

http://mrncciew.com/2013/04/07/configuring-tcp-mss/

HTH

Rasika

**** Pls rate all useful responses *****

gregalink · ‎12-30-2013

I had forgotten to save the putty log file from my console output when I was on site. I will be sure to attach it next time. Also, I believe the APs I already have registered should stay connected long enough for me to issue the above commands. Thanks again for the info.

Trouble getting Cisco 2600 Series AP to stay joined to WLC 5508