Max retransmissions to AP

[Ramprasad2]

HI Team,

i have Cisco 9800 model wlc and ap model 9120axi running version  17.9.3

issue :Max retransmissions to AP 

ap automatic dis join to wlc and automatic join agin

so can we enable in  link latency ? ( if yes what is impact )

can we Adjust MSS size incense ? ( if yes what is impact )

can we Enable Jumbo MTU ? ( if yes what is impact )

[Ramprasad2]

WLC model 9800-L-F ( running OS 17.9.3)

AP 9120Axid

in my production network approx 161 access points running but day by day any random access points dis join to wlc and how to identify that next 1 hour or 1 day or 1 minutes which ap dis join

so how to identify that how to start span for access point. 

[Rich R]

Fair enough - at least then the WLC port.

[Ramprasad2]

Hi

1 what is you opinion if we will check ( 

show platform qos queue config gigabitEthernet 1/0/1)  ?

2

#what is your opinion qos queue-softmax-multiplier 1200  ?

[Rich R]

I don't think that command even exists?  Either way I'm not familiar with it so can't comment.

Interestingly I found this bug which sounds like it could be related:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwd18473
The status is terminated which usually means TAC could not replicate the issue in lab.
Run radioactive trace on some of your APs and/or packet captures to check your WLC and AP logs for similar symptoms to what is described in the bug (ignore the rubbish stylesheet formatting somebody has carelessly pasted in there - I'll report that).  If you're seeing the same symptoms then open a TAC case and ask TAC to re-open that bug.  Make sure they link your case to that bug (they clearly didn't bother with the original case because it says 0 support cases!)

[Ramprasad2]

hi

we have used only PSK base auth. and in my case max retransmission to AP logs generate.

[Rich R]

Sorry I don't understand what you mean by that or how it's relevant to your question.
You'll need to explain what you mean.

PSK is a client authentication option used on a WLAN.

AP join is a function of the CAPWAP control connection between AP and WLC on UDP port 5246

[Ramprasad2]

as shared link https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwd18473 in this sowing dot1x. so my case i have use in wlan psk.

i know its working udp 5246 with CAPWAP.

i face max retransmission to ap but you shared link in mention below output is different

CAPWAP Retransmissions - CAPWAP_WTP_EVENT not received by AP
CSCwd18473  

Customer Visible

Notifications

Save Bug

Open Support Case

Description

Symptom: CAPWAP retries seen which intermittently leads to AP disconnections and additionally to slow roams. Noticed roaming delay is caused by WLC's state machine stuck at S_AUTHIF_ADD_MOBILE_ACK_WAIT_DOT1X stage for several seconds, even for 11r roaming events. From RA traces, we can isolate the delay to "T_AUTHIF_ACK_AWAIT_DOT1XAUTH_PUSH_RESP_80211R_ROAM": 2022/09/26 23:22:47.110422 {wncd_x_R0-7}{1}: [msc-fsm] [19457]: (debug): @msc_event {"entity":"8425.3fa4.cc4e/client_auth_interface_sm:17037289", "label":"S_AUTHIF_ADD_MOBILE_ACK_WAIT_DOT1X", "data":{"transition":"INIT_TO_1X_ADD_MOBILE_ACK_WAIT"}, "type":"CircleEvent", "color":"FFFF00", "radius":"0.7", "pop_source":"true", "dst":{"id":"$n_$p_0x7ffffa6d15e4", "type":"Transition", "straight":"true", "stroke_width":"2.0"}} 2022/09/26 23:22:49.215787 {wncd_x_R0-7}{1}: [msc-fsm] [19457]: (debug): @msc_event {"entity":"8425.3fa4.cc4e/client_auth_interface_sm:17037289", "label":"S_AUTHIF_ADD_MOBILE_ACK_WAIT_DOT1X", "data":{"transition":"T_AUTHIF_ACK_AWAIT_DOT1XAUTH_PUSH_RESP_80211R_ROAM"}, "type":"CircleEvent", "color":"FFFF00", "radius":"0.7", "push_source":"true", "src":{"id":"$n_$p_0x7ffffa6d15e0", "type":"Transition", "straight":"true", "stroke_width":"2.0"}} Conditions: This was seen after customer upgraded to 17.6.4 Workaround: Issue not seen when AP is moved to Secondary WLC Further Problem Description: Confirmed that WLC is sending the WTP Event Response from the RA Traces as well as AP Span Port pcaps. Working case: AP log Sep 28 15:56:57 kernel: [*09/28/2022 15:56:57.5110] Received Capwap Control Msg From AC. Sep 28 15:56:57 kernel: [*09/28/2022 15:56:57.5110] DTLS encrypted Pkt recieved dst_ip: 25.32.132.28 dst_port: 5251 src_ip: 10.195.96.4 src_port: 5246 Sep 28 15:56:57 kernel: [*09/28/2022 15:56:57.5110] Received Capwap Control Msg. Sep 28 15:56:57 kernel: [*09/28/2022 15:56:57.5110] Control message: length = 16. Sep 28 15:56:57 kernel: [*09/28/2022 15:56:57.5110] Queue Empty. Sep 28 15:56:57 kernel: [*09/28/2022 15:56:57.5110] Msg Type = CAPWAP_WTP_EVENT_RESPONSE(10) Capwap State = Run(11). Sep 28 15:56:57 kernel: [*09/28/2022 15:56:57.5110] Wtp Event Response from 10.195.96.4 Sep 28 15:56:57 kernel: [*09/28/2022 15:56:57.5110] Capwap control packet processed. Freeing packet 0x5581616000. Sep 28 15:57:00 kernel: [*09/28/2022 15:57:00.7390] Received CAPWAP_DATA_KEEPALIVE_TIMER_EXPIRY Capwap Timer Msg. Sep 28 15:57:00 kernel: [*09/28/2022 15:57:00.7400] [CAPWAP RX] DATA: 10.195.96.4[5247] -> 25.32.132.28[5251] len 58 Sep 28 15:57:02 kernel: [*09/28/2022 15:57:02.2590] Received Capwap watchdog update msg. Sep 28 15:57:07 kernel: [*09/28/2022 15:57:07.0110] Received CAPWAP_HA_FAST_HEARTBEAT_TMR Capwap Timer Msg. Sep 28 15:57:07 kernel: [*09/28/2022 15:57:07.0110] Sending packet to AC WLC Log 2022/09/28 15:56:58.114268 {wncd_x_R0-0}{1}: [capwapac-smgr-srvr] [18724]: (debug): Mac: 687d.b478.0d60 Session-IP: 25.32.132.28[5251] 10.195.96.4[5246] WTP Event Request from AP 2022/09/28 15:56:58.114299 {wncd_x_R0-0}{1}: [ewlc-infra-capwap-dgram] [18724]: (debug): dgram handle, index is 0, udplite 0 2022/09/28 15:56:58.114328 {wncd_x_R0-0}{1}: [ewlc-capwapmsg-sess] [18724]: (debug): Encrypted DTLS message send. Dest IP: 25.32.132.28[5251], length:37 2022/09/28 15:56:58.114330 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [18724]: (debug): Mac: 687d.b478.0d60 Session-IP: 25.32.132.28[5251] 10.195.96.4[5246] CAPWAP Message buffer sent to DTLS for send. Buffer size: 16, count of buffers: 1 2022/09/28 15:56:58.114340 {wncd_x_R0-0}{1}: [capwapac-smgr-srvr] [18724]: (debug): Mac: 687d.b478.0d60 Session-IP: 25.32.132.28[5251] 10.195.96.4[5246] WTP Event Response sent to AP with sequence number: 191 Non-Working case: AP Log Sep 28 15:57:35 kernel: [*09/28/2022 15:57:35.5230] CAPWAP control packet sent to 10.195.96.4 Sep 28 15:57:35 kernel: [*09/28/2022 15:57:35.5230] HeartBeat request sent to 10.195.96.4 Sep 28 15:57:35 kernel: [*09/28/2022 15:57:35.5230] Received Capwap watchdog update msg. Sep 28 15:57:35 kernel: [*09/28/2022 15:57:35.5240] [CAPWAP RX] CTRL: 10.195.96.4[5246] -> 25.32.132.28[5251] len 57 Sep 28 15:57:35 kernel: [*09/28/2022 15:57:35.5240] Received Capwap Control Msg From AC. Sep 28 15:57:35 kernel: [*09/28/2022 15:57:35.5240] DTLS encrypted Pkt recieved dst_ip: 25.32.132.28 dst_port: 5251 src_ip: 10.195.96.4 src_port: 5246 Sep 28 15:57:38 kernel: [*09/28/2022 15:57:38.3730] Received CAPWAP_MSG_TIMER_EXPIRY Capwap Timer Msg. Sep 28 15:57:38 kernel: [*09/28/2022 15:57:38.3740] Re-Tx Count=1, Max Re-Tx Value=3, SendSeqNum=200, NumofPendingMsgs=1 WLC Log 2022/09/28 15:57:36.129567 {wncd_x_R0-0}{1}: [capwapac-smgr-srvr] [18724]: (debug): Mac: 687d.b478.0d60 Session-IP: 25.32.132.28[5251] 10.195.96.4[5246] WTP Event Request from AP 2022/09/28 15:57:36.129595 {wncd_x_R0-0}{1}: [ewlc-infra-capwap-dgram] [18724]: (debug): dgram handle, index is 0, udplite 0 2022/09/28 15:57:36.129617 {wncd_x_R0-0}{1}: [ewlc-capwapmsg-sess] [18724]: (debug): Encrypted DTLS message send. Dest IP: 25.32.132.28[5251], length:37 2022/09/28 15:57:36.129619 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [18724]: (debug): Mac: 687d.b478.0d60 Session-IP: 25.32.132.28[5251] 10.195.96.4[5246] CAPWAP Message buffer sent to DTLS for send. Buffer size: 16, count of buffers: 1 2022/09/28 15:57:36.129619 {wncd_x_R0-0}{1}: [capwapac-smgr-srvr] [18724]: (debug): Mac: 687d.b478.0d60 Session-IP: 25.32.132.28[5251] 10.195.96.4[5246] WTP Event Response sent to AP with sequence number: 200

[Rich R]

You should open a TAC case and to let TAC investigate further.

[stayd]

@Ramprasad2 have you already opened the TAC ? If yes, can you share with us the result ?

[gnburgos]

I got the same issue and the only way to bring back the AP is the command redundancy force-switchover and this move the control to the secundary wlc and then bring back to primary with the same command redundancy force-switchover

[Bastian Altenhoff]

Hello,

thanks gnburgos we had the same issue and after using the command "redundancy force-switchover" and back it worked. Thanks