3602 and 3702 APs do not join while being connected over a VPN link

vladakoci · ‎03-05-2015

I had a trouble while implementing WLAN services at our branch being connected over a Cisco VPN link. The APs were constantly associating and disassociating the controllers, establishing a Capwap tunnel and dropping it.

I have simulated this on my table, having two spare routers configured for a VPN link. I got the same AP misbehavior even if the routers were interconnected by a cable, means no ISPs in between and no Internet issues. I tried different WLAN controllers with different OS versions, tried a local and flexconnect AP mode, always instable. Tried to exercise the IOS version on the routers, this did not help either. Then I took an older 1142 AP and this one worked well, being perfectly stable. Then I tried to configure the interlink on the routers with just pure Ethernet ( no IPSEC, no GRE ) and configured ip mtu 1376 on the Ethernet interfaces and got the same instability on 3602 and 3702 APs. 1142 AP works fine.
Controllers used are AIR-CT5508-K9 with 7.6.130.0, and AIR-CT8510-K9 with 7.6.120.0

Is anyone aware of any troubles on 3602 and 3702 while having lower MTU in the network path. Seems like it might be an MTU path discovery issue.

Thanks.

Vlad

Scott Fella · ‎03-05-2015

There has been a few forum post with regards to AP's joining from sites connected back to VPN. Lowering the MTU, you also have to account for the overhead and then also define the MTU on the access point or globally on all access points. Here are some links to review and hope it helps:

http://networkcanuck.com/2013/06/10/troubleshooting-mtu-size-over-ipsec-vpn/

http://mrncciew.com/2013/04/07/configuring-tcp-mss/

-Scott

-Scott
*** Please rate helpful posts ***

vladakoci · ‎03-05-2015

We have TCP MSS set to 1240 globally on the controller, but this has nothing to do with a Capwap tunnel.

We cannot change MTU on the network path. I am not sure how we could change permanently MTU on an AP, not even taking into account that it could be also on the other side - controller.

It seems to me the description of the older bug below perfectly fits the problem with 3602 and 3702 and newer controller OS version. I tried also higher MTU sizes on the Ethernet interfaces of the routers in the network path, like for example 1492 and any number below 1500 causes that. As said 1142 AP works perfectly fine, so this seems to be AP model related and might be a bug in the code.

Problem 14: LWAPP APs do not join WLC if network MTU is less than 1500 bytes

This is because of Cisco bug ID CSCsd94967. LWAPP APs might fail to join a WLC. If the LWAPP join request is larger than 1500 bytes, LWAPP must fragment the LWAPP join request. The logic for all LWAPP APs is that the size of the first fragment is 1500 bytes (including IP and UDP header) and the second fragment is 54 bytes (including IP and UDP header). If the network between the LWAPP APs and WLC has a MTU size less than 1500 (as might be encountered when using a tunneling protocol such as IPsec VPN, GRE, MPLS, etc.), WLC cannot handle the LWAPP join request.

You will encounter this problem under these conditions:

WLC that runs version 3.2 software or earlier
Network path MTU between the AP and WLC is less than 1500 bytes

In order to resolve this issue, use any one of these options:

Upgrade to WLC software 4.0, if the platform supports it. In WLC version 4.0, this problem is fixed by allowing the LWAPP tunnel to reassemble up to 4 fragments.
Increase the network path MTU to 1500 bytes.
Use 1030 REAPs for the locations reachable via low MTU paths. REAP LWAPP connections to 1030 APs have been modified to handle this situation by reducing the MTU used for REAP mode.

Scott Fella · ‎03-05-2015

Open a TAC case and see what they say.

-Scott

-Scott
*** Please rate helpful posts ***

vladakoci · ‎03-06-2015

I feel have narrowed this down.

I realized this is not ip mtu issue. I changed ip mtu from 1376 up to 1500 and even with 1500 this occurs, but just not so often so I had to wait for a longer time until the AP dropped the tunnel

I tried to connect the 3602 AP directly to the built-in switch in the 1812 router that I am using for the test and it worked perfectly fine even if I had ip mtu set to 1376 on the path to the controller.

So it is clear the WS-C3560CG-8PC-S switch that I use in front of the router to power AP over PoE is a cause, in combination with 3602 or 3702 AP.

We have a standard company config for the switches and this has been applied. There are many config lines and features set, but what I did is that I removed them one by one and seem to have found it.

Our standard port config for the LWAPP for the small sites is

switchport mode access

switchport access vlan 1

switchport port-security maximum 25

switchport port-security

switchport port-security aging time 2

switchport port-security violation restrict

switchport port-security aging type inactivity

ip device tracking maximum 10

ip arp inspection trust

ip dhcp snooping trust

no logging event link-status

priority-queue out

mls qos trust dscp

srr-queue bandwidth share 10 80 5 5

srr-queue bandwidth shape 0 0 0 0

queue-set 1

snmp trap mac-notification change added

snmp trap mac-notification change removed

spanning-tree portfast

spanning-tree bpduguard enable

I've configured the port lines in a one by one way to see the effect on the AP. With this config

!

interface GigabitEthernet0/8

switchport mode access

switchport port-security maximum 25

switchport port-security violation restrict

switchport port-security aging time 2

switchport port-security aging type inactivity

switchport port-security

ip device tracking maximum 10

ip arp inspection trust

no logging event link-status

priority-queue out

ip dhcp snooping trust

it works perfectly fine, but when I added this line

mls qos trust dscp

and had this config

!

interface GigabitEthernet0/8

switchport mode access

switchport port-security maximum 25

switchport port-security violation restrict

switchport port-security aging time 2

switchport port-security aging type inactivity

switchport port-security

ip device tracking maximum 10

ip arp inspection trust

no logging event link-status

priority-queue out

mls qos trust dscp

ip dhcp snooping trust

the 3602 AP started dropping the Capwap tunnel.

It works perfectly fine when I plug the 1142 AP in, means I keep the port config the same and the 1142 AP is stable.

I tried another WS-C3560CG-8PC-S HW, means another same switch, and also tried two IOS versions c3560c405ex-universalk9-mz.150-2.SE6.bin and c3560c405ex-universalk9-mz.152-2.E1.bin, but there is no dependency, means it does not work correctly no matter what IOS or HW I take.

vladakoci · ‎03-10-2015

I captured qos statistics while having connected 3602 AP on1G/100M and 1142 AP on 1G

For 3602 AP on 1G, g0/8 being an AP port and g0/10 being the port leading to the router.

sh mls qos int g0/8 statis
GigabitEthernet0/8 (All statistics are in packets)

dscp: incoming
-------------------------------

0 - 4 : 16 0 536 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 456 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
dscp: outgoing
-------------------------------

0 - 4 : 718 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 0 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
cos: incoming
-------------------------------

0 - 4 : 1078 0 0 0 0
5 - 7 : 0 0 0
cos: outgoing
-------------------------------

0 - 4 : 739 0 0 0 0
5 - 7 : 0 0 238
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 754 21 238
queue 2: 0 0 0
queue 3: 0 0 0

output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 0 0
queue 2: 0 0 0
queue 3: 0 0 0

Policer: Inprofile: 0 OutofProfile: 0

mmbn2021#sh mls qos int g0/10 statis
GigabitEthernet0/10 (All statistics are in packets)

dscp: incoming
-------------------------------

0 - 4 : 554 0 0 0 38
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 155 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
dscp: outgoing
-------------------------------

0 - 4 : 16 0 538 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 440 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
cos: incoming
-------------------------------

0 - 4 : 954 0 0 0 0
5 - 7 : 0 0 0
cos: outgoing
-------------------------------

0 - 4 : 566 0 0 0 0
5 - 7 : 0 440 45
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 554 12 57
queue 2: 0 0 0
queue 3: 428 0 0

output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 0 0
queue 2: 0 0 0
queue 3: 34 0 0

Policer: Inprofile: 0 OutofProfile: 0

For 3602 AP on 100M, g0/8 being an AP port and g0/10 being the port leading to the router.

sh mls qos int g0/8 statis
GigabitEthernet0/8 (All statistics are in packets)

dscp: incoming
-------------------------------

0 - 4 : 33 0 978 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 1 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 934 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
dscp: outgoing
-------------------------------

0 - 4 : 2047 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 0 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
cos: incoming
-------------------------------

0 - 4 : 2124 0 0 0 0
5 - 7 : 0 0 0
cos: outgoing
-------------------------------

0 - 4 : 2110 0 3 0 4
5 - 7 : 0 0 583
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 2135 70 583
queue 2: 0 0 0
queue 3: 0 0 0

output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 0 0
queue 2: 0 0 0
queue 3: 0 0 0

Policer: Inprofile: 0 OutofProfile: 0

mmbn2021#sh mls qos int g0/10 statis
GigabitEthernet0/10 (All statistics are in packets)

dscp: incoming
-------------------------------

0 - 4 : 1731 0 0 0 166
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 206 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
dscp: outgoing
-------------------------------

0 - 4 : 36 0 978 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 1 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 991 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
cos: incoming
-------------------------------

0 - 4 : 2619 0 0 0 0
5 - 7 : 0 0 0
cos: outgoing
-------------------------------

0 - 4 : 1043 1 0 2 0
5 - 7 : 0 991 108
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 1012 34 136
queue 2: 0 0 0
queue 3: 963 0 0

output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 0 0
queue 2: 0 0 0
queue 3: 0 0 0

Policer: Inprofile: 0 OutofProfile: 0

For 1142 AP on 1G, g0/8 being an AP port and g0/10 being the port leading to the router.

sh mls qos int g0/8 statis
GigabitEthernet0/8 (All statistics are in packets)

dscp: incoming
-------------------------------

0 - 4 : 22 0 3 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 356 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
dscp: outgoing
-------------------------------

0 - 4 : 668 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 0 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
cos: incoming
-------------------------------

0 - 4 : 488 0 0 0 0
5 - 7 : 0 0 0
cos: outgoing
-------------------------------

0 - 4 : 702 0 3 0 0
5 - 7 : 0 0 351
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 721 37 351
queue 2: 0 0 0
queue 3: 0 0 0

output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 0 0
queue 2: 0 0 0
queue 3: 0 0 0

Policer: Inprofile: 0 OutofProfile: 0

mmbn2021#sh mls qos int g0/10 statis
GigabitEthernet0/10 (All statistics are in packets)

dscp: incoming
-------------------------------

0 - 4 : 591 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 93 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
dscp: outgoing
-------------------------------

0 - 4 : 29 0 4 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 370 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
cos: incoming
-------------------------------

0 - 4 : 984 0 0 0 0
5 - 7 : 0 0 0
cos: outgoing
-------------------------------

0 - 4 : 49 0 1 0 0
5 - 7 : 0 370 66
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 33 17 82
queue 2: 0 0 0
queue 3: 354 0 0

output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 0 0
queue 2: 0 0 0
queue 3: 0 0 0

Policer: Inprofile: 0 OutofProfile: 0

For 3602 AP being on 1G it seems there are drops and maybe Capwap tunnel not so stable because of this.

For 1142 AP being on 1G it seems there are no drops and Capwap tunnel maybe stable all the time because of this.

For 3602 AP being on 100M it seems there are no drops and Capwap tunnel maybe stable all the time because of this.

Does anyone have an explanation on this behaviour ?

vladakoci · ‎03-10-2015

I have tried another test.

Because of voice implementation we have our own queueset 1 defined:

# show mls qos queue-set

Queueset: 1

Queue : 1 2 3 4

----------------------------------------------

buffers : 10 80 5 5

threshold1: 100 225 400 400

threshold2: 100 225 400 400

reserved : 50 50 50 50

maximum : 400 225 400 400

Queueset: 2

Queue : 1 2 3 4

----------------------------------------------

buffers : 25 25 25 25

threshold1: 100 200 100 100

threshold2: 100 200 100 100

reserved : 50 50 50 50

maximum : 400 400 400 400

I've changed the settings of g0/10 interface ( interface going to the router ) so that the standard Cisco ( default values ) queueset is used, means I used queue-set 2

!

interface GigabitEthernet0/10

ip arp inspection trust

load-interval 30

queue-set 2

ip dhcp snooping trust

Even with that I have interface output drops ( seen in both sh int g0/10 and sh mls qos int g0/10 statis ) and the Capwap tunnel unstable.

Obviously the Capwap tunnel is very sensitive to output drops. I wonder what is different in Capwap implementation on C3602 and C1142 ? I think this might be the key.

vladakoci · ‎03-10-2015

Blank.