Catalyst 2960+ and output drops towards access points

[trondaker]

Hi,

We have mainly the 2960+ as our branch switches with AP1832i as the access points hanging off them. We see many output drops on these interfaces towards the aps. The ten first ports have aps:

someswitch#sh interfaces counters errors

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards

Fa0/1 0 0 0 0 0 161
Fa0/2 0 0 0 0 0 723732
Fa0/3 0 0 0 0 0 497194
Fa0/4 0 0 0 0 0 74529
Fa0/5 0 0 0 0 0 211933
Fa0/6 0 0 0 0 0 60689
Fa0/7 0 0 0 0 0 2247
Fa0/8 0 0 0 0 0 114212
Fa0/9 0 0 0 1 0 295955
Fa0/10 0 0 0 0 0 59143

Granted, this is a 8 week period, but we are having issues with skype sessions being disconnected, bad quality and so on through wireless. Switching to LTE works fine, so we know the Skype infrastructure is ok. No other ports running wired voip-phones are having these issues, and the wlans are all straight 5 ghz, no 2,4 enabled. Site survey shows no interference sources, and the actual datarate going to this branch is very low (10 mb/s in total). Could these output drops be the source of the wireless problems? Is the 2 MB buffers just not enough to handle the bursty nature of ap-traffic? Could i alleviate this with enabling qos on the switch, and putting important traffic in the priority queue?

[Georg Pauwen]

Hello,

can you post the running config of the switch ? Queue tuning might help...

[trondaker]

Config, pretty plain, just removed aaa-servers, passwords and some snmp/acls. Nothing related to ports.

version 15.0
service config
service nagle
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
service password-encryption
!
hostname someswitch
!
boot-start-marker
boot-end-marker
!
no logging console
enable secret xxx
!
username navadmin password xxx
aaa new-model
!
!
aaa authentication login default group tacacs+ local
aaa authentication enable default group tacacs+ enable
aaa authentication dot1x default group radius
aaa authorization exec default group tacacs+ none
aaa authorization commands 1 default group tacacs+ if-authenticated
aaa authorization commands 15 default group tacacs+ if-authenticated
aaa authorization network default group radius
aaa accounting exec default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+
!
!
!
!
!
!
aaa session-id common
clock timezone CET 1 0
clock summer-time DST recurring last Sun Mar 2:00 last Sun Oct 3:00
system mtu routing 1500
vtp domain nav
vtp mode transparent
!
!
no ip domain-lookup
ip domain-name nm.local
!
!
!
dot1x system-auth-control
!
!
!
!
!
errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause security-violation
errdisable recovery cause pagp-flap
errdisable recovery cause link-flap
errdisable recovery cause psecure-violation
errdisable recovery interval 3600
!
spanning-tree mode rapid-pvst
spanning-tree portfast bpduguard default
spanning-tree portfast bpdufilter default
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
spanning-tree vlan 1,3,30,100-103,112,122,999 priority 4096
!
vlan internal allocation policy ascending
!
vlan 3
name mgmt
!
vlan 30
name wifi
!
vlan 100
name nav
!
vlan 101
name pub
!
vlan 102
name voice
!
vlan 103
name video
!
vlan 112
name kontaktsenter_voice
!
vlan 122
name forvaltning_voice
!
vlan 666
name quarantine
!
vlan 999
name komm
!
!
!
interface r FastEthernet0/1 - 48
description __802.1X__
switchport mode access
authentication event fail action authorize vlan 666
authentication event no-response action authorize vlan 666
authentication port-control auto
authentication violation protect
mab
no snmp trap link-status
spanning-tree portfast
!
!
interface r GigabitEthernet0/1 - 4
description __UPLINK__
switchport mode trunk
!
interface Vlan1
description __Mgmt nett__
ip address 10.198.33.11 255.255.255.0
no ip route-cache
!
ip default-gateway 10.198.33.1

!
!
line con 0
exec-timeout 15 0
privilege level 15
line vty 0 4
access-class 1 in
exec-timeout 15 0
privilege level 15
transport input ssh
line vty 5 15
access-class 1 in
exec-timeout 15 0
privilege level 15
transport input ssh
!
ntp server 10.65.0.1 prefer
ntp server 10.65.0.2
end

[Georg Pauwen]

Hello,

you don't have any QoS enabled. Try and enable it globally (mls qos) and then post the output of:

show mls qos interfaces statistics

Also, post the output of:

show buffers

[trondaker]

No, mls qos is disabled company wide for the branches as we havent seen the need - but ill enable it tonight for this switch and get the output :)

[Georg Pauwen]

Hello,

understood. There is a good chance that mls qos queue-set tuning will diminish or eliminate the discards...which are usually cause by bursty traffic...

[trondaker]

Should i just enable auto-qos, or do you have a specific queue-set in mind?

[Joseph W. Doherty]

"Could these output drops be the source of the wireless problems?"

Possibly. "Real-time" application types are often especially sensitive to packet drops.

"Is the 2 MB buffers just not enough to handle the bursty nature of ap-traffic?"

Possibly as is and/or the way buffers are allocated to support the ports. I.e. for the latter, you might not be running out of physical buffer space, you might be running out of logical buffer space.

"Could i alleviate this with enabling qos on the switch, and putting important traffic in the priority queue?"

Maybe, maybe not. Often enabling QoS leads to additional drops (using defaults) because of the way the switch "reserves" buffers for QoS egress queues. Even setting/using PQ for this traffic might, or might not help, again, especially, if using default values.

BTW, part of you problem might also be on the wireless side. Wi-Fi can also be "tough" on time real-time traffic. Site survey's are good for showing conditions at the time of the survey, which is why so much of the newer Enterprise wireless has built-in dynamic monitoring and on-the-fly adjustment capabilities.

[trondaker]

So enabled mls qos and auto qos - just to have boilerplate-config in place to test:

interface FastEthernet0/1
description __802.1X__
switchport mode access
srr-queue bandwidth share 1 30 35 5
priority-queue out
authentication event fail action authorize vlan 666
authentication event no-response action authorize vlan 666
authentication port-control auto
authentication violation protect
mab
mls qos trust cos
no snmp trap link-status
auto qos trust
spanning-tree portfast

someswitch#show mls qos interface fa0/1 buffers
FastEthernet0/1
The port is mapped to qset : 1
The allocations between the queues are : 15 25 40 20

someswitch#show mls qos interface fa0/1 queueing
FastEthernet0/1
Egress Priority Queue : enabled
Shaped queue weights (absolute) : 25 0 0 0
Shared queue weights : 1 30 35 5
The port bandwidth limit : 100 (Operational Bandwidth:100.0)
The port is mapped to qset : 1

Will be interesting to see what this does to the drops in Skype tomorrow :)

[trondaker]

So traffic is flowing through now, but a couple of observations:

someswitch#show mls qos interface fa0/2 statistics

FastEthernet0/2 (All statistics are in packets)

dscp: outgoing
-------------------------------

0 - 4 : 457589648 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 327141458 0 0 0
30 - 34 : 0 0 0 0 51858801
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 6504834 0 415925 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0

cos: outgoing
-------------------------------

0 - 4 : 843210347 0 0 706331 590
5 - 7 : 1710 0 0
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 1710
queue 1: 708825 932914 680003
queue 2: 0 0 3522178
queue 3: 0 0 839363784

output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 24
queue 1: 13962 0 0
queue 2: 0 0 71668
queue 3: 0 0 773809

Policer: Inprofile: 0 OutofProfile: 0

someswitch#show mls qos maps dscp-output-q
Dscp-outputq-threshold map:
d1 :d2 0 1 2 3 4 5 6 7 8 9
------------------------------------------------------------
0 : 03-03 03-03 03-03 03-03 03-03 03-03 03-03 03-03 04-01 04-01
1 : 04-02 04-01 04-02 04-01 04-02 04-01 02-01 02-01 02-01 02-01
2 : 02-01 02-01 02-01 02-01 02-02 03-01 02-01 02-01 02-01 02-01
3 : 02-01 02-01 01-03 01-03 02-01 02-01 02-01 02-01 02-01 02-01
4 : 01-03 01-03 01-03 01-03 01-03 01-03 01-03 01-03 02-03 02-03
5 : 02-03 02-03 02-03 02-03 02-03 02-03 02-03 02-03 02-03 02-03
6 : 02-03 02-03 02-03 02-03

someswitch#show mls qos maps dscp-cos
Dscp-cos map:
d1 : d2 0 1 2 3 4 5 6 7 8 9
---------------------------------------
0 : 00 00 00 00 00 00 00 00 01 01
1 : 01 01 01 01 01 01 02 02 02 02
2 : 02 02 02 02 03 03 03 03 03 03
3 : 03 03 04 04 04 04 04 04 04 04
4 : 05 05 05 05 05 05 05 05 06 06
5 : 06 06 06 06 06 06 07 07 07 07
6 : 07 07 07 07

I removed the incoming output as it doesnt really matter much here, but i see that traffic outgoing in the DSCP 46/EF is correct, but the outgoing cos stats doesnt have a lot of packets for cos 5? And the cos-to-dscp map maps 46 to 5 - something wrong here?

[Joseph W. Doherty]

Well remember ToS is available in every IP packet but CoS is only available in tagged frames. Might that account for the difference?