EF Marked Packets Inconsistently Matching Class-Default

[bbenner]

2911 - 15.2(4)M6a - T1 WAN

I'm having difficulty figuring out why I get intermittent ef marked packets in the class-default queue.  The matched voice ACL (UDP based port range) counters agree with the realtime counters exactly.

#sh policy-map int gi0/0 in
 GigabitEthernet0/0

  Service-policy input: MarkInbound

    Class-map: MarkVoice (match-any)
      189681 packets, 19213876 bytes
      30 second offered rate 0000 bps, drop rate 0000 bps
      Match: access-group name VoiceDataACL
        189681 packets, 19213876 bytes
        30 second rate 0 bps
      QoS Set
        dscp ef
          Packets marked 189681

sh ip access-lists VoiceDataACL
Extended IP access list VoiceDataACL
    10 permit udp any range 16384 32767 any range 16384 32767 (189681 matches)

sh policy-map int se0/0/0:0 out class class-default

<truncated output>

        Class-map: class-default (match-any)
          1784475 packets, 1527263608 bytes
          30 second offered rate 75000 bps, drop rate 0000 bps
          Match: any
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/100005/0
          (pkts output/bytes output) 1786915/1455101962
          bandwidth remaining 4%
            Exp-weight-constant: 9 (1/512)
            Mean queue depth: 0 packets
            dscp       Transmitted         Random drop      Tail drop          Minimum        Maximum     Mark
                    pkts/bytes            pkts/bytes       pkts/bytes          thresh         thresh     prob

            default  1772601/1453731273   8331/9553582    91453/124110423         20            40  1/10
            ef         14314/1370689         3/472          218/34293             36            40  1/10

The problem is manifested on WAN calls where the far end hears degraded voice quality, but the near end has no problems.  How can ef marked packets get queued in the class-default class-map?

This same config is present in dozens of other routers, but this one site is the only one with errant ef packets.

I appreciate any help.

[bbenner]

Here is the full class and policy map configs:

class-map match-any realtime
 match ip dscp ef
class-map match-any MarkBusinessData
 match access-group name BusinessDataACL
class-map match-any Business
 match ip dscp cs2  af21  af22  af23
class-map match-any MarkCallSetup
 match access-group name CallSetupACL
class-map match-any Video
 match ip dscp cs4  af41  af42  af43
class-map match-any MarkMissionCritical
 match access-group name MissionCriticalACL
class-map match-any MarkGeneralData
 match access-group name GeneralDataACL
class-map match-any General
 match ip dscp cs1  af11  af12  af13
class-map match-any MarkVoice
 match access-group name VoiceDataACL
class-map match-any MarkVideo
 match access-group name VideoDataACL
class-map match-any MissionCritical
 match ip dscp cs3  af31  af32  af33  cs6  cs7
!
policy-map etm-Company
 class realtime
  priority 768
  police 768000 conform-action transmit  exceed-action set-dscp-transmit af41 violate-action set-dscp-transmit af41
 class Video
  bandwidth remaining percent 39
  random-detect dscp-based
 class MissionCritical
  bandwidth remaining percent 39
  random-detect dscp-based
 class Business
  bandwidth remaining percent 16
  random-detect dscp-based
 class General
  bandwidth remaining percent 1
  random-detect dscp-based
 class class-default
  bandwidth remaining percent 4
  random-detect dscp-based
policy-map shape-etm-Company
 class class-default
  shape average 1536000
   service-policy etm-Company
policy-map MarkInbound
 class MarkVoice
  set ip dscp ef
 class MarkVideo
  set ip dscp af41
 class MarkCallSetup
  set ip dscp af31
 class MarkMissionCritical
  set ip dscp af32
 class MarkBusinessData
  set ip dscp af21
 class MarkGeneralData
  set ip dscp af11
 class class-default
  set ip dscp default

[zskadar]

Did you find any solution for this problem? I've ran to this same issue at a customer. There are a bunch of 2901 routers with same IOS, almost same configuration (there are 2,10,100M links), and about 5 from the 40 routers cannot match RTP traffic based on dscp ef markings. I've switched on dscp-based WRED too on class default, and I've seen the same, there are the dscp ef marked packets instead of the LLQ. The routers run with IOS 15.1.3T image. These are voice gateways with ISDN BRI or FXS voice interfaces, this voice traffic goes through the WAN connection. Almost every routers do it's job fine, but this 5 don't. I've double-checked everything many times, but I couldn't find the solution. I did not find any relevant bug in the IOS release notes...

Regards,

Zsolt

[bbenner · Accepted solution]

Definitely a code problem.  Short version: after many weeks of chasing them, Cisco dropped in a test version of IOS code.  Problem cleared.  Cisco asked to remove test code, problem came back.  We are waiting for Cisco to release a mainline version with the fix.  According to the emails:

They have committed the fix to the following images: 15.6(2)T1 and 15.4(3)M6

cco release date (15.6(2)T1) is 6/21/2016

cco release date (15.4(3)M6) is 7/29/2016

The same fix will be committed to other versions like 15.5 and 15.6M but they will published by Sept or Oct this year.  Looking for bug ID...

[zskadar]

Thanks for your fast response,

I've found an almost relevant bug (CSCth59123), what is similar as our issue, but the adviced workaround hasn't worked. I've tried an upgrade to the latest recommended IOS (15.4.3M5), but there was a side effect, the GET VPN configuration went wrong. I did it remotely in working hours, so it was a nightmare, I had to go to that site and resolve this issue immediately, I had to roll back to 15.1.3T. It seems based on your story it's currently an unresolved issue, what will fix these following releases. I thought I'll move to the latest 15.1.3T4 on the current train, but it's completely unnecessary based on your experiences, I have to go to one of the following release, and I have to resolve te additional syntax or config change on the GET VPN...

Thanks!

Regards,

Zsolt

[bbenner]

CSCuy23951

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy23951

Symptom:  Priority packets fall under class-default

Basic config to recreate issue:

policy-map BU
class BU
priority
class class-default
random-detect dscp-based >>>> not needed, added to match affected packets against DSCP marked by BU class-map

F340.22.17-2921-2#sh policy-map interface
Serial0/2/0:0

Service-policy output:  BU

queue stats for all priorty classes:
Queueing
queie limit 64 packets
(queue depth/total drops/no-buffer drops)0/0/0

(pks output/bytes output) 189858/263143188

Class-map:  BU (match-any)
189858 packets, 263143188 bytes
30 second offered rate 1463000 bps, drop rate 0000 bps
Match:  dscp ef(46) >>>>>> EF
189858 packets, 263143188 bytes
30 second rate 1463000 bps
Priority:  Strict, b/w exceed drops: 0

Class-map:  class-default(match-any)
174 packets, 13707 bytes
30 second offered rate 0000 bps, drop rate 0000 bps
Match:  any

queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 176/16479
Exp-weight-constant:  9 (1/512)
Mean queue depth:  0 packets
dscp Transmitted Random drop Tail drop Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

default 175/13731 0/0 0/0 20 40 1/10
ef 2/2772 0/0 0/0 36 40 1/10

Conditions:
Channelized interface VWIC

Tested a hwic-1t and build-in Gig Int, no issue.

Workaround:
Change the queueing policy from priority to bandwidth results in the problem no occurring or using just the queueing policy and removing the shaping policy results in the problem not occurring.

Further Problem Description: 
There is a clear interaction between channelized interfaces and QoS.

[zskadar]

Unfortunately it's not match to my issue, I use built-in gig ports as wan connection in this case. I've tried remove the shaping from the multi-level QoS config, but there was nothing changed. I've tried many kind of changes, remove other classes, change the name of the classes to re-order the queues on the output, but nothing worked. I hope, the upgrade what you mentioned will resolve this issue. The strange thing is, only five of the fifty routers are affected, but this five always fail. The other ones work perfectly with the same configuration.

[Jose Castillo]

Hi, I wrote this Bug regarding EF drops on VWICs.

We never found problems with GigaB interfaces, please notice that there are some traffic that will always "fall" under class-default but its policies does not take any effect on the traffic, for example BFD.

Note: I think one of my peers was working on similar issue on GigaB Interfaces but on ASR1Ks, ISRg2s no issues were found on your tests.