Post interface QoS Service Policy deployment - message "counters not available (install failure)"

[fowler.david]

Hi all,

Hope someone has seen this before.

We recently started deploying interface based QoS configuration out to access switches. During the deployment to the first switches, 2 Cat 6500s, we encountered this issue

 GigabitEthernet1/24

  Service-policy input: TAG-INBOUND-MARKING-AND-POLICING

    class-map: TAG-VOIP (match-all)
      Match: ip dscp ef (46)
      police :
        128000 bps 8000 limit 8000 extended limit
      counters not available (install failure)

    class-map: TAG-SIGNALING-ACL (match-all)
      Match: access-group name SIGNALING-ACL
      police :
        32000 bps 8000 limit 8000 extended limit
      counters not available (install failure)

    class-map: TAG-SIGNALING-CS3 (match-all)
      Match: ip dscp cs3 (24)
      police :
        32000 bps 8000 limit 8000 extended limit
      counters not available (install failure)

It doesn't manifest on all ports on an individual switch or switch blade, only some. The port in question above was working correctly until I removed and reapplied the service policy to the interface.

I have a TAC case open but hoping someone else can provide an insight. TAC has initially suggested a TCAM issue?

Possibly an IOS bug?

[InayathUlla Sharieff · Accepted solution]

Hello,
Could you please provide me with the following info;-

Show mls qos ip--> Then check if you are seeing Out Acl-Alink as Failure
<snippet of show mls qos ip o/p>

    Vl901  5 Out ACL-Alink  0     F     --    0

     Vl901  5 Out DGM-Collo  0    F     --    0

     Vl901  5 Out Speditor-  0    F     --    0

This means while attaching the service policy, there was not enough aggregate policer available, so it failed. And hence you are seeing the weird output for counter values.

Also it would be worth to check the “show plat hardware qos capacity” we see aggregate index available but that could be current situation.

6500#show platform hardware capacity qos .. Check whats the % used by the Aggregate policers:

QoS Policer Resources

  Aggregate policers: Module                      Total         Used     %Used

                      5                            1024          874       85%

  Microflow policer configurations: Module        Total         Used     %Used

                                    5                64            1        1%

It means later some of the aggregate index got released. in this case removing the service-policy from the affected vlan and try reconfiguring it. This should fix the issue.
but noticed that when we removed and reconfigured the policy-map the issue again popped up and the TCAM utilization was close 98%!

QoS Policer Resources
   Aggregate policers: Module                      Total Used     %Used
                       5                            1024 1008       98%  ---<<
   Microflow policer configurations: Module        Total Used     %Used
                                     5                64 1        1%

If you are seeing high TCAM then remove few unused class-maps to leverage few TCAM utlization after which we should be able to see the counters.

HTH

Regards

Inayath

***********Plz dont forget to rate post.

[fowler.david]

Hi Inayath

Thanks for your reply.

It is indeed related to the Aggregrate Policer resources. It's at 99% ...

QoS Policer Resources
  Aggregate policers: Module                      Total         Used     %Used
                      5                            1024         1021       99%
                      6                            1024         1021       99%
  Microflow policer configurations: Module        Total         Used     %Used
                                    5                64            1        1%
                                    6                64            1        1%

We've applied the policy per interface which is incrementing the policers per class per interface resulting in hitting the resource limit.

Are there any recommended courses of action for per interface QoS? Only ones I've seen have been to use either VLAN based or named aggregate policers, both of which take a cumulative approach to policing the ports in the VLAN or policer 'group'.

Thanks,

David

[InayathUlla Sharieff]

David,

sorry to say no ther way.

This error appears every time a policy is applied to an interface. Based on
the Cisco info for this message, it appears that the message simply means
that the hardware can only support so much QoS.

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note0918
6a00801b42bf.shtml#qm_agg

The Cisco workaround is to configure VLAN based QoS. I can apply a
service-policy to a VLAN but I cannot enter the "mls qos vlan based" command
on an interface

2)

Or reduce the config on the ports.

HTH

Regards

Inayath

***********Please dont forget to rate all usefull posts.******

[InayathUlla Sharieff]

This is the workaround for the TCAM issue:

1\ If you have implemented the service−policy command on many Layer 2 interfaces which belong to
one VLAN, you can implement VLAN based policing instead of switch port based. This is an
example:
Switch(config)#interface range fastethernet x/y − Z
Switch(config−if)#mls qos vlan−based
Switch(config−if)#exit
Switch(config)#interface vlan 100
Switch(config−if)#service−policy input Test_Policy

•\ If possible, use the same ACLs across multiple interfaces in order to reduce TCAM resource contention.
Disable QoS marking statistics. The no mls qos marking statistics command does not allow the max
of 1020 AgIDs to be implemented. This is because it allocates the default policer for set dscp policers.
The downside of this is there are no statistics for the specific policer because they all share the default
policer.
Switch(config)#no mls qos marking statistics

HTH

Regards

Inayath

***********Please dont forget to rate all usefull posts.******

[fowler.david]

Thanks Inayath.

We tested it in the Lab yesterday so we now know where we stand.

Do you know if there is an aggregate policer limit on the 2960, 3750/X and 6880 series switches? Sadly I haven't been able to find anything to confirm.

Thanks,

David.

[InayathUlla Sharieff]

3560--   <http://tools.cisco.com/squish/38C9D>
http://tools.cisco.com/squish/38C9D

2960--  <http://tools.cisco.com/squish/2c616>
http://tools.cisco.com/squish/2c616 

HTH

[Flanger23]

Hello.

Sorry for reviving an old thread, but I'm in similar situation with 7613 router and in need of an advise.

QoS Policer Resources
 Aggregate policers: Module Total Used  %Used
                        7   1024  1024  100%
                        9   1024   544  53%
                       10   1024   549  53%
 Microflow policer configurations: Module Total Used  %Used
                                     7     64     1     1%
                                     9     64     1     1%
                                    10     64     1     1%

Module 7 is RSP720-3C-GE, modules 9 and 10 are ESM20G line cards used for transport. Other modules are just classic WS-X6148A-GE-TX port aggregation cards, which I use as L3 termination points for the customers. Each such interface is running its own service-policy for both ingress and egress traffic. There are many interfaces, which are failed to be policed. I see "F" flag in agg id and relevant "counters not available (install failure)" message.

As you can see from code snippet, aggregate policer usage is not distributed evenly across these line cards, although they police the same module interfaces. Sample output from "show mls qos ip":

 Int   Mod Dir Class-map DSCP Agg Trust Fl AgForward-By AgPoliced-By    
                              Id        Id
-------------------------------------------------------------------------------
 Gi1/25 9  Out ACL1        0 0*   --    0     n/a           n/a
 Gi1/25 9  Out class-defa  0 446  --    0  31470170954 11023539517

 Gi1/25 7  In  ACL1        0 0*   No    0     n/a           n/a
 Gi1/25 7  In  class-defa  0 1021 dscp  0  1940059309  13914755
 Gi1/25 7  Out ACL2        0 0*   --    0     n/a           n/a
 Gi1/25 7  Out class-defa  0 1022 --    0     3122           0

Is it possible to somehow distribute QoS usage more evenly across these cards?