Re: %NHRP-3-QOS_POLICY_APPLY_FAILED

tom.kivlin · ‎12-08-2010

As per the subject, we are receiving this error in the log buffer on a 3845 router running 12.4(24)T4. Our setup is that we're running DMVPN, and have configured per-tunnel QoS for DMVPN, exactly as per the Cisco documentation (http://www.cisco.com/en/US/docs/ios/sec_secure_connectivity/configuration/guide/sec_per_tunnel_qos.html) and the issue we're finding is that on the hub end (the 3845), the following full error is being received:

Dec 8 10:04:11: %NHRP-3-QOS_POLICY_APPLY_FAILED: Failed to apply QoS policy qos_setoutbound_lb-sh_pm mapped to NHRP group LB_tunnel0 on interface Tunnel0, to tunnel 192.168.175.11 due to policy installation failure

There is not much info on this error... I've looked it up, and all I can find is this fairly useless document: http://www.cisco.com/en/US/docs/ios/system/messages/guide/sm_cn07.html#wp652530

Has anyone seen this sort of issue before when configuring per-tunnel QoS for DMVPN? Any assistance would be much appreciated. I'll try and include some show command outputs when I can.

wzhang · ‎12-08-2010

Hi,

Could you share your QoS configuration on the hub router? One restriction with per-tunnel QoS for DMVPN is that, you cannot have an user-defined class under the parent policy-map with queuing actions. The only class allowed is "class-default" with action "shaping", and all user-defined classes with queuing actions would have to be applied to the child policy under the shaper. This would be one reason you'd see the NHRP-3-QOS_POLICY_APPLY_FAILED error you observed.

Hope this helps,

Thanks,

Wen

tom.kivlin · ‎12-09-2010

Interesting, thanks for that Wen. Our QoS is primarily aimed at ensuring bandwidth for certain applications, but we do use the priority command to ensure voice traffic is queued faster so that will be our problem, I think; in essence our QoS is configured as follows:

A series of ACLs define the traffic to match on (some port-based such as TCP25, TCP3389 etc. and some address-based such as any traffic from certain servers)
A series of class maps reference these ACLs
A policy map is then used to set outbound bandwidth and priority statements against those class maps - our requirement here is to ensure certain traffic gets a stated amount of the bandwidth, and the priority statement ensures voice traffic is queued first (unless we've got that wrong!?)
A second policy map is then used to shape traffic to the remote router (based on a class map that refers to an ACL matching the destination IP range) and also applies the first policy map

The policy maps are as follows:

policy-map qos_setoutbound_lb-sh_pm

class qos_voip_cm

priority 3072

class qos_rdp_cm

bandwidth 2048

class qos_msds_cm

bandwidth 2048

class qos_rmi_cm

bandwidth 1024

class qos_http_cm

bandwidth 5632

class class-default

fair-queue

random-detect dscp-based

policy-map qos_shapeoutbound_lb-sh_pm

class qos_to_lb-sh_cm

shape average 18000000

service-policy qos_setoutbound_lb-sh_pm

The command on the tunnel interface is: ip nhrp map group LB_tunnel0 service-policy output qos_setoutbound_lb-sh_pm

I think that the policy map that sets the priority and bandwidth statements would be the problem then, as that affects the queuing? I'd be interested in what you think a solution might be - maybe we mark traffic on the ingress to the hub router (from the LAN) and then let the QoS engine queue based on that?

Frustrating that this restriction isn't listed in the Cisco document, which has a "restrictions" section....

Cheers,

Tom

wzhang · ‎12-09-2010

Hi, Tom:

The problem has nothing to do with the queuing actions defined under the child service-policy, but rather you have a user-defined class called "class qos_to_lb-sh_cm" under the parent shaper. So instead of:

policy-map qos_shapeoutbound_lb-sh_pm

class qos_to_lb-sh_cm

shape average 18000000

service-policy qos_setoutbound_lb-sh_pm

Change that to:

policy-map qos_shapeoutbound_lb-sh_pm

class class-default

shape average 18000000

service-policy qos_setoutbound_lb-sh_pm

What you had is something you'd need to do before the per-tunnel QoS feature, where you'd have to provide per-spoke classification to each spoke shaper. With the new feature, you no longer need that since the policy will be applied on a per-spoke basis from the NHRP group. I'm assuming the policy you have is just for spokes from one nhrp group, and not for the aggregate traffic.

This is a generic restriction introduced with the HQF enhancement that went into 12.4(20)T, so it's not specific to DMVPN. Take a look at this link and look for "Shaping Behavior on GRE Tunnel":

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6558/white_paper_c11-481499.html

Thanks,

Wen

tom.kivlin · ‎12-10-2010

Thanks for that, Wen - I tried it and it still gives this error. However, I wonder if I've found out why now. The documentation does have the following restriction: "You cannot configure a per-tunnel QoS policy on a tunnel interface and a separate QoS policy on the outbound physical interface at the same time."

I had another read through and checked our configuration, and the source of the Tunnel0 interface is Gi0/0.995 and another sub-interface, Gi0/0.10 has got a service policy applied to it as well - is it perhaps this that is now stopping the policy from installing?

Thanks for your help on this; it's much appreciated.

wzhang · ‎12-10-2010

Hi, Tom:

I think that could be it. We did add multiple policy support for HQF in 12.4(22)T and later, however it's not supported for DMVPN. For details, see link below and look for the last bullet in the "restrictions" section....

http://www.cisco.com/en/US/docs/ios/qos/configuration/guide/qos_hqf_mply_support.html#wp1053987

Thanks,

Wen

tom.kivlin · ‎12-10-2010

Darn, I was fearful of that

I will go away and think of a way round this, and will post back my results if we get a working solution in place, in case anyone else has this issue.

Thanks again for your help with this, it's been much appreciated.

Cheers,

Tom

tom.kivlin · ‎12-20-2010

Hi Wen,

I've been considering a solution for this and have been trying to verify using GNS3 to no avail (I can't get GNS to work - separate issue). If we were to add an additional physical, routed, interface to the hub routers do you think we would be able to do the following?

Gi0/0 remains connecting to the LAN
Gi0/1 remains connecting to the WAN, but we remove sub-interfaces and apply the legacy service-policy to this interface
Gi1/0 (let's say) is also connected to the WAN, and this is what the tunnel interface uses as it's source interface and we use the per-tunnel QoS for DMVPN on this interface only

The reason I'm asking is because of the restriction I've seen in the documentation, "You cannot configure a per-tunnel QoS policy on a tunnel interface and a separate QoS policy on the outbound physical interface at the same time". This restriction doesn't seem 100% clear on whether it is so for all outbound physical interfaces, or just the one the tunnel interface uses as it's source; would you know?

For info, I know that our ISP etc. would accept this (they're part of the same group as I work for) and we're able to source the hardware to be able to do it; it's just this QoS for DMVPN (while still maintaining QoS for the legacy sites as well) that is the problem...

Thanks again,

Tom

Michael Sullenberger · ‎12-20-2010

The restriction,

"You cannot configure a per-tunnel QoS policy on a tunnel interface and a separate QoS policy on the outbound physical interface at the same time."

applies to the actual outbound physical interface for the GRE/IPsec tunnel packets, which may not be the same as the 'tunnel source ...' interface. So what you are proposing should work as long as you can split your traffic. Traffic that is going to the WAN (Internet) directly is routed out Gi0/1 (perhaps use a 0/0 (default) route for this traffic). Traffic that is going out through DMVPN is routed out the mGRE tunnel interface (use specific routes learned over DMVPN for this traffic). The tunnel packets then must be routed out the Gi1/0 interface. To do this you could use VRF-lite.

VRF example:

Main configuration

ip vrf Outside

Crypto configuration if using PSK add:

crypto keyring Outside vrf Outside

pre-shared-key address 0.0.0.0 0.0.0.0 key <--- Assuming wildcard-PSK

On the Gi1/0 interface configure

ip vrf forwarding Outside

and on the tunnel interface configure

tunnel vrf Outside

Then you can configure a separate 0/0 (default) route for the tunnel traffic in the Outside VRF,

ip route vrf Outside 0.0.0.0 0.0.0.0

Note, this does require that you have "two" connections to your ISP.

Mike.

tom.kivlin · ‎12-21-2010

Thanks, Mike that's all very useful information; I'll be talking to our ISP (as I said, part of the same company, which is handy!) about the VRF-side of things; I'm sure it won't be a problem as that's what we use elsewhere on our WAN.

Thanks again for the help I've received; I'm persevering with Dynamips/Dynagen (without GNS3) and will update once I've confirmed, and also once we've implemented (if we do).

Cheers,

Tom

tom.kivlin · ‎12-21-2010

Update: I have successfully trialled this using Dynamips/Dynagen; the addition of a physical interface, using that for DMVPN WAN routes and the tunnel source meant the per-tunnel QoS finally appeared on the tunnel (using "show dmvpn detail"):

HUB#sho dmvpn det

Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete

N - NATed, L - Local, X - No Socket

# Ent --> Number of NHRP entries with same NBMA peer

NHS Status: E --> Expecting Replies, R --> Responding

UpDn Time --> Up or Down Time for a Tunnel

==========================================================================

Interface: Tunnel1, IPv4 NHRP Details

Type:Hub, Total NBMA Peers (v4/v6): 1

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network

----- --------------- --------------- ----- -------- ----- -----------------

1 x.x.x.x x.x.x.x UP 00:56:26 D x.x.x.x/32

NHRP group: LB_tunnel1

Output QoS service-policy applied: qos_shapeoutbound_lb-sh_pm

Previously, when running this command we'd receive, "Output QoS service-policy applied: None"

Thanks for everyone's help; I'll try to remember to update once we've done the work just to confirm it's worked as expected.