As per the subject, we are receiving this error in the log buffer on a 3845 router running 12.4(24)T4. Our setup is that we're running DMVPN, and have configured per-tunnel QoS for DMVPN, exactly as per the Cisco documentation (http://www.cisco.com/en/US/docs/ios/sec_secure_connectivity/configuration/guide/sec_per_tunnel_qos.html) and the issue we're finding is that on the hub end (the 3845), the following full error is being received:
Dec 8 10:04:11: %NHRP-3-QOS_POLICY_APPLY_FAILED: Failed to apply QoS policy qos_setoutbound_lb-sh_pm mapped to NHRP group LB_tunnel0 on interface Tunnel0, to tunnel 192.168.175.11 due to policy installation failure
There is not much info on this error... I've looked it up, and all I can find is this fairly useless document: http://www.cisco.com/en/US/docs/ios/system/messages/guide/sm_cn07.html#wp652530
Has anyone seen this sort of issue before when configuring per-tunnel QoS for DMVPN? Any assistance would be much appreciated. I'll try and include some show command outputs when I can.
Could you share your QoS configuration on the hub router? One restriction with per-tunnel QoS for DMVPN is that, you cannot have an user-defined class under the parent policy-map with queuing actions. The only class allowed is "class-default" with action "shaping", and all user-defined classes with queuing actions would have to be applied to the child policy under the shaper. This would be one reason you'd see the NHRP-3-QOS_POLICY_APPLY_FAILED error you observed.
Hope this helps,
Interesting, thanks for that Wen. Our QoS is primarily aimed at ensuring bandwidth for certain applications, but we do use the priority command to ensure voice traffic is queued faster so that will be our problem, I think; in essence our QoS is configured as follows:
The policy maps are as follows:
shape average 18000000
The command on the tunnel interface is: ip nhrp map group LB_tunnel0 service-policy output qos_setoutbound_lb-sh_pm
I think that the policy map that sets the priority and bandwidth statements would be the problem then, as that affects the queuing? I'd be interested in what you think a solution might be - maybe we mark traffic on the ingress to the hub router (from the LAN) and then let the QoS engine queue based on that?
Frustrating that this restriction isn't listed in the Cisco document, which has a "restrictions" section....
The problem has nothing to do with the queuing actions defined under the child service-policy, but rather you have a user-defined class called "class qos_to_lb-sh_cm" under the parent shaper. So instead of:
shape average 18000000
Change that to:
shape average 18000000
What you had is something you'd need to do before the per-tunnel QoS feature, where you'd have to provide per-spoke classification to each spoke shaper. With the new feature, you no longer need that since the policy will be applied on a per-spoke basis from the NHRP group. I'm assuming the policy you have is just for spokes from one nhrp group, and not for the aggregate traffic.
This is a generic restriction introduced with the HQF enhancement that went into 12.4(20)T, so it's not specific to DMVPN. Take a look at this link and look for "Shaping Behavior on GRE Tunnel":
Thanks for that, Wen - I tried it and it still gives this error. However, I wonder if I've found out why now. The documentation does have the following restriction: "You cannot configure a per-tunnel QoS policy on a tunnel interface and a separate QoS policy on the outbound physical interface at the same time."
I had another read through and checked our configuration, and the source of the Tunnel0 interface is Gi0/0.995 and another sub-interface, Gi0/0.10 has got a service policy applied to it as well - is it perhaps this that is now stopping the policy from installing?
Thanks for your help on this; it's much appreciated.
I think that could be it. We did add multiple policy support for HQF in 12.4(22)T and later, however it's not supported for DMVPN. For details, see link below and look for the last bullet in the "restrictions" section....
Darn, I was fearful of that
I will go away and think of a way round this, and will post back my results if we get a working solution in place, in case anyone else has this issue.
Thanks again for your help with this, it's been much appreciated.
I've been considering a solution for this and have been trying to verify using GNS3 to no avail (I can't get GNS to work - separate issue). If we were to add an additional physical, routed, interface to the hub routers do you think we would be able to do the following?
The reason I'm asking is because of the restriction I've seen in the documentation, "You cannot configure a per-tunnel QoS policy on a tunnel interface and a separate QoS policy on the outbound physical interface at the same time". This restriction doesn't seem 100% clear on whether it is so for all outbound physical interfaces, or just the one the tunnel interface uses as it's source; would you know?
For info, I know that our ISP etc. would accept this (they're part of the same group as I work for) and we're able to source the hardware to be able to do it; it's just this QoS for DMVPN (while still maintaining QoS for the legacy sites as well) that is the problem...
"You cannot configure a per-tunnel QoS policy on a tunnel interface and a separate QoS policy on the outbound physical interface at the same time."
applies to the actual outbound physical interface for the GRE/IPsec tunnel packets, which may not be the same as the 'tunnel source ...' interface. So what you are proposing should work as long as you can split your traffic. Traffic that is going to the WAN (Internet) directly is routed out Gi0/1 (perhaps use a 0/0 (default) route for this traffic). Traffic that is going out through DMVPN is routed out the mGRE tunnel interface (use specific routes learned over DMVPN for this traffic). The tunnel packets then must be routed out the Gi1/0 interface. To do this you could use VRF-lite.
ip vrf Outside
Crypto configuration if using PSK add:
crypto keyring Outside vrf Outside
pre-shared-key address 0.0.0.0 0.0.0.0 key
On the Gi1/0 interface configure
ip vrf forwarding Outside
and on the tunnel interface configure
tunnel vrf Outside
Then you can configure a separate 0/0 (default) route for the tunnel traffic in the Outside VRF,
ip route vrf Outside 0.0.0.0 0.0.0.0
Note, this does require that you have "two" connections to your ISP.
Thanks, Mike that's all very useful information; I'll be talking to our ISP (as I said, part of the same company, which is handy!) about the VRF-side of things; I'm sure it won't be a problem as that's what we use elsewhere on our WAN.
Thanks again for the help I've received; I'm persevering with Dynamips/Dynagen (without GNS3) and will update once I've confirmed, and also once we've implemented (if we do).
Update: I have successfully trialled this using Dynamips/Dynagen; the addition of a physical interface, using that for DMVPN WAN routes and the tunnel source meant the per-tunnel QoS finally appeared on the tunnel (using "show dmvpn detail"):
HUB#sho dmvpn det
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding
UpDn Time --> Up or Down Time for a Tunnel
Interface: Tunnel1, IPv4 NHRP Details
Type:Hub, Total NBMA Peers (v4/v6): 1
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
----- --------------- --------------- ----- -------- ----- -----------------
1 x.x.x.x x.x.x.x UP 00:56:26 D x.x.x.x/32
NHRP group: LB_tunnel1
Output QoS service-policy applied: qos_shapeoutbound_lb-sh_pm
Previously, when running this command we'd receive, "Output QoS service-policy applied: None"
Thanks for everyone's help; I'll try to remember to update once we've done the work just to confirm it's worked as expected.