new circuit causing voice issues

[cdickerson75]

Have a client that is trying to put in a new ATT "Network on Demand" circuit.  This circuit allows the customer to adjust their bandwidth on the fly as demand warrants.  We have a pair of Cisco 3750s in a stack on each side of this circuit.  We are plugging the circuit into a gigabit port on the top switch of the stack.  The port is configured to auto negatioate and comes up at 1000/Full and "switchport mode trunk" is the only thing configured on the port.

The Problem.

Data traffic routes across the circuit no problem, fast, responsive, no dropped packets.

Voice traffic has major issues.  About 80% of the traffic is being lost, but only in one direction.  One location can be heard with no problems, but when the other location speaks, you only get about 20% of the conversation.  ATT says we need to rate-limit the traffic. If we bump the speed of the circuit up to a full 1Gb on the Network on Demand portal, the voice issue is resolved.

Don't believe this is a QoS issue as ATT doesn't have any provisions to honor DSCP tags through their network.  We have configured their circuit for real-time communications.  Don't believe it is a routing issue because works fine when the speed is bumped to 1Gb.  Don't believe it is a rate-limit issue because we are doing this cutover after hours and the circuit is in no way getting overloaded with traffic to cause it to drop packets.

Any ideas?  Thanks

-Craig

[Reza Sharifi]

If we bump the speed of the circuit up to a full 1Gb on the Network on Demand portal, the voice issue is resolved.

Does it only work at 1Gb? What if you prevision it for 500Mb or 100Mb?

HTH

[cdickerson75]

The circuit is initially provisioned only for 500mb.   sorry, left that out of the initial description.

[Reza Sharifi]

Thanks for the info. Yes, 500Mb is a lot of bandwidth specially for voice traffic and also since the circuit is not overloaded. Since you have issues with one location and not the other one, try applying qos to both sides of the trunk to the voice vlan and test again.

[cdickerson75]

Doesn't QoS only kick in when the bandwidth is being fully utilized?  When we are doing the testing, maybe a few Mb is going across the line. I can't see how the switch is deciding it needs to drop some packets.

[Joseph W. Doherty]

Doesn't QoS only kick in when the bandwidth is being fully utilized?

That's correct, but realize a port is fully utilized as soon as one frame/packet needs to queue behind another.

When we are doing the testing, maybe a few Mb is going across the line. I can't see how the switch is deciding it needs to drop some packets.

A very common thought, but often incorrect.  VoIP can be impaired at the millisecond level, most bandwidth utilization is compute multi-second to multi-minute.

[chrihussey]

In which direction is the traffic being lost and is this consistent or sometimes the direction changes? Do the remote sights have the same service?

[cdickerson75]

yes it is always the same direction.  Yes the remote sites are using the same Network On Demand service.  It is a Multipoint EVC I believe they call it.

[chrihussey]

Which direction though? If the ATT service does not provide any form of prioritization then your only recourse is to prioritize traffic on the transmit. You won't be able to control the receive.

Rate limiting and amount of bandwidth isn't as important as the prioritization. If you are preserving the QoS markings on transmit and implement something simple like Auto QoS you may solve the problem. Possibly try "auto qos voip trust" on the interface connected to ATT.

[Joseph W. Doherty]

If your physical hand-off is gig, but your CIR is 500, if your provider enforces your CIR, you're asking for VoIP issues if there's concurrent data traffic.

What you should do is "shape" for your CIR. On a 3750 you can "shape" an egress port using srr-queue bandwidth limit #.

[cdickerson75]

I think I tried the srr-queue command it wasn't appearing to be a valid command.  

interface gig1/0/23

srr-queue bandwidth limit 50

I also tried a rate-limit command.

rate-limit output  500000000 93750000 187500000 conform-action transmit exceed-action drop

rate-limit input 500000000  93750000 187500000 conform-action transmit exceed-action drop

And tried a Police command.

mls qos

!

ip access-list extened ACL_LIMIT

 permit ip any any

!

class-map match all CLASS_LIMIT

 match access-group name ACL_LIMIT

!

policy-map POLICY_LIMIT

 class CLASS_LIMIT

  police 500000000 1000000 exceed-action drop

!

interface gig1/0/23

 service-policy input POLICY_LIMIT

[Joseph W. Doherty]

You don't want to rate-limit or police.

The bandwidth limit command might require QoS to be globally enabled.