3 Second Delay in Audio

[mpaul]

I have a Call Manager 4.01 Sr1a implementation with approximately 55 7940/7960 IP Phones. They are connected with 2 3560 48 port inline power phones, a 3550 24 port and then a 3750 24 Port Gigabit switch. I have a voice and data vlan with all the voice servers and phones in the voice vlan. I recently upgraded to the 6.05 firmware load on the phone and noticed a strange thing. For off-net and on-net calls sometimes, maybe 3 out of 10 calls users will hear a delay in the audio. Once a call is established you try to speak and hear nothing for 3 seconds. After the 3 seconds you can hear the other caller. Some delay in the RTP stream. I've opened a TAC case and they helped me take the firmware back to 6.04 but I am still hearing the same delays. They've wanted me to gather packet traces but since this is so random it is difficult to track down. I essentially have to do a port span and be right at the phone to capture the traces correctly. Wondering if anybody has seen this before?

I've swapped switch ports, ethernet cables, phones. I'm using auto qos voip cisco-phone on the switch ports for the QOS features. There's no inter-vlan routing going on for the voice traffic.

The switches are in a hub-and-spoke design with all the switches hanging off the gigabit switch. They're uplinked with each other via copper gigabit.

[jeffkelso]

I found it. It was right under my nose and I was looking over it. Thanks, I'm going to give it a try.

[aattravi]

I have the same problem in a ip to ip calls.

CCM3.3(3)sr4 , 50 ip phone, two cat3550.

[rvincent]

Would anyone from Cisco care to comment on this?

I've got two customers, one with cm 4.0, the other with 3.3.4, and both are experiencing this delay in the audio.

Rob

[tavery2000]

The problem we are "ALL" facing is a combination of issues. First issue is in all verison of 3.3(?) call manager. According to our DE, the problem was not discovered until 4.0. Another issue is within the signaling setup bit. As of CCM 3.3(4) and up Cisco changed the call signaling for voice and video to PHB CS3 or DSCP 24, previously classified as AF31 or DSCP 34. Voice Bearer has not changed and is still EF or DSCP 46. The following steps were taken to resolve our delay issues.

1). Upgraded to CCM 4.0.2a. We made the upgrade to support Cisco VT Advantage and resolve a few known issues in 3.3.3. If you are running IPCC 3.5.1 or higher, with AD, you will need patch ES-13.

2). changed all LAN qos to conform with the new CS3 and DSCP 24.

I don't believe the lastest phone load will resolve the problem. Working for a Brokerage Firm, with over 2k ip phones, 3 sec delay is not acceptable. Since making the changes our delay issue has been resolved. Unfortunitly I'm not sure if CCM 4.0.2 fixed the problem or changes to QoS was the fix.

Jumping to 4.0 for 3.3.4 is a big change and should be the last resort. However, skip 4.0.1 if you plan to deploy any video (Tandberg or Cisco VT Advantage).

Thanks

Tom

[mpaul]

I'm curious if you were using the auto-qos feature on your switches. For the site that is experiencing this, I noticed this command and used it for the first time, otherwise I would've manually put out the commands I've used at some other sites, 3560/3550 switches.

interface FastEthernet0/2

switchport access vlan 20

switchport voice vlan 10

srr-queue bandwidth share 10 10 60 20

srr-queue bandwidth shape 10 0 0 0

mls qos trust device cisco-phone

mls qos trust cos

no mdix auto

auto qos voip cisco-phone

spanning-tree portfast

The auto qos voip cisco-phone command automatically added the MLS QOS statements and the srr-queue statements.

For those ports that aren't attached to the cisco phones we're not using this command on the switch port, printers, servers.

[mciarfello]

I'm also curious what your switch configs looked like too because changing the DSCP types should not matter to the switch, the switch will still give it priority. (Unless you were specifically classifying specific DSCP types.) You bring up an interesting topic and I think we all would like to explore this some more.

Our switches are set to trust DSCP markings on CCM, GW ports and trunk ports and use AutoQos for phone ports. I have verified end-end QoS using a sniffer at various points in the network.

I am going to start deploying 7-1-1 on a limited basis tomorrow, so we'll see if a phone load fixes it. It will take me some time to report back findings since the delay didn't happen every day. Running CCM 3.3.3SR4.

If anyone else had any success with firmware upgrades, I would LOVE to hear from ya.

[tavery2000]

Cat IOS version should be around 12.2.20 or 18. The QoS is vlan-base to avoid TCAM Errors.

A basic config is as follows (without srr-queueing);

qos map cos 3 to dscp 24

qos map cos 4 to dscp 34

qos map cos 5 to dscp 46

qos

class-map match-all Voice-control

description VoIP Control Traffic

match access-group name Voice-Control

class-map match-all Gold-Data

description Unity Server

match access-group name Gold-Data

class-map match-all Tanberg

match access-group name Tandberg-Ports

class-map match-all Video

description Video Traffic only

match ip precedence 4

class-map match-all Voice

description VoIP Bearer Traffic

match access-group name Voice

policy-map Tandberg

class Tanberg

set ip dscp 34

policy-map ACCESS-LAN-EDGE-IN

class Voice

set ip dscp 46

class Gold-Data

set ip dscp 18

class Voice-Control

set ip dscp 24

interface FastEthernet4/9

switchport trunk encapsulation dot1q

switchport trunk native vlan XXX

switchport mode trunk

switchport voice vlan XXX

qos trust device cisco-phone

qos trust cos (not needed, depends on IOS/OS)

tx-queue 3

priority high

ip access-list extended Gold-Data

remark Match IP Address of Unity Server

permit ip any host 10.250.17.9

permit ip host 10.250.17.9 any

ip access-list extended Tandberg-Ports

permit tcp any any eq 4224

permit udp any any eq 5445

remark Tanberg and Cisco VT Advant ports

ip access-list extended VoIP-Control

permit tcp any any eq 1720

permit tcp any any range 11000 11999

permit tcp any any range 2000 2002

permit udp any any eq 1719

ip access-list extended Voice

remark Match UDP ports for VoIP Bearer

permit udp any range 16384 32767 any (not recommended)

For CATOS use the auto qos feature along with aux voice vlans. you will want to trust CoS on the trunk links on both sides...

- set port qos 1/1 autoqos trust cos (again you can do the range of gig 1/1-2)

Thanks

[tavery2000]

The auto qos command is a great tool for gathering voice port info, srr- queues, etc. We generally use it as a baseline or start for network wide qos parameter.

[jeffkelso]

I am running CCM 4.02a already. I am noticing the problem with this version, so the issue must be with the QOS. What changes exactly are needed to conform to the new standards? Are we talking about IOS updates?

[togi]

The cisco guys have given me a phone load of P0030600T017 which isn't even out yet to solve this problem. Hopefully, I'll implement it tonight if not this week.

T

[mciarfello]

What version does that translate to when displayed on the phone? Actually, strike that, I already have that image.

Here is my log entry from the 4th of Nov:

• Put firmware back to 5.0(6.0) due to problems with intermittent delay in receiving audio. First manifested itself on the 29th as delayed audio when transferring. David converted the 2nd floor back to 5.0(6.0). On the 30th, we converted users at Health that did a lot of xfers. 11-4-2004 saw audio delays on direct calls also.

Looks like you have two choices: Try the Engineering special image and see if that resolves the issue, or try the 7-1-1 image (which is supposed to contain the fixes in the ES image) and see if that resolves your problem.

PLEASE report back your findings. And thanks

[jerake]

I applied the 7.1.1 load to all 600+ of our IP phones, both 7960's and 7940's. It's only been on for one day, but my "chronic" ppl who, without fail, always tell me if they experience the delay issue and echo have said that it has not happened. I will have to give it at least a week before I can know for sure that the problem has been resolved.

However, another issue has appeared: phones are randomly freezing up. This usually seems to happen when a user is switching over to a call that has come in on call waiting. The only way to fix it is to unplug the phone/plug it back in. Today alone it's happened five times over our three locations. We are using CCM version 3.3(4)sr2, OS version 2000.2.6sr5.

If the freezing keeps happening, I will have to switch back to load 6.0.5. Also, this load apparently can determine whether or not the call is in "secure" mode by showing a small lock icon next to the call timer. I find it odd that this icon shows up on some calls, but not all, and that it's even appearing at all since we're using a version of CCM that doesn't support those enhanced security features.

TMH

[mpaul]

I wanted to wait about a week before I posted back on this one. On Sunday night this past week I applied the latest firmware updates to the 7960/40 phones at our site. Monday went OK so I felt confident in applying it to my client's site. On Monday evning I applied the firmware, 7.1.1 to my client site having this problem. I was on-site on Thursday and asked them if they were hearing the 3 second delay anymore. A quiet poll around the office and nobody has heard the delay, at least since Monday. Still hearing some echo to the PSTN but that's on the gateway of course and I'm massaging that one along.

I did not experience the freezing to the phones Jerake had experienced in the previous post. Both of my sites have 4.01 call managers and that's a little different from Jerake's site.

As a side note, they were experiencing a clipping when they went to unity, 'ello, cisco unity.' As of Monday this problem went away as well. The clipping has stopped and they hear 'Hello, Cisco Unity.' Thanks for the replies on this one. I'll keep an eye on both sites and post back if we have a problem.

[jerake]

An addendum:

Whatever the freezing was, I think it had to do with the security features in 7.1.1 made for 4.0, while we're using 3.3(4) (just my speculation). The freezing problem disappeared when I went to load T017, given to me by Cisco TAC (which seems to be based off of the 6.0.X loads).

TMH

[ikobisher3]

Is there any way to get this firmware (T017) in some other way without opening a TAC case? Is it restricted to be shared ?

Thanks

IK