I have a Call Manager 4.01 Sr1a implementation with approximately 55 7940/7960 IP Phones. They are connected with 2 3560 48 port inline power phones, a 3550 24 port and then a 3750 24 Port Gigabit switch. I have a voice and data vlan with all the voice servers and phones in the voice vlan. I recently upgraded to the 6.05 firmware load on the phone and noticed a strange thing. For off-net and on-net calls sometimes, maybe 3 out of 10 calls users will hear a delay in the audio. Once a call is established you try to speak and hear nothing for 3 seconds. After the 3 seconds you can hear the other caller. Some delay in the RTP stream. I've opened a TAC case and they helped me take the firmware back to 6.04 but I am still hearing the same delays. They've wanted me to gather packet traces but since this is so random it is difficult to track down. I essentially have to do a port span and be right at the phone to capture the traces correctly. Wondering if anybody has seen this before?
I've swapped switch ports, ethernet cables, phones. I'm using auto qos voip cisco-phone on the switch ports for the QOS features. There's no inter-vlan routing going on for the voice traffic.
The switches are in a hub-and-spoke design with all the switches hanging off the gigabit switch. They're uplinked with each other via copper gigabit.
I have the exact same problem, however, we are running CM 3.3.3 SR4, and the topology is different. But the problem is exactly the same. The only common thing is that the phones are on 6.05. I know this is not an answer to your problem, but maybe if two people have the same problem, somebody in the Forum will answer more promptly.
And you are right, this is impossible to catch with a Sniffer. I once had this problem with fw 3.x, about a year ago, and the TAC engineer said it was a bug in the firmware where the ARP request from the phone has to wait 10 seconds or less, until the stream gets established. However, I would think this bug was solved then, and it should not reappear.
We have observed this on some customers with older echo-cancellation code on their gateways, the G.165 ECAN as opposed to the new G.168 ECAN. Assuming you are using IOS gateways, go into the voice-port configuration and set 'no echo-cancel enable'. We didn't troubleshoot extensively after pinning it down to the ECAN, but we thought it had something to do with the G.165 suppressor feature where the conversation ends up being half-duplex for the first few seconds (by design). Obviously you wouldn't leave this in permanently, but run through a few dozen test calls and see if you can replicate the problem with the ECANs out of the loop. Upgrading IOS to the most recent IOS revisions, which support the new G.168 ECAN and use it by default, would resolve the issue if that's the case.
There are some other issues that could cause it. You could look at CSCdz52758, which gets flushed out by a hardware issue specific to the 3640 router if that's what you have for a voice gateway. You can see if that's the case by double-tapping the ? button on the phone and seeing if the MaxJtr value has gone raving bonkers. Even if it hasn't, use of the ? button should be able to help you figure out if the phone is receiving RTP for those first few seconds by comparing approximate Rx and Tx packet counts. So long as you don't put the call on hold (and even this doesn't matter on very recent phone loads) the counts should be reasonably even.
Thanks for the response. However, my problem happens between IP Phones connected to the same switch. So, there is no gateway in the middle. The switch is a 6500 and the phones are connected in the same VLAN (different module, however) and AutoQos is configured on the 6500.
This is the bug that I was referring earlier, but it is supposed to be fixed by now:
CSCeb19725: Intermittent one way, no way audio for the first 10 seconds between IP phones with no gateway involved. CallManager traces show the calls being connected but no voice stream is being passed. Review of a sniffer trace between phones show that the streams do not get established until the phone issues an ARP reqeust roughly 10 seconds after the skinny message comes in to cut through audio.
Maybe it is a different thing, but what I've heard it could be also present in 5.0.6 IP Phone fw.
Just to let everyone know, I have 333sr4a, phone load 6.0(5) and TAC told me to roll back to 6.0(3) for the 7940/60 phones (if you have a 7914 sidecar use the compatible 4.0(0) fw).
Hopefully this will work.
I do not see rolling back to older firmware as a good solution. I'm holding out for Cisco to release new firmware that will address the issue in their current release.
Are there any plans for this in the near future?
I have the same issue with delay between ip phones. This delay only appears to happen between 6509 switches (over the gig connection). So far, tac has me going in circles. I'm running CCM 4.0.2 with the latest phone load. The biggest problem is i'm unable to reproduce the problem when tac wants me too. All QOS parameters are in place and I still get about a 3 second delay. Very strange and difficult to troubleshoot.
From reading these messages and taking note on my own network, I believe that the issue is independent of device types. It appears to happen randomly on various types of switches and gateways. From what I see it must be a telephone firmware issue.
I've got a customer experiencing the same problem. Is even happening between IP phones in the same 6513 switch.
Call Mgr 4.0 , phones have 6.05 load.
Has anyone been given a fix ? What is Cisco saying about it?