Voice and IP Phone response delay

[prestonhartley]

All,

We are experiencing an issue with voice delay and even phone response delay during the beginning of a call.  When a user picks up the phone there is a noticeable delay before any audio is actually heard (2-5 seconds). Users also complain that they are unable to answer the call, meaning they hit the speaker phone button on the phone but it doesn't repond for a few seconds.

There is an ASA (version 8.4(3)) that is in between the call managers and the remote locations with the IP phones.

I have already checked QoS from the CUCM all the way to the remote end.  It is good.  I have also not found any packet loss over my network or the carrier network.  I do not have access to the data center or the ASA.  I currently have the people responsbile trying to prove out their side but I need some ideas on possible problems?  Also, has anyone else experienced this and if so what was the problem and the fix?

CUCM Version: 8.6

Phone Load: SCCP42.9-2-3S

Thank you!           

Message was edited by: Preston Hartley Update: The problem does not present itself on every call, when it does happen I see multiple packet retransmissions in wireshark.

[Jonathan Schulenberg]

The next step I would reocmmend is to run a Wireshark at both the phone and the CUCM node to which the phone is registered. This would let you see what happens: is the packet lost, arriving too late, corrupted, fragmented, etc.

https://supportforums.cisco.com/docs/DOC-11599

The ASA code is capable of doing Layer 7 inspection/manipulation of the SCCP traffic. It could have a bug where it doesn't understand the newer SCCP version or it could be rate limiting the connections to each CUCM server.

Another possibility is that there is traffic shaping somewhere which is delaying delivery of the packets. Again, seeing both sides is probably your best next bet.

Please remember to rate helpful responses and identify helpful or correct answers.

[prestonhartley]

Thanks for the response.  I ran the packet capture on both sides.  I am seeing retransmits on the remote side where the phone resides.  On the CUCM side I am seeing lots of unseen acknowledgements.  It thought they were just at the beginning of my capture but they run throughout the capture.

[prestonhartley]

I have a change in today to stop the SCCP inspection and just let it pass through.  Will post if it works or not.

[prestonhartley]

Ok, an update:

After reviewing packet captures further it appears we are dropping traffic somewhere which is causing the retransmits on the phone side.  I am not sure where the packet loss is coming from or what is causing the re-transmits.  Thoughts?

We are not rate limiting in to the CUCM servers, I do have a traffic shape on my ethernet circuits though.  However, I do not see any drops on my outbound policy or interface. 

The skinny inspection on the ASA's has been stopped about 30 minutes ago.  So far I haven't heard of any complaints but will need to wait until the end of the day to verify whether it is working or not.

[prestonhartley]

Problem has been resolved.  The solution was removing the skinny inspection on the firewall.  It is just one command, no skinny inspect.  Once we did this a lot of the little problems and the delay were resolved.  Having an ASA in between your IP phones and your call managers is not a very good design.

[Rodrigo Gurriti]

prestonhartley is that a bug ?

[jjbaker]

Actually PrestonHartley you are very wrong.

Just because you or your vendor did not configure it correctly does not make it a bad design.  Fire-walling off you voice apps is actually a very good design and most experienced engineers would recommend this as best practice design just as they would recommend for any other mission critical servers.  I understand you may be frustrated from the troubleshooting you have endured but extending poor advice about something you clearly are uneducated in is not going to help anyone.  Can you imagine what this world would be like if every time something did not work properly from misconfiguration (you know.. that human error thing) someone just throws their hands up in the air and says "well that's not a good design".

[sebastien_michelet]

I found the exact same issue (same symptoms and same fix) with the following versions: CUCM 9.1 and ASA 8.4.7. Disabling skinny inspection fixed the issue. I have a TAC case pending to find out if there is a fix in a later ASA release (9.x).

[Ben Austin]

I have the same issue with CUCM 9.1 and ASA 8.4.7. Did you find out if there is a bug fix in a later release. I was not able to find any known bugs that matched these symptoms in the bug search.