cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
10264
Views
5
Helpful
30
Replies

Notice: UC560 Voice Mail / AA issue being investigated

Steven DiStefano
VIP Alumni
VIP Alumni

We are currently investigating several cases where UC560s running on  the latest SWP 8.0.4 are experiencing Audo Chop and distortion with  Voice Mail Greeting & Recorded Voice Mail Messages.

We  believe we understand the issue now is with CUE 8.0.2 and are close to implementing a  solution

Please watch CSCti85760  in your Bug Toolkit for status.

In the mean time if  you do experience this issue, use CCA to reload CUE (we have seen this  help when the problem occurs and found very agressive proactive action  of every 2-3 days will prevent this).

If you have a factory UC560 ready to install,  please only upgrade the IOS to 15.0(1)XA3A.  Do not upgrade CUE (Stay with CUE 7.1.3)

We removed the UC560 SWP from CCO until we have a solution. 

*If you need 15.0(1)XA3a IOS image, you can get it from 8.0.4 software pack for uc540 platform which is still posted and has no problem.

NEW INFO:

Downgrading is NOT an option for CUE.  DO NOT attempt to downgrade CUE to 7.1.3 if you are running CUE 8.0.2 as it is not supported.

30 Replies 30

Steven Holl
Cisco Employee
Cisco Employee

I've got someone seeing this on an AIM running 7.2.2, as well, and am currently working with CUE developers on it.  May be realted, or may not.

I see a high jitter value (~800ms) on a single pack from CUE to CME.

I'm working with development in regards to this, and currently pending some coredump collections and rtp pacer debugs.  If it ends up being a new bug, I'll try to remember to report back to this thread since it could be related.  My issue may be related to CSCtf70144 and just never ported to the AIM platform, but I know CSCtf70144 was fixed for the UC5xx platforms.

finalconnect
Level 3
Level 3

Steve,

As a Cisco partner, I appreciate the honesty that such a post required. Most vendors including Cisco of the past, would simply cover this up as just another "bug", but this is not just another bug. This "bug" impacts everyone using a system and the core of what a phone system is supposed to provide not just a feature.

As a partner having two customers with this problem, I appreciate the effort Cisco has put into helping me, the partner, and my customer. In the end, we need the problem resolved as quickly as possible so that it does not delay future orders we have in the pipeline.

Here is how to RELOAD ONLY CUE if you find you need to using CCA 2.2.5 (click to improve picture)

Steven DiStefano
VIP Alumni
VIP Alumni

Latest Status and request for Partners who want to try workaround.

As we now have reproduced the problem and have a team of developers looing at this as I write, we also are taking the action to produce a work around.

We will be using the Event Manager to create a sort of CRON job that can kick off and reboot CUE every morning at 4AM, since we know that it takes about 2 days to get into the problem where CUE is sending packets with 40ms packetization delay (should be 20ms).

We are advising Partners that rebooting CUE (not the whole UC560), will prevent the problem from occuring, but we realize this is a manual task.

When we do have this ready, are there any Partners who would like to use it?

Send me an EMAIL sdistef@cisco.com

We are also considering the following if we cannot figure out the fix in the next couple days, but its a bit complex and time consuming.  Let me explain.

Putting CUE back on CUE 7.1.3 will eliminate this problem.  However, the data structure of CUE 8.0.2 is NOT backward compatible with CUE 7.1.3 (so it cant be saved and restored).   Also, as you all know, these systems are configured in such a way where configuration is sent to BOTH IOS/CME and CUE, so there is no way to simply restore CUE to factory defaults and load the old CUE SW and rebuilt only it.   The only way to go back to CUE 7.1.3, is to factory default the entire system, downgrade CUE (I would probably factory reset again after that) and then rebuild the system.  This obviously will take 3-4 hours if you have all the data you need.

The only value an RMA option would provide is getting you on older code, and you would upgrade IOS to 15.0(1)XA3A (but leave CUE alone on 7.1.3) and then rebuild it in a staging (offline) way.  This is very possible using text files for phones and users as I have documented in TEL#25 on this community.  Then it would be more like a physical cutover of one box to the other.  You would have to transfer licenses and make absolutely sure the HW VWICs are identical in both systems to have this work, and this also is manually intensive, so we are really not encouraging this option, but want to let you know it is being considered on the table at this point.

This is IOS CLI, entered on the UC560 with privilege level 15 in Configuration Terminal mode.   When done, save it using 'write memory' which copies running to startup-config.  While IOS will not reboot with this, it's still a good idea to save it in case the router looses power or reloads due to something else.


This is intended to be a short term work around only, until we address the root cause of the CUE packetization problem that seems to occur after a period of time ranging from a couple days to 4-5 days.   The reboot of CUE initializes it preventing the longevity issue.  UC560 CUE reboot should take about 5 minutes on a UC560 with about 50 voicemail boxes.

The 4am is a time we picked, but you can adjust if that's not a good time.  When CUE reloads, all call processing continues as normal, just voice mail and AA are reported as unavailable, please try later.

NOTE:  PLEASE REMEMBER TO TAKE THIS OUT WHEN WE FIX THE ROOT CAUSE


Since this is fairly new, the SBCS is being briefed on this as well so this is pretty much real time.  We tried this script on several systems and it worked well.

event manager applet cue_reset

event timer cron cron-entry "0 4 * * *"

action 1 cli command "enable"

action 2 cli command "service-module integrated-Service-Engine 0/0 reset" pattern "confirm"

action 3 cli command "y"

wr mem

***If you made any changes in CUE and didnt save via CCA, please do that before executing this script (normal best practice anyway here).

Some have asked how do you know if CUE was reset.  here is how...

UC_560#service-module integrated-Service-Engine 0/0 session
Trying 10.1.10.2, 2002 ... Open
banner login ^Cisco Configuration Assistant. Version: 2.2 (5). Mon Sep 27 17:08:07 EDT 2010^

User Access Verification

Username: ciscoadmin
Password: ************
se-10-1-10-1# show hardware
se-10-1-10-1 uptime is 0 weeks, 0 days, 1 hours, 44 minutes
# of Processors:              2
CPU:                          e500v2
Revision:                     3.0 (pvr 8021 0030)
BogoMIPS:                     133.12
SKU:                          Unavailable
UDI:                          Unavailable
Chassis Type:                 UC500
Chassis Serial:               FHH1313P04X
Module Type:                  Integrated-Service-Engine
Module Serial:               
Compact Flash:                2097MB
SDRAM (MByte):                1024
se-10-1-10-1#

There have been a few inquiries on how to change the time of day that the EEM script reloads CUE. The script that Steve mentions reloads the CUE everyday at 4AM.

event  manager applet cue_reset

event timer cron cron-entry "0 4 * *  *"

action 1 cli command "enable"

action 2 cli command "service-module  integrated-Service-Engine 0/0 reset" pattern "confirm"

action 3 cli  command "y"

To change the time of day when this script executes modify the "event timer" configuration line. Here is a description of the different parameters-

http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_06.html#wp1157622

cron-entry

Text string that consists of five fields separated by spaces. The fields  represent the times and dates when CRON timer events will be triggered.  Fields and corresponding values are as follows:

minute—A  number in the range from 0 to 59 that specifies when a CRON timer event  is triggered.

hour—A  number in the range from 0 to 23 that specifies when a CRON timer event  is triggered.

day-of-month—A  number in the range from 1 to 31 that specifies the day of the month  when a CRON timer event is triggered.

month—A  number in the range from 1 to 12 or the first three letters (not  case-sensitive) of the name of the month in which a CRON timer event is  triggered.

day-of-week—A  number in the range from 0 to 6 (Sunday is 0) or the first three  letters (not case-sensitive) of the name of the day when a CRON timer  event is triggered.

Is it supported to upgrade new UC560 systems with IOS and phone loads to 8.0.4 and leave CUE 7.1.6?

Hi Dan,

Yes, it is supported.

- Anna

finalconnect
Level 3
Level 3

Is anyone having issue with this work-around? I have one client that having voice mail issues after the auto reboot. I have now had 3 consecutive days where the cue has failed to reload properly after the routinge, forcing us to reboot in and manully reload the CUE again.

I am trying to figure out if there is anyone else having that same problem or similar CUE issues when leveraging the work around.

CISCO: Is there any debug I can run to help Cisco support figure out the cause of the hanging?

NOTE: This is not garbling. The last time it happened, the auto attendent greeting was playing, but no extensions or options worked from within the auto attendent. When I "serv inte 0/0 sess" from clI, it would go into EXEC mode (unpriviledged) with the factory standard name "integrated engine>". Again, the correct AA greeting would play, but no extensions would connect nor would the AA options connect.

THanks Cisco. Will be trying "reload" instead of "reset" early tomorrow morning.

UPDATE1: CSCti85760 -  Choppy VM / AA - SWP 8.0.4

In lieu of the current manual proactive reboot procedure as a workaround to  avoid this issue we have an EEM script to automate this effort.

We have been able to reproduce the voice quality issue experienced with CUE in our lab over the past week on UC560. Since then we have been running Release Candidate images on our UC560’s and have seen positive results. We will be conducting more tests on the Release Candidate images before it is made available for general use. If you would like to help be a part of the feedback process before a general release is made available please send your request to uc500-rc-request@cisco.com with your Cisco.com User ID in the subject line of the email. You will then receive an email notification with instructions on how to download the firmware.

Removed EEM script from this post since we now have RC1 and RC2 images available>

If there are any questions/concerns feel free to reach out to me at antyeung@cisco.com

Thanks,
Anthony

Message was edited by: Anthony Yeung

Has anyone had "real" world success with RC3 solving the Audio Chop issue.  My customer has been dealing with this for quite a while and I don't want to implement something that doesn't have in production successes.  They don't deserve anymore anxiety with this system.

Regards,

Rick Turner

CCIE (Voice) #25838

Hi Rick,

From the customers that have provided feedback we have not received any reports of CSCti85760 reoccuring. Our engineering team is also confident that the Release Candidate image properly addresses CSCti85760. Based on the engineering effort, results from internal tests, and the positive responses we received with respect to the Release Candidate image we will be bundling it into the next Software Pack Release.

If you have any questions or concerns please feel free to reach out to me directly - antyeung@cisco.com

Regards,

Anthony

Do you know about when the new Software Pack will be released?