Unity Connection, MWI synchronicity and the moon is made of cheese

[steakandeggs]

Is there a document out there somewhere listing the top ten causes for MWI synchronization problems out there somewhere? I had one user where MWI would not go on or off. Everyone else's MWI worked. I don't know what was different about this user BUT found that the only thing that fixed the problem was to reset the MWI by performing a reset under the user's mailbox in the CUC admin interface. I've read that if the phone system is offline when a mailbox status changes, MWIs get out of synch. Is there any other reason MWIs would get out of synch? And BTW does "phone system offline" only mean this: call setup to CUC has occured, connection with CM goes, caller completes leaving a message and then hangs up.

Finally, how does CUC determine that MWI doesn't need to be lit? Below is the cuNotifyer meesage I found regarding the mailbox that needed to be manually synced:

03/22/2010 20:17:17.887 |11764,,,Notifier,12,Billy Bob, mwi extension=8734: doesn't need MWI update with (Voice+Fax+Text)mail count= 11 and MWI state=ON|

I'm not sure I understand why CUC couldn't simply blindly send the mwi on or off every time the mailbox is fully read or every time the mailbox is unread in any way immediately after an event like leaving or check a message has occurred.

Finally, is there a CUC document available to the public or partners that highlights the architecture of CUC like there is with Unity?

[David Hailey]

If the moon is made of cheese, then here is the wine:  http://www.cisco.com/en/US/docs/voice_ip_comm/connection/7x/troubleshooting/guide/7xcuctsg050.html

Hailey

Please rate helpful posts!

[steakandeggs]

I've read this document, is it correct to say that MWIs only become unsynchronized in the following cases:

•After a server is restored by using the  Disaster Recovery System.

•After upgrading a system.

•After a WAN outage in a system that has  distributed voice messaging through Cisco Unified Survivable Remote  Site Telephony (SRST) routers or Cisco Unified Communications Manager  Express routers in SRST mode.

What does distributed voice messaging mean in the context above? Centralized CUCM and decentralized CUC? Also, what exactly is CME in SRST mode? What I'm trying to understand is WHY do MWIs get out of synch.

B

[David Hailey]

Your question is actually fairly wide in terms of what could cause

issues with MWI. I am on the road at the moment so I will follow up

more in a bit when I'm back at the office. BUT, it would incorrect to

say that those are the only reasons MWI could get out of sync. As for

the architecture questions, I'll include more in a follow-up as needed.

Sent from my iPhone

On Mar 23, 2010, at 1:38 PM, steakandeggs

[David Hailey · Accepted solution]

OK.  So, the 3 scenarios that you noted are actually 3 cases where Cisco recommends that you go ahead and perform a resync of MWI after such an event has occurred.  They are not the only cases where MWI could get out of sync; however, it is probably safe to say that it's highly likely that MWI would always be out of sync in these cases so that's why it's recommended procedure.  There are a lot of questions to be addressed so I'll cover as best as I can.

First, let's look at how MWI works at a high-level.  First, a .wav file (message) is queued so CUC checks that the mailstore is online.  If so, the message is left in the user's mailbox and then CUC initiates an MWI event using the MWI on/off numbers.  CUC is essentially going off-hook as the Subscriber and dialing the MWI on/off number.  Now, the synchronization of MWI moves outside of just CUC as CUCM gets involved.  CUCM updates the DB and real-time info which is replicated to the cluster Subscriber servers.  The CUCM subscriber the user's phone is registered to then sends a SCCP message to the phone letting it know to turn MWI on/off (in this scenario, the light would go on because we're assuming a new message is being left for a CUC subscriber).

So, with Unity and/or CUC there are potential areas of failure which are well-documented in the link I sent you and there is also a Unity MWI troubleshooting guide that has a wealth of information in it as well.  Sometimes, it's as simple as the subscriber account is not configured to update MWI.  However, you could also run into scenarios where the system doesn't have enough ports to accomodate all of the calls it is taking and MWI requests.  In these cases, you have to dedicate more ports for MWI or potentially add more ports (assuming you're not maxed out already).  Since CUC has to talk to CUCM to actually get the phone to light up, another issue could be a network latency issue between those 2 systems.

On the CUCM side, DB replication within the cluster could cause issues with MWI.  If replication is bad, then Subscriber servers may miss a DB update(s) that contains one or more MWI events.  You can use the CU Reporting tool (or web to a phone) to compare what CUCM thinks MWI should be vs. what Unity/CUC says it should be.  Another item of importance here is dial plan.  The MWI numbers need to be unique throughout the cluster and your voicemail ports need to be configured with the proper CSS(s) so that when CUC goes off-hook, it can dial out properly.

For architecture questions, the best thing to do is look at the SRND and read up on the various deployment models.  The link for the Cisco Voice Messaging section of the 7x SRND is http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/srnd/7x/vmessage.html.  I'm not passing the link blindly.  I have to go back to it regularly to keep myself fresh on the best practices for Cisco.  You'll also find info on CME and SRST there as well.  In the context of your question, SRST and CME are very similar at the core but CME has more features.  So, you can configure a failover architecture where phones failover to SRST or failover to CME based on requirements and etc.

I hope this helps as there really is no one single (or even simple) answer that can explain exactly why MWI can get out of sync.  There are a lot of moving parts in the equation.  Definitely look thru the SRND and the Unity MWI troubleshooting guide.  While it focuses on Unity and not CUC, it still contains a lot of information and scenarios where you'd have to troubleshoot MWI synchronization.

Hailey

Please rate helpful posts!

[steakandeggs]

Your thoroughness is encouraging, and the VM SRND is very good. I guess what I'm looking for is details on functionality at the basic application level. Let me present a couple of very specific questions:

You say:

If so, the message is left in the user's mailbox and then CUC initiates  an MWI event using the MWI on/off numbers.

I say, according the log message I posted earlier, CUC thought that the MWI was already lit, and did not even try to signal CUCM. My question is this: what is the algorithm CUC runs through to decide to send the MWI signal to CUCM? The answer to this would completely clarify half the equation.

Secondly, once CUCM receives an MWI on light it appears to make a decision as the whether or not the phone MWI is actually on. Is this true? If so, that is another point of potential failure, noting your point about clustered CUCM issues, etc. Third, what is the algorithm CUCM runs through to decide to send the MWI signal  to the phone?

Finally, to whine just a bit: I never have issues with read/unread issues with email. I know the systems aren't exactly analogous, but there has got to be a better MWI implementation than this. I suspect that MWI works this way because of integration with legacy PBXs. Why can't CUC always send MWI upon every receipt of every voicemail? And why does CUCM need to make a guess as to the actual MWI state of the phone? I think the real answer is to fix the whole thing by running an MWI IMAP client on every phone that pulls or gets pushed an MWI status directly off the CUC. TINY static application. Hardly any memory or CPU required. Happiness.

[David Hailey]

Alright, let's see if I can give you what you're looking for:

What makes Unity/CUC signal an MWI on/off event?  MWI is pretty uniform in operation so don't get caught up in the mix of Unity and Unity Connection in the links I've provided.  The answer to your question on that is here: http://www.cisco.com/en/US/docs/voice_ip_comm/unity/5x/troubleshooting/guide/ex/5xcutsg080e.html#wp1049419

Basically, the voicemail system will check to see if the user has unread messages in the Inbox - if not, it signals MWI on for new messages.  If the user already has unread messages (i.e., the system believes MWI to already be on), then there is no need to re-signal MWI.

In turn, if you read a message in the Inbox the system will check to see if there are any other unread messages.  If yes, MWI stays on.  If all messages are read, the system will signal MWI off.

Now for CUCM, the MWI state for each phone is written to the DB.  So, it's a DB check - MWI is either marked as on or off for each phone.

As for your whine, there are some fundamental issues I see with your proposal.  The first is that a CUC cluster only supports up to a maximum of 7500 IMAP clients in an HA cluster - this assumes that 1) you are using IMAP Idle (standard IMAP is 75% reduction in capacity) and 2) you are using 7845's for server hardware.  If every phone is running IMAP to implement MWI and you also have users that are leveraging IMAP for Integrated Messaging then your VoIP solution can only scale so far before you have to add additional CUC servers just to support MWI via IMAP for your phones.  Some of the clusters I've built and worked on have been upwards of 15,000 phones so this wouldn't be a great solution IMO.  The second issue here is that voicemail is not associated specifically with a phone.  It's associated with an extension as far as the phone is concerned.  However, extensions aren't always static and they aren't always only configured on one phone as you know.  So, unless the IMAP client is running as a service on the phone that can accomodate for things like EM, DN changes, etc...well I think you see where I'm going here.

Again, I hope this helps sort things out for you a bit.  I've rarely had chronic issues with MWI in my implementations so I don't mind the basic operation as it's worked that way for years and is sort of an "if it ain't broke, don't fix it" deals.  Even third party MWI clients (such as those used by Microsoft UM clients) implement the basic operation in the same way.  The process names may be different but the primary operations are the same.  If you still have questions, well...this thread isn't going anywhere.

Hailey

Please rate helpful posts!

[William Bell · Accepted solution]

Hailey has addressed the architectural and fundamental processes, but I did want to come in with a basic answer to one of your earlier questions.  You asked (paraphrasing here) if there was a way to tell Unity to send another MWI "on" message to CUCM even if Unity thinks that the light is already "lit".  If you have Unity 7.1,  go to the phone system configuration (Telephony Integrations>Phone Systems) and then edit the phone system object pointing to your CUCM cluster.  There is a parameter called "Send Message Counts".  Enable this option.

Now, this feature is supposed to send updates to the phone so that it shows the number of voicemails that are in the CUC inbox.  With SCCP integrations, the full intent of this feature is not available.  However, I recall reading the feature guide and as I was reading this thread it made me wonder "what if...".  So, I tested this out.  If you check that box, CUC will indeed send a message notification to the CUCM each time the mailbox receives a new message.  This works regardless of whether CUC already has a MWI-On state set or not.  Now, that doesn't fully satisfy your needs but it does address at least one of the issues you saw and noted in your original post.

Another thing I guess you could look at and see if it works for you is tweaking the setting and timers related to resending a successful message waiting indication.  Go to Telephony Integrations>Port Group.  You will see your defined port groups.  Edit one of the port groups.  In the "Port Group Basics" configuration page you will see a section with multiple options for Message Waiting Indication.  Take a look at "Retries After Successful Attempt" and "Retry Interval After Successful Attempt".  Basically, these two parameters work together and it is a rudementary way to have the CUC system resend the MWI multiple times.  The default behavior is to only send the initial notification.  You can configure the system to send up to 10 notifications and you can specify upto 100ms in between each successful notification.  I set this up in my lab and testing with MWI-On and MWI-Off scenarios.  It works as advertised for both.

If you really have that many problems with MWI.  You could also look at using the MWI resync schedule to be more aggressive than what most folks use.  My experience is that most people set their scheduled MWI resync once a day (usually in the wee hours of the morning).  With CUC, you can tweak the schedule to run more aggressively if you really needed this.  Though, you have to take all things into consideration with respect to port allocation during your busiest hours.  In general, I wouldn't recommend doing more than one MWI sync a day.  If you have that many MWI issues, you have something else going on in your environment that needs attention.  As Hailey said, MWI is simplistic but it works most of the time.

Outside of the above small contributions, I think that Hailey laid out all of the MWI ins and outs.  Maybe one of the workarounds above will satisfy your needs.  Though, none of them are exactly analogous with IMAP.

HTH.

Regards,
Bill

Please remember to rate helpful posts.

[steve_reckley]

In my scenario when a new message is left it will not turn the light on.  If I trigger the sync from the phone system in unity the light will turn on.  If I listen to the message the light will remain on.  If I again trigger the sync it will turn the light out.  Is there a service that might need to be restarted?