Unity 5 - Exchange 2007 - MWI Delayed

[nrosselli]

After changing the partner server on our Unity 5 server to our exchange 2007 server MWI is delayed 10 - 20 minutes. All else appears to be functioning normally with Unity and there are no errors in the application log on unity or exchange. Does anyone have an idea what might be causing the delay?

[mciarfello]

Re-run permissions wizard?

Re-run message store config wizard?

Reboot exchange?

Reboot Unity?

Check domain stuff (replication, GC, PDC, etc.) Reverify against Unity design guide.

Reverify any software versions and dependencies, service packs, etc.

Is the message getting delivered to the mailbox on time? Just after the message got delivered, go into UnitySA, under the subscriber, under messages and check to see if Unity thinks the MWI light should be on or off. (should say ON.)

So we want to see if the delay is in delivering the message, unity thinking it got delivered to send the MWI notification off, or delay in MWI being sent out.

Maybe the old entry for the old exchange server is still in SQL causing the hangup. SQL, UnityDB, some table that says "server" or something if memory serves.

I personally might cowboy it and delete the old entry (row). I would officially suggest you open a TAC case or find a Cisco document saying it's ok before doing so.

[nrosselli]

I have re-run both permission wizard and the message store config wizard. I have also rebooted unity. Have not rebooted exchange yet. The messages are being delivered right away just MWI is being delayed. If I go to the subscriber's UnitySA page the indicator lamps says off. If I click the refresh status button for MWI it will trigger the lamp to go on. The same is true when listening to or deleting the message. The MWI will eventually turn the lamp off after 15 minutes or so. If I click the refresh status button the lamp turns off right away. This appears to be the behavior for all subscribers. I looked in the UnityDB and did see a servers table but only my Unity server was listed. I noticed that my exchange 2003 message stores are still in the message store table but so are my exchange 2007 stores but I don't think that is the issue. I have a TAC open and am waiting for a callback. I will update this thread when I resolve the issue.

[mciarfello]

Yea, please let us know.

[nrosselli]

So after over a week of troubleshooting and sending traces to TAC here is there response:

Here is the trace analysis from the AvNotifierMgs trace:

We get two notifications from Exchange in the same millisecond, the second of which appears to be the voice message:

12:46:00:009 (AvDiagnostics_MC,1225,Notifier,24) [Thread 3716] Rosselli Nicholas has 1 messages of type Voice(0x2) Urgent and Normal

Once this is detected, Unity attempts to turn on the user's MWI:

12:46:00:026 (AvDiagnostics_MC,1180,Notifier,0) [Thread 3716] MWI ON Requested: Name= Rosselli Nicholas DTMFID=3147 #NewMsgs=1

Which is completed about .1 seconds later:

12:46:00:104 (AvDiagnostics_MC,1194,Notifier,12) [Thread 4640] Completed MWI task for mailuser=Rosselli Nicholas, extension=3147, status=On

Searching for the next activity for this account, the next hit is:

13:11:39:948 (AvDiagnostics_MC,1212,Notifier,21) [Thread 3716] NotifyQ popped eNOTIFYQ_ACTION_MSG_DELETED [1], mailbox='cn=rosselli cn=Recipients OU=WORKGROUP O=Steinway & Sons', arg1=2, arg2=3, arg3=1, varMessageData= (VT_EMPTY).

Unity determines it needs to turn the MWI off for this user, queues the task, and it completes:

13:11:40:137 (AvDiagnostics_MC,1194,Notifier,12) [Thread 4640] Completed MWI task for mailuser=Rosselli Nicholas, extension=3147, status=Off

The user Roselli receives the message at 12:30 and delete it at 12:46, the delay is entirely in the exchange environment. Unity did not get the new message notification until 12:46:00:009 , at that point, it turned on the user's MWI in less than a second. When the message was deleted, we didn't receive the notification until 13:11:49.948, and once again toggled the MWI in less than a second. The notifications are getting held up on the exchange server. This is further backed up by the fact that at 12:46, we receive notifications for 7 different new/read/deleted messages in about a tenth of a second.

Here is when the notifications come all by a sudden, noticed th time:

12:46:00:041,AvDiagnostics_MC,1212,3716,-1,,Notifier,21,eNOTIFYQ_ACTION_MSG_ DELETED [1],'cn=rosselli cn=Recipients OU=WORKGROUP O=Steinway & Sons',1,3,1,,VT_EMPTY 12:46:00:040,AvDiagnostics_MC,687,3716,-1,,Notifier,1,LOW,CAvNotifier::LogPo ppedEvent,(.),0x00000000

12:46:00:040,AvDiagnostics_MC,1212,3716,-1,,Notifier,21,eNOTIFYQ_ACTION_MSG_ DELETED [1],'cn=rosselli cn=Recipients OU=WORKGROUP O=Steinway &

Sons',1,3,1,,VT_EMPTY 12:46:00:041,AvDiagnostics_MC,687,3716,-1,,Notifier,1,LOW,CAvNotifier::LogPo ppedEvent,(.),0x00000000

12:46:00:041,AvDiagnostics_MC,1212,3716,-1,,Notifier,21,eNOTIFYQ_ACTION_MSG_ DELETED [1],'cn=rosselli cn=Recipients OU=WORKGROUP O=Steinway & Sons',1,3,1,,VT_EMPTY

12:46:00:040,AvDiagnostics_MC,687,3716,-1,,Notifier,1,LOW,CAvNotifier::LogPoppedEvent,(.),0x00000000

12:46:00:040,AvDiagnostics_MC,1212,3716,-1,,Notifier,21,eNOTIFYQ_ACTION_MSG_DELETED [1],'cn=rosselli cn=Recipients OU=WORKGROUP O=Steinway &Sons',1,3,1,,VT_EMPTY

12:46:00:041,AvDiagnostics_MC,687,3716,-1,,Notifier,1,LOW,CAvNotifier::LogPoppedEvent,(.),0x00000000

So at this point you need to check the exchange server to verify what is happening there, why is not sending the information, so I would recommend you to open a case with Microsoft.

I a will open a ticket with Microsoft but I was wondering if anyone knows where to look on exchange to see why the delay is happening.

[nrosselli]

So after a while of going back and forth with Microsoft & Cisco we have finally got to the root of this issue. It turns out Microsoft has an issue with MAPI Notifications being delayed when running Exchange 2007 on Windows Server 2008. They have a hotfix which is supposed to be released with Exchange 2007 SP1 Update Rollup 4. In the mean time MS has provided me with a private fix (KB951251)which resolved my issue.

[mciarfello]

That's awesome follow up. Thanks. I gave you 5 points for answering your own question and coming back with a follow up.

[ahochau_2]

For what it is worth, I just had the same issue with Exchange 2007 running on Server '03. At some point during the install CSA or my AV software (McAfee) enabled the windows firewall. I turned off the windows firewall and the MWI is functioning normally. Why it would be a "delayed" and not a "never" thing with the firewall enabled bends my mind.

Having the firewall enabled also prevented me from being able to upgrade the failover server.

I know the issue is resolved, but this might help someone searching the same symptoms.

[hypnotoad]

I'm having the same problem after going to Exchange 2007 SP1 and Windows Server 2008. I've rerun Permissions Wizard and rebooted with no luck. I am excited to see that you found the fix. Is there any way you could send me that hotfix? I really don't want to burn incidents on this.

--Patrick

[ranpierce]

Can I get that hotfix?

Randy

[nrosselli]

The fix I have is only for exchange 2007 SP1 with Update Rollup 2. If you have installed Update Rollup 3 my fix will not install.

[kelvin.blair]

If it is a hot fix, generally you can call Microsoft support reference the KB and they will send you the hotfix without burning a case.

[tfelmly]

Does anyone have this patch and willing to share? We are having the same exact issue with delayed MWI's (Unity 5.1, Server 2008 with Exchange 2007 SP1).

Microsoft CANNOT find this patch in their database anymore. We would really like to resolve this ASAP. We do not have any rollups applied at this point. If the patch will only work on Rollup 2 then we can get to that point.

Thanks!

Todd

[nrosselli]

I have the hotfix and yes you will need to be on rollup 2 to install it. I sent you an email

[tfelmly]

Another gold star for you! Thanks for the assistance. Instant MWI's again :)