Re: How To: Kill a voicemail that is looping

aaronkent · ‎08-19-2004

Unity_[server-name] account shows 53GB of items deleted in 7 days.

A user was over his size limit and the message could not be delivered. We fixed the users limits and made arrangements to reclaim the space from our exchange servers Unity Information Store.

Now that the message can be delivered, we see it is not the case, it is still looping. To be a bit more technical, our transaction logs and message tracking center shows that message is still replaying back and forth in a cycle from Unity_(server-name) to unity.messaging@sitename.com. I added the Unity_(Server-name) to my mailbox so I could see what is happening, and shows the VM going into the drafts from the inbox and back out over and over; up to 95,000 times a day.

Any help would be appreciated.

aaronkent · ‎08-19-2004

Here is a parsed report of 1 days messages...

Log Stats:

Messages 99,698

Recipients 104,893

Log Records 786,156

Message Bytes 21,524,230,167 / 20,527.11 / 20.05

Largest Message 11,780,005

Encrypted Messages 0

TOP 10 RECIPIENTS - Sorted by Message Count

EMAIL ADDRESS MSG COUNT BYTES

============= ========= =====

/O=sitename/OU=ITCVA/CN=USERS/CN=UNITY_UNITY 94076 21456365108

/O=sitename/OU=ITCVA/CN=RECIPIENTS/CN=MFERGUSON 180 211870

/O=sitename/OU=ITCVA/CN=CONFIGURATION/CN=SERVERS/CN=ITCEXCH1/CN=MICROSOFT PUBLIC MDB 162 479099

luis.sato@apptis.com 131 133786

/O=sitename/OU=ITCVA/CN=RECIPIENTS/CN=BFORD 95 297081

/O=sitename/OU=ITCVA/CN=RECIPIENTS/CN=JFERGUSON 80 324261

mferguson@planetgov.com 74 197650

C=US;A= ;P=sitename;O=ITCVA;DDA:SMTP=FRANSWA(a)MAIL.PLANETGOV.COM; 74 315459

/O=sitename/OU=ITCVA/CN=RECIPIENTS/CN=DBELL 60 207776

/O=sitename/OU=ITCVA/CN=RECIPIENTS/CN=JHOUSE 60 173675

EVENT TYPE DISTRIBUTIONS

========================

SMTP Begin Submit Message:97870 / 12.45%

SMTP: Started Message Submission to Advanced Queue

SMTP Submit Message to AQ:97643 / 12.42%

SMTP: Message Submitted to Advanced Queuing.

SMTP Submit Message To Cat:97595 / 12.41%

SMTP: Message Submitted to Categorizer

SMTP Message Categorized and Queued for Routing:97591 / 12.41%

SMTP Message Categorized and Queued for Routing

SMTP Submit Message To SD:96159 / 12.23%

SMTP Store Driver: Message Submitted from Store

SMTP Local Delivery:95504 / 12.15%

SMTP: Message Delivered Locally

SMTP SD Local Delivery:95399 / 12.13%

SMTP Store Driver: Message Delivered Locally to Store

SMTP Message Queued for Local Delivery:95266 / 12.12%

SMTP Message Queued for Local Delivery

SMTP Begin Outbound Transfer:3846 / 0.49%

SMTP: Started Outbound Transfer of Message

SMTP End Outbound Transfer:2829 / 0.36%

SMTP: Ending Outbound Transfer

SMTP Message Routed and Queued for Remote Delivery:2056 / 0.26%

SMTP Message Routed and Queued for Remote Delivery

Report transfer in:1244 / 0.16%

A delivery receipt or NDR was received from a server, connector,or gateway.

SMTP BadMail:1113 / 0.14%

SMTP: Message Sent to Badmail

Message rerouted:699 / 0.09%

The message was routed to an alternate path.

Message transfer in:432 / 0.05%

The message was received from a server, connector, or gateway.

SMTP AQ Failed Message:272 / 0.03%

SMTP: Advanced Queue Failed to Deliver Message

SMTP NDR All:211 / 0.03%

SMTP: Non-Delivered Report (NDR) Generated

SMTP Message Scheduled to Retry Local Delivery:178 / 0.02%

SMTP Message Scheduled to Retry Local Delivery

Message transfer out:83 / 0.01%

The message was sent to a server,connector, or gateway.

SMTP SD Gateway Delivery:83 / 0.01%

SMTP Store Driver Submitted Message to MTA

SMTP Message Routed and Queued for Gateway Delivery:83 / 0.01%

SMTP Message Routed and Queued for Gateway Delivery

GRAND TOTALS

============

Messages 99,698

Recipients 104,893

Log Records 786,156

Message Bytes 21,524,230,167 / 20,527.11 / 20.05

Largest Message 11,780,005

Encrypted Messages 0

Exparse completed after 64.03 Seconds. 12278 entries per second.

aaronkent · ‎08-19-2004

Here is a small snapshot log of the message tracking log file...

# Message Tracking Log File

# Exchange System Attendant Version 6.5.6944.0

# Date Time client-ip Client-hostname Partner-Name Server-hostname server-IP Recipient-Address Event-ID MSGID Priority Recipient-Report-Status total-bytes Number-Recipients Origination-Time Encryption service-Version Linked-MSGID Message-Subject Sender-Address

2004-8-16 0:0:0 GMT - - - MAILSERVER - /O=INTELLTECH/OU=ITCVA/CN=USERS/CN=UNITY_UNITY 1027 17E80BE1935EC143B0334DFE778FCC85017C2047@mailserver.domain.root.inc 0 0 172563 1 2004-8-16 0:0:0 GMT 0 - c=US;a= ;p=INTELLTECH;l=MAILSERVER-040816000000Z-5752 FW: Message from 7275731605 EX:/O=INTELLTECH/OU=ITCVA/CN=USERS/CN=UNITY_UNITY -

2004-8-16 0:0:0 GMT - - - MAILSERVER - /O=INTELLTECH/OU=ITCVA/CN=USERS/CN=UNITY_UNITY 1019 17E80BE1935EC143B0334DFE778FCC85017C2047@mailserver.domain.root.inc 0 0 172563 1 2004-8-16 0:0:0 GMT 0 - - FW: Message from 7275731605 - -

2004-8-16 0:0:0 GMT - - - MAILSERVER - /O=INTELLTECH/OU=ITCVA/CN=USERS/CN=UNITY_UNITY 1025 17E80BE1935EC143B0334DFE778FCC85017C2047@mailserver.domain.root.inc 0 0 172563 1 2004-8-16 0:0:0 GMT 0 - - FW: Message from 7275731605 - -

2004-8-16 0:0:0 GMT - - - MAILSERVER - /O=INTELLTECH/OU=ITCVA/CN=USERS/CN=UNITY_UNITY 1024 17E80BE1935EC143B0334DFE778FCC85017C2047@mailserver.domain.root.inc 0 0 172563 1 2004-8-16 0:0:0 GMT 0 - - FW: Message from 7275731605 - -

2004-8-16 0:0:0 GMT - - - MAILSERVER - Unity.Messaging@domain.com 1033 17E80BE1935EC143B0334DFE778FCC85017C2047@mailserver.domain.root.inc 0 0 172563 1 2004-8-16 0:0:0 GMT 0 - - FW: Message from 7275731605 Unity.Messaging@domain.com -

2004-8-16 0:0:0 GMT - - - MAILSERVER - Unity.Messaging@domain.com 1036 17E80BE1935EC143B0334DFE778FCC85017C2047@mailserver.domain.root.inc 0 0 172563 1 2004-8-16 0:0:0 GMT 0 - - FW: Message from 7275731605 Unity.Messaging@domain.com -

2004-8-16 0:0:0 GMT - - - MAILSERVER - Unity.Messaging@domain.com 1023 17E80BE1935EC143B0334DFE778FCC85017C2047@mailserver.domain.root.inc 0 0 172563 1 2004-8-16 0:0:0 GMT 0 - - FW: Message from 7275731605 Unity.Messaging@domain.com -

2004-8-16 0:0:0 GMT - - - MAILSERVER - Unity.Messaging@domain.com 1028 17E80BE1935EC143B0334DFE778FCC85017C2047@mailserver.domain.root.inc 0 0 172563 1 2004-8-16 0:0:0 GMT 0 - - FW: Message from 7275731605 Unity.Messaging@domain.com -

2004-8-16 0:0:1 GMT - - - MAILSERVER - /O=INTELLTECH/OU=ITCVA/CN=USERS/CN=UNITY_UNITY 1027 17E80BE1935EC143B0334DFE778FCC85017C2048@mailserver.domain.root.inc 0 0 285203 1 2004-8-16 0:0:1 GMT 0 - c=US;a= ;p=INTELLTECH;l=MAILSERVER-040816000001Z-5753 FW: Message from 8007780532 EX:/O=INTELLTECH/OU=ITCVA/CN=USERS/CN=UNITY_UNITY -

2004-8-16 0:0:1 GMT - - - MAILSERVER - /O=INTELLTECH/OU=ITCVA/CN=USERS/CN=UNITY_UNITY 1019 17E80BE1935EC143B0334DFE778FCC85017C2048@mailserver.domain.root.inc 0 0 285203 1 2004-8-16 0:0:1 GMT 0 - - FW: Message from 8007780532 - -

2004-8-16 0:0:1 GMT - - - MAILSERVER - /O=INTELLTECH/OU=ITCVA/CN=USERS/CN=UNITY_UNITY 1025 17E80BE1935EC143B0334DFE778FCC85017C2048@mailserver.domain.root.inc 0 0 285203 1 2004-8-16 0:0:1 GMT 0 - - FW: Message from 8007780532 - -

2004-8-16 0:0:1 GMT - - - MAILSERVER - /O=INTELLTECH/OU=ITCVA/CN=USERS/CN=UNITY_UNITY 1024 17E80BE1935EC143B0334DFE778FCC85017C2048@mailserver.domain.root.inc 0 0 285203 1 2004-8-16 0:0:1 GMT 0 - - FW: Message from 8007780532 - -

2004-8-16 0:0:1 GMT - - - MAILSERVER - Unity.Messaging@domain.com 1033 17E80BE1935EC143B0334DFE778FCC85017C2048@mailserver.domain.root.inc 0 0 285203 1 2004-8-16 0:0:1 GMT 0 - - FW: Message from 8007780532 Unity.Messaging@domain.com -

eschulz · ‎08-19-2004

I have seen a case similar to this before. In that case we were able to send new messages (email and voice) to all mailboxes. However, forwarded messages with a voicemail attachment were not getting through. This failure occurred whether the message was forwarded by Unity (as it does with NDRs from outside callers) or forwarded manually using a client like Outlook Express. Unfortunately, we never found the root cause of this delivery failure. A number of changes were made in the Exchange environment (including a migration off of Ex5.5).

It looks like you may be hitting a similar issue. the "FW: Message from 8007780532" indicates that the failing message is probably an NDR being forwarded from the Unity_(server) mailbox. While I can't help with the root cause, I can maybe offer a workaround that ought to stop the looping.

By adding the following DWORD reg key and setting it to 1, you will stop Unity from automatically forwarding NDR messages from the Unity_(server) mailbox. You’ll need to restart Unity for the change to set in. The downside is now NDR messages from outside callers will collect in the Unity_(server) mailbox. You MUST monitor this mailbox and manually keep it clean. If that mailbox fills up then nobody will get their voicemail.

HKLM\SOFTWARE\Active Voice\Notifier\DisableSystemMailboxCleanout

While the root cause probably lies out in Exchange land, Unity really should be a bit more considerate and not keep trying to hammer out the same message to the point that it might bring down your message store. I’ll look into getting this enhanced by submitting a bug.

Regards,

Eric

aaronkent · ‎08-19-2004

Eric,

FYI... (May make a difference between the reg parameters)

We are running Unity 4.0(3) sr1 with no ES patches.

I looked in the registry and did not see DisableSystemMailboxCleanout under Notifier.

One thing to add is that we changed HKLM\SOFTWARE\Active Voice\ Conversations\1.0\

DWORD: Exclude Return Receipts

from 0 (false) to 1 (true).

After the reboot we still had this endless message flow still. We believe that may keep similar issues from happening in the future, but as of now the existing messages are stuck in this never ending cycle.

Blurb from UnityAdvancedSettingsTool

- This change does not affect whether Cisco Unity includes receipts when subscribers check messages by using the Cisco Unity Inbox. To exclude receipts from the Cisco Unity Inbox, use the "Unity Inbox - Exclude Return Receipts From Inbox" setting.

eschulz · ‎08-19-2004

The registry key I mentioned above is not present by default. It would need to be created.

I beleive the "Exclude Return Receipts" setting only controls whether or not NDRs in a subscriber mailbox are present when checking messages via the TUI. It does not impact delivery of NDRs.

aaronkent · ‎08-19-2004

I must have overlooked the "ADD" part of your post.

We will add the key in tomorrow night and see if it helps. Thanks for the feedback.

jdiegmueller · ‎10-28-2004

Eric--

Just to follow-up, I ran in to this recently as well; was able to trace it down to this same issue (looping NDR between Example Administrator and Unity Messaging System - SERVERNAME).

The only reason I mention it is that you comment "I’ll look into getting this enhanced by submitting a bug", so I thought I'd chime in letting you know someone else in the field has run in to this.

Thanks.

-jd

eschulz · ‎10-28-2004

CSCef63450

If you've got a TAC case you should have it attached to the bug. That'll help ensure it gets the proper dev priority.

jdiegmueller · ‎10-28-2004

I've requested the CSE handling the service request to do so.

Thanks.

-jd

aaronkent · ‎10-28-2004

You can look at the notes from haschulz. We determined that there were some minor issues that caused this to happen.

1. Not watching the limits on mailboxes.

2. Not cleaning out Unity accounts before deleting mailbox .

3. Needed to add myself to the NDR distro list and pull EADMIN off.

4. Added some registry keys.

(FIX ACTION)

Stop Unity and add the Unity_(SERVERNAME) mailbox to a mailbox admin that way you can delete the message and break the cycle.

If you want I can pull the case notes for you to look at on this posting.