Secondary Unity server not placing messages in user mailboxes

[RAustin70]

Here is a fun one.  I am running Unity v12.5.1.11900-57, an upgrade from 11.5SU6 a few months ago.  We have calls coming in from 2 CUCM SIP trunks in a route group, circular pattern.  I just noticed after a customer call that the secondary server is taking the calls, recording the message as if nothing is wrong, BUT it never gets to the called party's mailbox.  RTMT even shows the call going through.

Last night I set the secondary to primary, and immediately received MWI and SMTP notifications of the missing test calls I made, and the messages were in my inbox.  What the Heck?  It seems the messages hang in limbo.  Any ideas how I can fix this?

I am running Active/Active, 5,000 user OVA, 100 ports on each server, been running fine for 4 years before the upgrade.

Rob

- If there is a better room to post Unity questions let me know and I can jump to it.

[Scott Leport]

Hi Rob,

In Unity Connection are your ports assigned to the second Unity Connection server configured and enabled to answer calls, perform MWI notification, send MWI requests and allow TRAP connections?

Regards,

Scott

[RAustin70]

Hey Scott, 

     Yes, all 200 ports (100 on each server) have the port behavior set to

Answer Calls

Perform Message Notification

Send MWI Requests

Allow Trap Connections

I re-verified real quick before clicking 'Reply'

Rob

[Scott Leport]

Hi Rob,

I take it all looks ok with your cluster, e.g. replication is good and your primary and secondary are clearly both the primary and secondary?

When you did the upgrade, did you also ensure the same locales / language packs were installed on both and the necessary services were restarted on both, as per this link (see Installing Unity Connection Language Files from Network Location or Remote Server section):

https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/connection/12x/install_upgrade/guide/b_12xcuciumg/b_12xcuciumg_chapter_0100.html

Can't think of anything else off the top of my head to be honest.

[RAustin70]

Yeah, all replication is locked in great, all services are as they should be.  Even sent TAC macro and micro traces of the tested events and they see nothing wrong. (case open since 7 April, no progress)

Yet another symptom is; if a voicemail comes in on the publisher, it cannot be played from the subscribers web interface, and vice versa.  it only plays silence, and if you try to download the message.wav file, it tries to save it as 0.xml.  BUT if you change the browser to point to the other server, it plays just fine.

There has to be some sort of communication issues between these servers that TAC has not picked up on yet.

At this point I may almost want to just kill/deactivate the subscriber, rebuild it, and see if that shakes anything loose.  But holding out for TAC to WebEx tomorrow, dig in and see what they can find.