MOH broken

[eschroedorica]

A few months ago, I migrated my CallManager 8.6.2 cluster from physical appliances to virtual ones. Part of this migration also changed the cluster IP addresses. This broke several things like db replication, which I was able to fix, but now I have found out that the cluster music on hold no longer works.

I've verified that the MOH servers are updated and registered, and that the fixed MOH source files are still there, but still I have silence on hold. Are there any things I can try to get my MOH back up and running?

Thanks,

Evan

[William Bell]

Evan,

You said "fixed MOH source files". Can you please clarify. Are you using "fixed audio sources" or "audio source files"?

Are you unicasting streams or doing multicast?

Since you mention that you have multiple "source files", have you verified that the audio source files are present on CUCM nodes you have provisioned for MOH? I think you can check this by SSH'ng to UCM and running the command: file list activelog mohprep/*. Run on the pub and each UCM that is a MOH server. You can also check by connecting to the CCMAdmin interface on each MOH server (you have to go directly to the server URL).

-Bill (http://ucguerrilla.com)

[eschroedorica]

Bill,

I'm using Audio source files, with multicast. When I took your suggestions to check the files, I found that the pub has the audio file, while the sub does not. I'm working on uploading the file to the sub now. I'll update if that works...

Thanks,

Evan

[eschroedorica]

Well... the files are now on all servers in the cluster, but I still have silence on hold.

[William Bell]

As Dave noted, I will assume that other configurations in UCM are the same as they were when this worked. You have restarted services so that takes us to the network. Just to be sure, you can use the following CLI command on the MOH nodes:

admin: show media streams device MOH

The command will run for a few seconds and then kick out a message that states results were written to a file. View the file and you should see your mutlicast streams. Example:

admin:file view activelog /platform/log/mediainfo.txt

(... stuff deleted...)

Actv=Y, QdPkts=63, PktOR=0, DtmfPL=0 DiscTimeSlice= 0 DiscPkts= 0 18:56:51

    Tx Stream: PktCnt=  818, PID=65535, PktSz=20ms, Payld=uLaw, IP=239.1.1.1:16384(16384), MCast=Y, Mute=N, UsrMd=N, Actv=Y, DtmfPL=0, DtmfQ=0 18:56:51

If you see the media stream listed in the output then UCM is streaming to the identified mcast group address. That is a good sign.

Getting back to IP address changes. When you provisioned mcast in your network, did you or your network admin apply any ACLs to restrict which addresses can stream? If you are using a config where you designate RPs or use auto-RP. Check the network nodes provisioned for said function.

-Bill

[Dave Jani]

Hi Evan,

From my experience silence is normally caused by a file that is referenced in the database but the actual media file itself is not available...

Rather than the IP address change, could it be as simple as part of the migration process? Did you do a backup & restore... & was it all nodes or just a restore of the Publisher & freshly installed Subscribers to which the MoH files were not re-uploaded as the Subscriber is what I would expect to be responsible for streaming the MoH?

You can check this via CCMAdmin Web Page or via CLI with the command "file list activelog mohprep", however you cannot download a copy via CCMAdmin (if it exists on the Publisher) so via CLI:

You will see a list of MoH files in various formats (ulaw, alaw, g729, etc.)

Retrieve the file using the command: file get activelog mohprep/

You can use specific filename: e.g. mohprep/XXXXXX.ulaw.wav (Remember to rename the file to remove the “.ulaw”/”.wb” etc otherwise the filename is unlikely to match that on the Publisher!!!)

You can use a mask: e.g. mohprep/*.ulaw.wav

During the file retrieval process you will be asked to specify a SFTP server, SFTP user ID, and location.  Keep in mind that the file will be retrieved using the following convention:

////mohprep/

From here you can edit and/or distribute the files as is required to suit your needs.

[Dave Jani]

Hi Evan,

Sorry, it looks like I doubled up on William's response but I guess that's what happens if you get distracted whilst typing it... lol. Have you reset the MoH servers & possibly may need to restart the Voice Media Streaming App under Servicability if that doesnt work?

[eschroedorica]

I've just reset both MOH servers and the Voice Media Streaming App on both servers... no go.

[Dave Jani]

This worked prior to the migration which implies that it is not related to Codec, Max Hops or Multicast enabled MRG... right? Is this a the central site or a remote site? Considering the IP address change, did you also change the IP address of the local VG and not change the Multicast moh command that routes via the loopback interface?

[eschroedorica]

Ok, so I made some progress. The reason we changed the IPs of the cluster was because we moved them off-site. The multicast address 239.1.1.x is not a routable address over our WAN, so that's why it wasn't working. I apparently don't understand how muticast works.

Disabling multicast now gives me the triple beep hold tone, but our custom hold music is still missing. I've been checking the media resource groups and lists, but I can't seem to find what I need to make it work again. Thoughts?

Also, is there significant benefit to multicast, and should I be making efforts to get it to work?

[William Bell · Accepted solution]

Evan,

Tone on hold (the beeps) typically indicates that the Media Resource Manager (MRM) (sub-process) is unable to allocate a MOH resource. This could happen for a variety of reasons such as codec mismatch, Media Resource Group List configuration issue, CAC (unicast only), etc.  In your case, I am going to take a wild guess and assume that when you disabled multicast MOH that you went to the MOH server and unchecked the "multicast enabled" check box.

If that is the case then I suspect that your Media Resource Group List (MRGL) for the held party has a multicast enabled MRG. Which is now kinda broken. I am not sure if behavior is different between versions but I have seen this exact issue in the past (on a couple of versions, 6x and 7x I think). What you need to do is create a new Media Resource Group that has a MOH server added and does not have the multicast MOH enabled. Put that MRG in the MRGL. Place it in a higher priority position than the multicast enabled MRG.

Now, as to the question on whether there is a significant benefit to using multicast. I say that the answer is the ubiquitous: "it depends". It comes down to server and network capacity. Server capacity is determined by the number half-duplex media streams (that is what MOH is) that a server can handle. A dedicated MOH server (provisioned for one purpose only) can handle more  than a co-resident solution (where you are running another service like call processing). The exact capacity number is based on platform spec (hardware or VM) and on the number of codecs you have enabled. The SRND has a formula for this (look at UC 8 SRND, table 17-8). The general base number is 250 streams for co-res and 500 for dedicated. But you have have to deduct for number of file sources X the number of codecs.

The next factor is network capacity. You really need to understand your environment to properly account for network capacity. You need to determie the number of concurrent MOH streams across a WAN link during core busy hour. You need to know how much BW you have available, how much of that BW is QoS protected, etc. The numbers will tell you what you need to do.

So, figure out server capacity and network capacity. From there, you can answer the MOH unicast vs. multicast question with hard data.

Now, if we are talking about a question of "designer's choice" where we are focused on highly scalable designs that can grow with the network, then I am a proponent for multicast MOH end-to-end. I try to work it into all of my designs if possible. Again, that is a personal preference and one I will back off of pretty quickly if the network is unable to address the pre-requisites. At that point, I am doing the math just like everyone else.

HTH.

-Bill (http://ucguerrilla.com)

[Dave Jani · Accepted solution]

Hi Evan,

Did you disable Multicast MoH in both the MoH server as well as the MRG?

Multicast over a WAN has many factors to consider, from whether it a true WAN (Layer 3) or a LES circuit for example (Layer 2), what routing protocols are in use and what is configured to propogate over the link(s).

OK, so at least you now have Tone-On-Hold which is a start.

Multicast vs Unicast is mainly considerable of server capacity & bandwidth. E.G, if streaming across a WAN/Link, you dont really want 50 phones to be streaming music-on-hold independantly and therefore would choose Multicast in order for each phone to latch onto the same stream.
The main disadvantage of this however, is that the stream starts whenever the first hold is requested and every subsequent request would just pick up the stream wherever it may be playing the file at that specific time.
I use the term disadvantage very loosely as it may be of benefit in a queuing scenario where the user is constantly on & off hold as they dont just keep hearing the same part of the music so comes down to design/personal preference.

Multicast MoH is my preference for scalabe solutions, however each to their own.

However, I would like to consider & understand the design that you have a bit more to be honest. If the CUCM servers have been moved off-site, what is left ON-site?
I would assume that you at lest have the local VG for PSTN connectivity (unless you are using SIP of course). My design preference would be to retain something on site for resilience such as local VG with SRST.
If this is in fact the case then I would suggest retaining Multicast MoH but drop the "Hop Count" and stream it from the local VG so that it does not need to traverse the WAN, saving bandwidth as well as not needing to figure out if/how you can get Multicast traffic routed across.

If you need any pointers in getting Multicast MoH streaming from an IOS device then please let me know.

[eschroedorica]

Thank you both Bill and Dave! I did not realize that you have to disable multicasting everywhere, I thought that if it was not available on the audio file, that the MRG would default to unicast, which is clearly not the case. As soon as I disabled multicast on the MOH server, audio file source, MRG and MRGL, my MOH started working again.

While I would prefer to use multicast in the long run, I don't think there are ever enough people on hold to really create a true requirement for it. Unicast will do for now.

Again, many thanks to both of you.