Solved: No MoH on Calls through FXO Ports?

Matthew Martin · ‎08-23-2016

Hello All,

CallManager: 10.5(2)
Router: ISR4321 (*version info below)

Cisco IOS XE Software, Version 03.13.04.S - Extended Support Release Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(3)S4, RELEASE SOFTWARE (fc3)

The MoH Audio source is setup to play as Unicast. The MRGL for the Device Pool and for the Gateway (*i.e. Gateway = ISR4321), has 4 total MRGs inside it. It contains the ISR4321's hardware CFB and TC resources listed first, then the next 3 MRGs in the list each contain ALL of the CallManager's Resources (*3 total CUCMs, 1 MRG for each which include that CUCM's: ANN, CFB, MOH and MTP). So all of the MoH Servers are listed in the MRGL.

I have read a few other posts and saw one containing this bug: CSCtl21854... But, my IOS version is not listed in the affected releases. And as for the Workarounds for that bug just in case.. I can't enter a "no ip cef" global config command because it says it's an incomplete command, and I didn't try the other one for CUCM Service Paramters because I didn't know what would get affected if I enabled it.

So basically, any internal extension to extension seems to work just fine. But, when I call in from an outside line and it goes through one of the ISR4321's FXO ports and I place the call on hold, all I get is dead air, no MoH...

Any ideas for where I could start looking to try and resolve this issue? The ISR4321s are replacing our Branch location's old 1861s, which did have working MoH, so maybe the issue is in the ISR's config... Not sure though???

Any thoughts or suggestions would be greatly appreciated! Let me know if you need more info, and I will provide it...

Thanks in Advance,
Matt

Manish Gogna · ‎08-23-2016

Hi Matt,

I would recommend trying the cucm service parameter first as it is not known to cause any negative impact. This service parameter determines whether music on hold (MOH) uses duplex (two-way) or simplex (one-way) audio streams. You can configure one of the these values for this service parameter:

True—Set this parameter to True in order to set up the connection for duplex streaming media when MOH is used. Because certain firewalls and NAT devices require duplex audio streams, this option facilitates interoperability with these products.
False—Set this parameter to False in order to set up the connection for simplex streaming media when MOH is used. The default value specifies False.

This does not carry the same performance impact as disabling CEF on the voice gateway. There is minimal signaling difference for this Duplex Streaming Enabled, for MOH.

Manish

View solution in original post

Manish Gogna · ‎08-23-2016

Hi Matt,

I would recommend trying the cucm service parameter first as it is not known to cause any negative impact. This service parameter determines whether music on hold (MOH) uses duplex (two-way) or simplex (one-way) audio streams. You can configure one of the these values for this service parameter:

True—Set this parameter to True in order to set up the connection for duplex streaming media when MOH is used. Because certain firewalls and NAT devices require duplex audio streams, this option facilitates interoperability with these products.
False—Set this parameter to False in order to set up the connection for simplex streaming media when MOH is used. The default value specifies False.

This does not carry the same performance impact as disabling CEF on the voice gateway. There is minimal signaling difference for this Duplex Streaming Enabled, for MOH.

Manish

Matthew Martin · ‎08-24-2016

Hey Manish, thanks for the reply.

Is it ok to change that Service Parameter during regular business hours? Is there any impact on call processing while saving the Service Parameters?

Also, do I need to set this to true on all CUCMs, or does it get applied to all Nodes automatically?

Thanks Again,
Matt

HARIS_HUSSAIN · ‎08-24-2016

Have you uploaded the MOH file on All the CUCM Nodes ??

Secondly you can collect below RTMT Trace and check

a. Call Manager traces.

b. Cisco IP Voice Media Streaming App.

c. Event Viewer - Application and Event Viewer - System logs.

Thanks

Please rate and Mark Correct

Matthew Martin · ‎08-24-2016

Hey, thanks for the reply.

Sorry, forgot to mention that in the OP... Yes, I uploaded the MoH wav file to all CUCM servers. Seems to work just fine internally. But, as soon as a call comes through the FXO port there's just dead air while on Hold.

Ok, I'll collect those RTMT logs as soon as I give that Service Parameter a shot.

Thanks Again,
Matt

Mohammed al Baqari · ‎08-24-2016

* Don't disable CEF. This is dangerous. Not sure why Cisco recommending disable CEF while they know the impact on router performance.

* Before enabling the service parameter try the following:

- From CUCM CLI issue the command [show perf query class "Cisco MOH Device"] while the call on hold. The counter MOHUnicastResourceActive should be at least 1. This will confirm that CUCM allocated MoH resource

- If you pass step one, then confirm the codec between MoH region and PSTN Gateway Region. Make sure the configured codec in enable in IPVMS service parameters

- From your voice gateway share the output of 'show call active voice br' during held call and

Matthew Martin · ‎08-24-2016

Hey Mohammed, thanks for the reply.

Ok, I ran that command while I was on Hold for an inbound call through the FXO port (*dead-air while on hold), then I ran that same command while I was on Hold for an internal call where the Music-on-Hold was working just fine. Running the Command on the main Publisher shows "no values are returned". But, running the command on our 2 Subscribers does show results.

For inbound PSTN call the "Active" value increments to "1" on CUCM-03. And on the internal call to the same phone that rung on the PSTN example, CUCM-02 increments to 1. Not sure why it would use one for an inbound PSTN call and the other CUCM on the ext to ext call example... I made the test calls a few times and ran the commands each time to make sure it wasn't a coincidence, and got the same results each time.

*Inbound FXO Call: Command Output on CUCM-03

admin:show perf query class "Cisco MOH Device"
==>query class :

 - Perf class (Cisco MOH Device) has instances and values:
    MOH_CUCM03  -> MOHHighestActiveResources      = 1
    MOH_CUCM03  -> MOHMulticastResourceActive     = 0
    MOH_CUCM03  -> MOHMulticastResourceAvailable  = 30
    MOH_CUCM03  -> MOHOutOfResources              = 0
    MOH_CUCM03  -> MOHTotalMulticastResources     = 30
    MOH_CUCM03  -> MOHTotalUnicastResources       = 250
    MOH_CUCM03  -> MOHUnicastResourceActive       = 1
    MOH_CUCM03  -> MOHUnicastResourceAvailable    = 249

*Internal Extension to Extension Call: Command Output on CUCM-02

admin:show perf query class "Cisco MOH Device"
==>query class :

 - Perf class (Cisco MOH Device) has instances and values:
    MOH_CUCM01  -> MOHHighestActiveResources      = 0
    MOH_CUCM01  -> MOHMulticastResourceActive     = 0
    MOH_CUCM01  -> MOHMulticastResourceAvailable  = 30
    MOH_CUCM01  -> MOHOutOfResources              = 0
    MOH_CUCM01  -> MOHTotalMulticastResources     = 30
    MOH_CUCM01  -> MOHTotalUnicastResources       = 250
    MOH_CUCM01  -> MOHUnicastResourceActive       = 0
    MOH_CUCM01  -> MOHUnicastResourceAvailable    = 250
    MOH_CUCM02  -> MOHHighestActiveResources      = 6
    MOH_CUCM02  -> MOHMulticastResourceActive     = 0
    MOH_CUCM02  -> MOHMulticastResourceAvailable  = 30
    MOH_CUCM02  -> MOHOutOfResources              = 0
    MOH_CUCM02  -> MOHTotalMulticastResources     = 30
    MOH_CUCM02  -> MOHTotalUnicastResources       = 250
    MOH_CUCM02  -> MOHUnicastResourceActive       = 1
    MOH_CUCM02  -> MOHUnicastResourceAvailable    = 249

I will add the results of the Voice Gateway's "show voice call active br" commands output in a new comment.

Thanks Again,
Matt

Matthew Martin · ‎08-24-2016

Ok, since I can see that the Active MOH counter was incrementing to 1 on CUCM-03, assuming that is good. But, just to be sure I adjusted the MRGL of the Gateway and moved CUCM-02-MRG up before CUCM-03-MRG in the list. Made another test call through PSTN and ran that command on CUCM-02 and CUCM-03, and this time the Active MOH was incrementing on CUCM-02. But, the result was still dead air...

So I then checked the Region relationships between the MOH_CUCM02 and MOH_CUCM03 and they were both set to 8 kbps. But, double clicking "?" button on the phone during the call showed the codec was g.711... So I modifed the relationship between both of the MOH Server's Regions and my Gateway's Region to 64 kbps. Re-ran the test call, but still the result was dead air while on hold...

Gateway Command Output During test Call: *copy and paste the output below into a notepad to get a better view of this data....

ISR-4321-TAC#show voice call st
CallID     CID  ccVdb      Port        Slot/Bay/DSP:Ch  Called #   Codec    MLPP Dial-peers
0x3A       1248 0x7F58A2F4B008 0/2/5            2/1:1   4019       g711ulaw 911025/4019
1 active call found

ISR-4321-TAC#
ISR-4321-TAC#
ISR-4321-TAC#show call active voice br
<ID>: <CallID> <start>ms.<index> (<start>) +<connect> pid:<peer_id> <dir> <addr> <state> 
  dur hh:mm:ss tx:<packets>/<bytes> rx:<packets>/<bytes> dscp:<packets violation> media:<packets violation> audio tos:<audio tos value> video to>
 IP <ip>:<udp> rtt:<time>ms pl:<play>/<gap>ms lost:<lost>/<early>/<late>
  delay:<last>/<min>/<max>ms <codec> <textrelay> <transcoded
  
 media inactive detected:<y/n> media cntrl rcvd:<y/n> timestamp:<time>
  
 long duration call detected:<y/n> long duration call duration :<sec> timestamp:<time>
 LostPacketRate:<%> OutOfOrderRate:<%>
  MODEMPASS <method> buf:<fills>/<drains> loss <overall%> <multipkt>/<corrected>
   last <buf event time>s dur:<Min>/<Max>s
 FR <protocol> [int dlci cid] vad:<y/n> dtmf:<y/n> seq:<y/n>
  <codec> (payload size)
 ATM <protocol> [int vpi/vci cid] vad:<y/n> dtmf:<y/n> seq:<y/n>
  <codec> (payload size)
 Tele <int> (callID) [channel_id] tx:<tot>/<v>/<fax>ms <codec> noise:<l> acom:<l> i/o:<l>/<l> dBm
  MODEMRELAY info:<rcvd>/<sent>/<resent> xid:<rcvd>/<sent> total:<rcvd>/<sent>/<drops>
         speeds(bps): local <rx>/<tx> remote <rx>/<tx>
 Proxy <ip>:<audio udp>,<video udp>,<tcp0>,<tcp1>,<tcp2>,<tcp3> endpt: <type>/<manf>
 bw: <req>/<act> codec: <audio>/<video>
  tx: <audio pkts>/<audio bytes>,<video pkts>/<video bytes>,<t120 pkts>/<t120 bytes>
 rx: <audio pkts>/<audio bytes>,<video pkts>/<video bytes>,<t120 pkts>/<t120 bytes>
 


Telephony call-legs: 1
SIP call-legs: 0
H323 call-legs: 1
Call agent controlled call-legs: 0
SCCP call-legs: 2
Multicast call-legs: 0
Total call-legs: 4
1248 : 58 15074380ms.1 (*10:01:18.085 PDT Wed Aug 24 2016) +1380 pid:911025 Answer  active
 dur 00:00:06 tx:0/0 rx:0/0 dscp:0 media:0 audio tos:0x0 video tos:0x0
 Tele 0/2/5 (58) [0/2/5] tx:4120/4120/0ms g711ulaw noise:-69 acom:6  i/0:-68/-82 dBm

1248 : 59 15074380ms.2 (*10:01:18.085 PDT Wed Aug 24 2016) +1380 pid:4019 Originate 4019 active
 dur 00:00:06 tx:0/0 rx:0/0 dscp:0 media:0 audio tos:0xB8 video tos:0x0
 IP 0.0.0.0:0 SRTP: off rtt:0ms pl:0/0ms lost:0/0/0 delay:0/0/0ms g711ulaw TextRelay: off Transcoded: No
 media inactive detected:n media contrl rcvd:n/a timestamp:n/a
 long duration call detected:n long duration call duration:n/a timestamp:n/a
 LostPacketRate:0.00 OutOfOrderRate:0.00

0    : 60 15078600ms.1 (*10:01:22.306 PDT Wed Aug 24 2016) +0 pid:0 Originate  connecting
 dur 00:00:04 tx:0/0 rx:0/0 dscp:0 media:0 audio tos:0x0 video tos:0x0
 IP 0.0.0.0:0 SRTP: off rtt:0ms pl:0/0ms lost:0/0/0 delay:0/0/0ms g729ar8 TextRelay: off Transcoded: No
 media inactive detected:n media contrl rcvd:n/a timestamp:n/a
 long duration call detected:n long duration call duration:n/a timestamp:n/a
 LostPacketRate:0.00 OutOfOrderRate:0.00

0    : 62 15078600ms.2 (*10:01:22.306 PDT Wed Aug 24 2016) +0 pid:0 Originate  connecting
 dur 00:00:04 tx:0/0 rx:0/0 dscp:0 media:0 audio tos:0xB8 video tos:0x0
 IP 10.18.2.1:16452 SRTP: off rtt:0ms pl:0/0ms lost:0/0/0 delay:0/0/0ms g711ulaw TextRelay: off Transcoded: No
 media inactive detected:n media contrl rcvd:n/a timestamp:n/a
 long duration call detected:n long duration call duration:n/a timestamp:n/a
 LostPacketRate:0.00 OutOfOrderRate:0.00


Telephony call-legs: 1
SIP call-legs: 0
H323 call-legs: 1
Call agent controlled call-legs: 0
SCCP call-legs: 2
Multicast call-legs: 0
Total call-legs: 4

ISR-4321-TAC#

The Gateway has the following configuration for the Dial-peers used to complete the call. I included the dial-peers and the voice class codec below...

ISR-4321-TAC#show run | sec dial-peer voice 911025
dial-peer voice 911025 pots
 preference 3
 destination-pattern [89][2-9]11
 direct-inward-dial
 port 0/2/5
 forward-digits all
ISR-4321-TAC#
ISR-4321-TAC#show run | sec dial-peer voice 4019  
dial-peer voice 4019 voip
 destination-pattern 4019
 session target ipv4:***CUCM-02 IP ADDRESS***
 voice-class codec 1  
 dtmf-relay h245-alphanumeric
 no vad
ISR-4321-TAC#
ISR-4321-TAC#show run | sec voice class
voice class codec 1
 codec preference 1 g711ulaw
 codec preference 2 g729r8

Thanks Again,
Matt

Mohammed al Baqari · ‎08-24-2016

I believe there is a codec problem. I can see that your MoH server is using G729 and there is a transcoder invoked to convert it to G711ul.

Do the following:

- Navigate to CUCM > System > Service Parameters > Select Pub-Server > Select Service as 'Cisco IP Voice Media Streaming App' > Scroll down to 'Supported MOH codecs' under Cluster Parameters > Select G711u,G711a,G729 (use Ctrl Key and select each codec) > save.

- After saving make sure that all 3 codecs are displayed on the right column of support MOH codecs

- In parallel you need to check the codec between CUCM_MOH_2 region and Trunk Region. Also, between CUCM_MOH_3 region and trunk Region. You don't need to check the region between CUCM_MOH_2 and CUCM_MOH_3. I think you will see 8 kbps used.

Matthew Martin · ‎08-24-2016

Hey Mohammed, thanks again for the reply.

We've already enabled all codecs under Service Paramters for the "Supported MOH Codec" setting a few years ago, but I double checked anyway and they are all still enabled.

Also, The Regions I had mentioned the relationships between were for Gateway ---to---> MOH_CUCM02 and the other relationship was for Gateway ---to---> MOH_CUCM03.

During my tests earlier I changed the relationships for GW --to-- both MOH_CUCM#s from "8 kbps", which was the original setting before I touched anything today while testing, to "64 kbps". But, the test had the same result. So after those tests I changed BOTH relationships back to 8 kbps, which is what they are currently set to right now, and results are still dead air.

Hmmm.... Now that I think about it, I only changed the relationships on the Gateway's Region side of things and never checked the MOH_CUCM's Region side...

I just checked the CUCM02's Region and it's relationship to my Gateway's Region is currently set to 64 kbps, and so is CUCM03's relationship to my Gateway Region. However, earlier when I changed the relationship on my GW's region to 64 kbps to each of the MOH_CUCMs (*like I just mentioned above), wouldn't that have made them match up? Since we are currently open right now and the majority of our IP Phones are in the 2 Regions where the CUCMs reside, I can't really modify the relationship from that direction because I don't want people's phone's to start dropping when I reset the Region(s).

This might be easier to visualize:

CUCM02's Region Relationship Config
MOH_CUCM02 Region --to--> Gateway_Region 64 kbps
CUCM03's Region Relationship Config
MOH_CUCM03 Region --to--> Gateway_Region 64 kbps

Gateway's Region Relationship Config: **Original Settings**
Gateway_Region --to--> MOH_CUCM02 Region 8 kbps
Gateway_Region --to--> MOH_CUCM03 Region 8 kbps

Gateway's Region Relationship Config: **New Settings for Test**
Gateway_Region --to--> MOH_CUCM02 Region 64 kbps
Gateway_Region --to--> MOH_CUCM03 Region 64 kbps

So given that I modified both GW to CUCM relationships to 64 kbps, wouldn't that have nullified any codec issues there might have been since now those bit rates would have matched with the CUCM Region's bit rate settings TO my Gateway? Does that make sense?

Thanks Again for the reply, very much appreciated!

Thanks,
Matt

Matthew Martin · ‎08-24-2016

Ok, so I just set the Duplex Streaming Enabled Service Parameter to true, as TAC got back to me and said to try that first, and that it would NOT have any negative side affects on call processing if I set and saved it during regular business hours, and now its working over PSTN..!!

I even messed with the Region relationships between the GW and the MOH_CUCM's config, setting them to 8 and 64 kbps and the MoH worked just fine under BOTH scenarios.

Also, how can I check if a Transcoder is being used on the ISR while a call is in progress (*i.e. what command)?

Is this not the right command to check that? I ran this command: "show voice dsp active" to see about the TC and I noticed when relationship on GW side is 8 kbps it shows this:

----------------------------FLEX VOICE CARD 0/2 -----------------------
               -------
                        *DSP ACTIVE VOICE CHANNELS*
DSP  DSPWARE         VOX DSP                 SIG DSP          PAK   TX/RX
TYPE VERSION CODEC   NUM CH TS VOICEPORT SLT NUM CH TS RST AI ABRT PACK COUNT
==== ======= ======  === == == ========= ===== === == == === == ==== ============
DM8147 37.3.10 g729r8 001 01 00 0/2/5    0/2 001 06 00   0  0    0    1925/1924

And when I switch the relationship to 64 kbps I get this output below. And since both are producing results in this command I assumed this wasn't a TC resource... Unless its using a TC no matter the relationship I set in CUCM...?

----------------------------FLEX VOICE CARD 0/2 -----------------------
               -------
                        *DSP ACTIVE VOICE CHANNELS*
DSP  DSPWARE         VOX DSP                 SIG DSP          PAK   TX/RX
TYPE VERSION CODEC   NUM CH TS VOICEPORT SLT NUM CH TS RST AI ABRT PACK COUNT
==== ======= ======  === == == ========= ===== === == == === == ==== ============
DM8147 37.3.10 g711ulaw  001 01 00 0/2/5  0/2 001 06 00   0  0    0    1155/1153

I'm just trying to see when a Transcoder is getting used, *i.e. under what circumstances?

Also, that command output shows this for the "edsp" lines:

edsp 0001 01 g729r8 p 0.1 IDLE 50/0/1.1
edsp 0002 01 g729r8 p 0.1 IDLE 50/0/2.1

Does that mean g729r8 is set on the Router as the default codec during SRST?

Thanks Again,
Matt

Manish Gogna · ‎08-24-2016

Hi Matt,

To check if a transcoder is invoked on a call you can use the "show sccp conn" command on the gateway hosting the transcoder. When checking region settings for MoH you also need to check which codecs are specified for MoH under the "Supported MOH Codecs " setting under IP Voice Media Streaming App service parameters. You can access it via System > Service parameters > Pub > IP Voice Media Streaming App.

Manish