Thanks, disabling OPUS codec

msantiagodcny · ‎01-05-2016

We are experiencing intermittent One-way audio (in either direction) on both inbound and outbound off-network calls.

Our Deployement

CUCM: Ver. 8.5

Publisher:

Sub1:

Sub2:

Phones are only registered to either Sub1 or Sub2.

I have verified that the issue can occur regardless of which Sub the phone is registered to.

Two CUBEs - each with its own SIP trunk to our PSTN provider

OCIS-FR-3945-1 (no issue)

E911-FR-3945-1 (issue)

I have narrowed the issue down to calls which are going through our E911-FR-3945-1 CUBE and have had 0 reports of issues going out the other.

Phones:

Issue has been observed and reported against all of our deployed phone types: 7945, 7965, 7975.

I have been able to reproduce the issue about one in every 10-15 calls. I have had some success in tracing these problem calls using Wireshark to observe the SCCP call setup, and also the RTP streams.

To my surprise, on every observed one-way audio call that has been traced and saved, the audio packet streams are observable in both directions.

Furthermore, when a call is verified to have one-way audio during the call, compiling the RTP streams verify that the full two-way audio conversation is taking place on our network as I can hear both sides of it, only the affected end user does not hear it during the time the live call is taking place.

I can provide any necessary information as needed to help resolve this issue.

pkinane · ‎01-05-2016

What is the call flow?

It seems as though it is phone -> CUCM -> SIP trunk to cube -> ITSP

Where did you take the pcap from?

Mohammed al Baqari · ‎01-05-2016

During the problem please get the following:

1. show call active voice brief from the CUBE along with calling/called numbers

2. show sccp connec from the CUBE

3. From the impacted phone, double press on help button '?' and check TX/RX packets are increasing or not.

msantiagodcny · ‎01-06-2016

Just recreated an Trouble call: one way audio.

Calling number: XXXXXX2911 (Us)

Called number: XXXXXX7574 (Them)

Calling Party could NOT hear Called party.

pcap trace shows 3 rtp streams for some reason.

But both sides of the conversation are there even tho no inbound audio could be heard at the ip phone during the call.

1. show call active voice brief from the CUBE along with calling/called numbers

164C : 18547971 -1649868374ms.1 +6530 pid:101 Answer XXXXXX7574 active

dur 00:06:11 tx:18554/2968640 rx:37107/5936980

IP XXX.XX.100.164:13812 SRTP: off rtt:0ms pl:0/0ms lost:0/0/0 delay:0/0/0ms g711ulaw TextRelay: off

media inactive detected:n media contrl rcvd:n/a timestamp:n/a

long duration call detected:n long duration call duration:n/a timestamp:n/a

164C : 18547972 -1649868374ms.2 +6520 pid:200 Originate XXXXXX2911 active

dur 00:06:11 tx:37107/5936980 rx:18554/2968640

IP XXX.XX.209.166:16910 SRTP: off rtt:0ms pl:0/0ms lost:0/0/0 delay:0/0/0ms g711ulaw TextRelay: off

media inactive detected:n media contrl rcvd:n/a timestamp:n/a

long duration call detected:n long duration call duration:n/a timestamp:n/a

2. show sccp connec from the CUBE

E911-FR-3945-1#show sccp connections summary

SCCP Application Service(s) Statistics Summary:

Total Conferencing Sessions: 0, Connections: 0

Total Transcoding Sessions: 0, Connections: 0

Total MTP Sessions: 0, Connections: 0

Total ALG-Phone Sessions: 0, Connections: 0

Total BRI-Phone Sessions: 0, Connections: 0

Total SCCP Sessions: 0, Connections: 0

E911-FR-3945-1#show sccp connections

Total number of active session(s) 0, and connection(s) 0

3. From the impacted phone, double press on help button '?' and check TX/RX packets are increasing or not.

Rx/Tx frames were increasing under Network Statistics

Call Statistics showed fluctuating Rcvr Lost Packets: anywhere from -27360 to 27425

No Jitter

Avg MOS LQK: 2.0

Codec: G.711u

Mohammed al Baqari · ‎01-06-2016

Can you upload the pcap file to look at the call?

msantiagodcny · ‎01-06-2016

phone -> CUCM -> SIP trunk to cube -> ITSP

This is the accurate call flow.

pcaps have been taken from the PC port on the phone.

Biggest surprise is that on trouble calls, the incoming audio packets are audible after trace, when they are not audible during the live call.

pkinane · ‎01-07-2016

How are you taking the pcaps from the phone: SPAN to pc port in the phones configuration or using SPAN in the switch?

Since you are getting the pcap at the phone and you are able to play back both streams from the pcap, but can't hear the audio at the phone, I am curious about how many people are having 1 way audio at this particular site.

If they are using a headset, have you tested what would happen if you use the handset or speaker? I am willing to bet you will have the same situation due to you seeing high numbers for "Rcvr Lost Packets". It is an easy test to get out of the way though.

When you go to stream 1 on the phones web page during a bad call, take note of the remote IP address. Is it the IP address of the CUBE? If it isn't the CUBE, what is it?

Vladislav Vyakin · ‎03-15-2017

Hello. I have same issue at one branch. Sometime once form 10-15 calls user have one way audio.

This issue even occur when call placed to neihbor phone (same subnet, same switch).

Is you resolve your issue?

pkinane · ‎03-16-2017

Can you send the output of show proc cpu hist from the switch the phones use?

Vladislav Vyakin · ‎03-16-2017

c2960X_st2_buz#show proc cpu hist 
      444444444444444444333334444433333444444444444444333334444433
      111222222222200000999991111199999000000000011111999991111188
  100                                                           
   90                                                           
   80                                                           
   70                                                           
   60                                                           
   50                                                           
   40 **********************************************************
   30 **********************************************************
   20 **********************************************************
   10 **********************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)

      444444444444444444444444444444444444444444444444444444444444
      244322322434354242244623242343244311432332332113233232226142
  100                                                           
   90                                                           
   80                                                           
   70                                                           
   60                                                           
   50              *       *                                  * 
   40 ##########################################################
   30 ##########################################################
   20 ##########################################################
   10 ##########################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

      444444644333333333333334444444544343333333333445545555655444444444544445
      665677066788887778778874667578776909898987787382393524070978757969084883
  100                                                                       
   90                                                                       
   80                                                                       
   70                                                                       
   60       *                       *                    *  **              
   50 *********               *********             ********************** *
   40 #########**************##########**************##########*************
   30 ######################################################################
   20 ######################################################################
   10 ######################################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0  
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

c2960X_st2_buz#show proc cpu sorted 5sec | exclude 0.00
CPU utilization for five seconds: 43%/0%; one minute: 41%; five minutes: 40%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process 
 162   175051667    19924109       8785 24.76% 22.75% 22.65%   0 Hulc LED Process 
 228     1334830     5523208        241  1.01%  0.42%  0.38%   0 HULC DAI Process 
 131     6098349      797212       7649  0.59%  0.62%  0.60%   0 hpm counter proc 
 230      516288     5615370         91  0.35%  0.14%  0.12%   0 HRPC ip device t 
  17     1776355     3873380        458  0.35%  0.30%  0.33%   0 ARP Input        
 173     2868023      158825      18057  0.29%  0.30%  0.29%   0 HQM Stack Proces 
 240      268077     3507366         76  0.29%  0.08%  0.06%   0 IP Host Track Pr 
 168         250         144       1736  0.17%  0.24%  0.06%   1 SSH Process      
 174      962601      634186       1517  0.11%  0.11%  0.11%   0 HRPC qos request 
  92      331746    35882605          9  0.11%  0.10%  0.11%   0 RedEarth Rx Mana 
 220      781390     1722132        453  0.11%  0.06%  0.06%   0 Spanning Tree    
  84       85261    26651974          3  0.11%  0.04%  0.04%   0 Draught link sta 
 121      330321    21291819         15  0.11%  0.11%  0.12%   0 HLFM address lea 
 237      111615     7970754         14  0.05%  0.05%  0.05%   0 UDLD             
 207       90420      553169        163  0.05%  0.05%  0.02%   0 IP Input         
 183      994605      265738       3742  0.05%  0.12%  0.12%   0 Power RPS Proces 
  91      491696    35541570         13  0.05%  0.10%  0.11%   0 RedEarth Tx Mana

Two switch in stack 2960X. As I read early high cpu for Hulc LED Process its normal when use PoE powered IP phones.

pkinane · ‎03-16-2017

Enable web access on both phones, then make the issue happen between two phones that are connected to the same switch if possible, when you reproduce the issue, keep the phone call active and browse to the web page for both phones. Once on the web page select stream 1 and document the remote address listed on the web page of phone A and the web page of phone B.

Here is an example of the page I am talking about:

https://supportforums.cisco.com/sites/default/files/legacy/5/5/3/90355-streaming1.jpg

Vladislav Vyakin · ‎03-16-2017

Sorry, but screens with russian locale.

I can translate if something needed.

And I amazing, I never seen before codec OPUS. What is it?

All internal normal call always with g722 and external call always with g711.

Screen 1 translate:

Steam Status - Active

Sender Codec - No

Rcvr Codec - OPUS

Screen 2 translate:

Steam Status - Not Ready - strange but 100% screenshot was make when call is active

Sender Codec - OPUS

Rcvr Codec - OPUS

Vladislav Vyakin · ‎03-16-2017

I get call trace from RTMT

In attacment two call.

I dont khow why but sip messanges is absent. I check this log first time atfer upgrade to 11.5 version. I check sericeability and trace enabled.

I check another failled calls from this branch, they always fail with 102 code.

As seen on trace phone just not sent ACK on on OK from CUCM and after timeout CUCM finish call. This behaviour is same for all phones in this branch.

pkinane · ‎03-17-2017

Vlad,

Here are two links about OPUS:

http://opus-codec.org/

https://en.wikipedia.org/wiki/Opus_(audio_format)

Here is an enhancement request I filed not long ago. If you open a TAC case for this issue, request they attach this enhancement to the case.

https://quickview.cloudapps.cisco.com/quickview/bug/CSCvc39371

Please disable the OPUS codec on your system and test again. This can be done in the Service Parameters. Once you are on the Service Parameter page and you've selected the CallManager service, search for opus and disable it.

Vladislav Vyakin · ‎03-21-2017

Thanks, disabling OPUS codec solve problem.

Intermittent One-Way Audio