Hi Team,
I am trying to fix audio issues by collecting different data metrics that can be used for analysis. Since onset of sudden transition from work from home , I am sure many customers like us will be dealing with sudden transition from office to work from home and looking for ways to improve user and customer experience by reducing audio issues.
While I am at it, I was going through cisco documents and still could not find a proper data / voip data metric that can be used for analysis and I have following questions:
- Is MOS supported for Cisco Jabber? We currently have users on 12.9.x and CUCM versions 10.5.x, 11.5.x and 12.5.x
The data is pretty inconsistent in terms of MOS evaluation and comes as unknown,fair, poor, excellent etc from the same device. My concern is mostly that it is unknown.
The inbuilt cisco CDR tool in all 3 mentioned versions do not always populate VarVQMetrics and other CMR and CDR related details and raw CDR files do not always have CMR data for Cisco Jabber.
What I have found so far:
K-Factor Data
K-factor represents an endpoint mean opinion score (MOS) estimation algorithm that is defined in ITU standard P.VTQ. It represents a general estimator that is used to estimate the mean value of a perceptual evaluation of speech quality (PESQ) population for a specific impairment pattern.
MOS relates to the output of a well designed listening experiment. All MOS experiments use a five-point PESQ scale as defined in ITU standard P.862.1, which describes the PESQ as an objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs.
The MOS estimate provides a number that is inversely proportional to frame loss density. Clarity decreases as more frames are lost or discarded at the receiving end. Consider the loss or discarding of these frames as concealment. Concealment statistics measure packet (frame) loss and its effect on voice quality in an impaired network.
K-factor represents a weighted estimate of average user annoyance due to distortions that are caused by effective packet loss such as dropouts and warbles. It does not estimate the impact of delay-related impairments such as echo. It provides an estimate of listening quality (MOS-LQO) rather than conversational quality (MOS-CQO), and measurements of average user annoyance range from 1 (poor voice quality) to 5 (very good voice quality).
K-factor gets trained or conditioned by speech samples from numerous speech databases, where each training sentence or network condition that is associated with a P.862.1 value has a duration of 8 seconds. For more accurate scores, the system generates k-factor estimates for every 8 seconds of active speech.
Consider K-factor and other MOS estimators to be secondary or derived statistics because they warn a network operator of frame loss only after the problem becomes significant. Packet counts, concealment ratios, and concealment second counters represent primary statistics because they alert the network operator before network impairment has an audible impact or is visible through MOS.
Table 2. Devices That Support K-factor (varVQMetrics) in CMRs
Device
K-factor (varVQMetrics) Support in CMR
Cisco Unified IP Phone 7906
X
Cisco Unified IP Phone 7911
X
Cisco Unified IP Phone 7921
X
Cisco Unified IP Phone7931
X
Cisco Unified IP Phone 7940
S
Cisco Unified IP Phone 7941
X
Cisco Unified IP Phone 7942-G
X
Cisco Unified IP Phone 7942-G/GE
X
Cisco Unified IP Phone 7945
X
Cisco Unified IP Phone7960
S
Cisco Unified IP Phone 7961
X
Cisco Unified IP Phone 7962-G
X
Cisco Unified IP Phone7962-G/GE
X
Cisco Unified IP Phone 7965
X
Cisco Unified IP Phone 7970
X
Cisco Unified IP Phone7971
X
Cisco Unified IP Phone7972-G/GE
X
Cisco Unified IP Phone7975
X
3x MGCP Gateways
G
5x MGCP Gateways
G
Ref link :
https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/service/12_5_1/cdrdef/cucm_b_cdr-admin-guide-1251/cucm_b_cdr-admin-guide-1251_chapter_01001.html
and :
Note that all audio quality metrics listed in Table 1 are stored in the CMR. For phones that do not generate CMR, such as Cisco TelePresence® TC, CE, CTX, and IX endpoints; Jabber® platforms (all platforms); Cisco IP Communicator; and Cisco Unified Personal Communicator, grading will be “N/A”.
The Cisco Network Analysis Module (NAM) reports SCS as well. An audio stream reported from NAM will be graded based on SCSR, the same as an IP Phone.
Ref link :
https://www.cisco.com/c/en/us/products/collateral/cloud-systems-management/prime-collaboration/white-paper-c11-735409.html
The language is same or rather identical in different series (e.g 10.x , 11.x ,12.x, etc)and support for devices has largely remained unchanged (at least in the guide if not real world).
Can someone confirm the following?
- Is K-factor (varVQMetrics) supported for Cisco Jabber (client services framework)? If not, if there is roadmap to add support?
- Is there a partial support for K-factor (varVQMetrics) like concealed seconds, several concealed seconds etc?
- If the above are supported in current iterations of Jabber, any documentation or a doc defect?
- Current list of supported phone models including soft phones. Interestingly widely used cisco 7821 and cisco 7841s are too not listed for k-factor data.
The other questions I have are listed below. They are mostly aimed at Cisco Jabber however other phone model details are appreciated.
- Since MOS is no longer considered a good mechanism, what call grading mechanism can be used for Cisco Jabber specifically and/or other phone models.
- Suggested industry (if any) and cisco suggested Jitter or average jitter per call?
- Acceptable average jitter for voice ?
- Acceptable percentage of packet loss (per call )? If this is codec dependent, then code wise suggested percentage of packet loss.
- Acceptable Concealed seconds
- Acceptable Latency in CUCM CDR terms
- Any other metrics that be can used?
- Any cisco tool that can be used to calculate percentages something similar voice bandwidth calculator?
I have referenced some of the other cisco docs and third party, it is getting complicated to have to do this manually:
Voice Over IP - Per Call Bandwidth Consumption
https://www.cisco.com/c/en/us/support/docs/voice/voice-quality/7934-bwidth-consume.html
Understanding Codecs: Complexity, Hardware Support, MOS, and Negotiation
https://www.cisco.com/c/en/us/support/docs/voice/h323/14069-codec-complexity.html
https://www.bandcalc.com/
http://www.voiptroubleshooter.com/diagnosis/emodel.html
Please note: to fetch all this data, I am also using a third party tool that allows me to capture some of this data in excel and csv formats that is parsed from RAW CDR files fetched from CUCM.
Regards
Bhuvan
