Troubleshooting Garbled Multicast Audio using InformaCast Paging Server

[ROBBY HARRELL]

I have a customer who is a small school county school system.  They have one school out of 8 which is have audio quality issues only with the voip paging through the Cisco IP phones at one location.   The audio during the page is slightly garbled, just enough that the message is usually understood, but sometimes it is not able to be understood.  All other locations that the school system received the Multicast pages crystal clear.   All the other locations use the same Carrier for the Metro Ethernet (Which I work for). 

 The customer has a flat network, with all layer 3 routing between school VoIP and Data Vlans done on the same Cisco 3850 switch at the school systems Admin building (Core Switch).   Voice and Data Vlans on each schools Cisco Switches are trunked through the Carriers Metro Ethernet (Layer 2) to the Core 3850 (Q in Q tagging).   The remote locations use Cisco 3560 Switches with each site having their unique Voice and Data VLans.

We have no issue with normal voip calls at the problem location.  Bandwidth and CIR on each Metro Circuit at each school is 1 gbps, and at the Core Location a 10 gbps interface with a 2 gbps CIR.  Bandwidth usage on the Core Location Metro WAN interface averages around 150 mbps during peak usage hours.   

All locations in the Callmanager are setup to use G711 protocol, with no transcoding.  Troubleshooting steps I have run include verifying WAN throughput using JPERF on PCs at the remote and Core location, updating the InformaCast server software, updating the phone loads to the latest for the 7945 and 6941 ip phones.  I have updated the software on the 3560 switches at the remote location with the issue to the latest 12.x release.   I have contacted Informacast and we were able to capture the audio leaving the paging server is crystal clear  that is destined for the location experiencing the audio issue.

I am not sure where look next for troubleshooting.   I do not know of any issues with multicasting using the Alcatel 7750 or the RAD ETX 204 Metro Ethernet devices the Carrier uses to transport the customer data between sites.   Has anyone had this type of issue before?  I have not encountered this issue in 10 years of dealing with InformaCast paging with our other customers sites.

Any help would be appreciated.

[Jonathan Schulenberg]

Packet captures are probably your friend here. If the audio is intact leaving the multicast group source, is it intact at the near and far WAN edge devices? Capture it one major hop at a time and see where it becomes distorted.
Assuming the Metro E year isn’t doing IGNP snooping, Lucky for you, Wireshark makes it easy to play RTP streams.

[ajay1komban]

Hi,

Did you try to capture and play  multicast stream  using any packet capture software ?

Please see below document to compare your configuration setting 

https://www.cisco.com/c/en/us/support/docs/unified-communications/paging-server/117059-configure-informacast-00.html

[ROBBY HARRELL]

The problem suddenly disappeared.  It was only happening at 2 remote locations near each other.   Some new optical transport equipment was put in the small central office of the Metro Ethernet provider about the same time as the problem stopped.  It was two small schools that each had 500 mbps CIR back to the School system's 2 gbps WAN circuit.   Both schools having the issue used the same small town's central office where the new Optical transport equipment was installed.  Never had issues with bandwidth usage ever getting more that about 10 percent of CIR.  No packet drops until we exceeded the CIR with site to site traffic generation tests.  Same carrier provides the Metro Ethernet for for their other schools with different central offices.   All I can deduce was that it was an issue with the old optical transport from the local central office back to the host site's central office.