One Way Audio from Remote Cisco Phones

[buggs2006]

Hi everyone,

Having a strange issue. Last week, we deployed a few 9951 Cisco phones to a remote site in Miami. The phones register to the Call Manager here at HQ over a DMVPN link and all call processing to and from the Miami phones happens locally at HQ. The voice gateways are also located here at HQ. The phones are successfully registered and they can place and receive internal and external calls no problem, but whenever they place an outbound call to an external number the audio is half-duplex. The user placing the call can hear everything but the person on the other end can't hear anything. I want to point out that this is an intermittent issue; when they place an outbound call to an external number the audio is half-duplex 90% of the time. But when they receive a call from an outside number the audio is half-duplex only about 10% of the time. I'm having a really hard time tracking the source of this issue.

The following are some of the things we've already troubleshot.

-All SIP and RTP ports are being allowed on the firewall here at HQ. This was also verified on the Comcast modem at the Miami site.

-After running a debug isdn q931 command, we are able to verify that the Miami phones have communication with the voice gateways here at HQ. Also, the voice gateways are running MGCP.

-The phones are fully registered to the Call Manager here at HQ.

Any help or insight on this would be greatly appreciated!

-Meekael

[Aseem Anand]

Hi,

1. Configure syslog server on the VG, enable the debugs and make a note of time and the calling and called party numbers. Check if the working and the non-working calls always use a specific channel/ channel range.

2. Collect the output of the command show controller T1 and look for any slip errors.

3. What is the clocking source you have configured?

Aseem

[buggs2006]

Hi Aseem,

Thanks for responding!

1. Currently the remote phones are configured to use a voice gateway here at HQ that only has one T1 PRI configured, which is sufficient for the minimal amount of users at the remote site. Comcast is due to swap out the modem in the remote site this morning because the particular model they're using right now has had a history of issues with voip in the past. After the modem has been swapped I will conduct further testing with debugs to see if there are a particular range of channels that are affected.

2. Since there is only one PRI in use on this particular gateway, only one port shows as being up but there are no slip errors.

3. The clocking source we have configured is Line.

[Ryan Huff]

So one-way audio is almost always, 'something blocking something'; it doesn't always have to be routing though. You'll want to note, the media (RTP) and signaling (SIP) traffic do not take the same paths in the network.

In order for the phone to register, it needs only to communicate with the Communications Manager server. However, in a centralized media deployment model for PSTN access as you describe, the media stream would be negotiated between the phone and the gateway interface that CUCM has advertised in the call setup (signaling).

IP phone to IP phone media streams simply negotiate between the phones, therefore if your IP Phone subnets can communicate with one another, station to station calling should work fine, as you've indicated.

When this issue does happen you're saying the IP phone can hear the PSTN but the PSTN cannot hear the IP phone; but it does not happen 10% of the time. Are all possible dynamic VPN connections between the phone subnet(s) in the Miami office and the network of the gateway interface at HQ being allowed?

Additionally, during one of these 'bad' calls where the PSTN cannot hear the IP phone, does the output of show call active voice on the voice gateway show both TransmitPackets and ReceivePackets for that call?

[buggs2006]

Hi Ryan,

Thanks for your insight!

Based on all of our troubleshooting up until this point, it certainly seems like the RTP packets are being blocked or dropped somewhere along the path. The strange thing is this: when we first deployed the phones nearly a month ago everything worked fine with outbound PSTN calls. Few days went by and the problem surfaced, but like I mentioned, happened about 90% of the time. Now, three weeks in, the one-way audio occurs every time one of the Miami users makes an outbound call to the PSTN. This issue does not occur when they receive PSTN calls on their phones. 

To your point about media and signaling taking different paths in the network, how would I go about tracing the path the RTP packets are taking? This would help us isolate the devices we're running packet captures on and save us a ton of time in the near future.

In reference to your question about all possible dynamic VPN connections between the phone subnet in the Miami office and the network of the gateway interface at HQ, we have verified that there is a rule on the firewall allowing ANY traffic from the data and voice vlan subnets in Miami. However, I just recently learned that there is no rule for traffic coming from the subnet that the actual DMVPN tunnel interfaces reside on, but my network lead does not seem to think this matters. What are your thoughts?

Finally, I will perform more test calls this evening after hours to provide the output of the show call active voice command you mentioned. I believe the output I received from the show rtp statistics command will displays something similar in regards to the transmit and receive packets. See the attached image; these statistics are from an active call where the one-way audio was occurring. As you can see, the RTP packets are being transmitted from the gateway but are not being received in return.

[Dennis Mink]

there are a few things you can do, 

you can use Wireshark and span ports and cap[ture for RTP, following an upstream or down stream direction, or alternatively set up permit rtp ACL in a downstream or upstream direction, while logging these ACLs, until you find a point in the path where you stop/start seeing RTP.  

[buggs2006]

Hi Dennis,

Thanks for your input on this issue. As I mentioned above, I am currently in the process of performing packet captures on every device along the path. Hopefully we can isolate exactly where the packet drops are taking place.

[Dennis Mink]

what ever you find, make sure you share it with us if you can, well worth knowing what you find

[Ryan Huff]

Well, your screenshot confirms any doubt about an RTP path issue; if the gateway isn't receiving audio from the phone, it obviously can't put it on a bearer channel and send it to the carrier (i.e PSTN can't hear but IP Phone can hear).

Regarding the DMVPN interface question; as long as the local networks on both sides of the tunnel (the remote network on the opposite side) are exempted from NAT (which I'd assume they are), then the actual tunnel interface wouldn't be the source address of the packet and therefore shouldn't need to be included in any ACLs.

I would traceroute the gateway interface from the IP phone network, then as Dennis talks about, SPAN each hop interface during a 'bad' call and see where the RTP flow breaks.

Let me know if I can be of any more help.

[buggs2006]

Hi Ryan,

Thanks again for your ideas. I am in the middle of performing packet captures on every hop in the traceroute path and will let you know what I find. Thanks for making yourself available.

This has turned out to be a real pain

[buggs2006]

Hi Aseem,

Below are the logs I've collected from the voice gateway during one of the call sessions. I conducted two tests, both are outbound calls from one of the effected phones to my cell phone:

Test Call 1

002121: Nov 16 11:06:00.439: ISDN Se0/0/0:23 Q931: TX -> SETUP pd = 8  callref = 0x6210

        Bearer Capability i = 0x8090A2

                Standard = CCITT

                Transfer Capability = Speech

                Transfer Mode = Circuit

                Transfer Rate = 64 kbit/s

        Channel ID i = 0xA9838E

                Exclusive, Channel 14

        Calling Party Number i = 0x2181, '2023570953'

                Plan:ISDN, Type:National

        Called Party Number i = 0xA1, '912408329196'

                Plan:ISDN, Type:National

002122: Nov 16 11:06:00.599: ISDN Se0/0/0:23 Q931: RX <- CALL_PROC pd = 8  callref = 0xE210

        Channel ID i = 0xA9838E

                Exclusive, Channel 14

002123: Nov 16 11:06:03.303: ISDN Se0/0/0:23 Q931: RX <- PROGRESS pd = 8  callref = 0xE210

        Progress Ind i = 0x8A81 - Call not end-to-end ISDN, may have in-band info

002124: Nov 16 11:06:09.747: ISDN Se0/0/0:23 Q931: RX <- CONNECT pd = 8  callref = 0xE210

002125: Nov 16 11:06:09.755: ISDN Se0/0/0:23 Q931: TX -> CONNECT_ACK pd = 8  callref = 0x6210

Test Call 2

002272: Nov 16 11:19:26.422: ISDN Se0/0/0:23 Q931: TX -> SETUP pd = 8  callref = 0x6223

        Bearer Capability i = 0x8090A2

                Standard = CCITT

                Transfer Capability = Speech

                Transfer Mode = Circuit

                Transfer Rate = 64 kbit/s

        Channel ID i = 0xA98397

                Exclusive, Channel 23

        Calling Party Number i = 0x2181, '2023570953'

                Plan:ISDN, Type:National

        Called Party Number i = 0xA1, '912408329196'

                Plan:ISDN, Type:National

002273: Nov 16 11:19:26.514: ISDN Se0/0/0:23 Q931: RX <- CALL_PROC pd = 8  callref = 0xE223

        Channel ID i = 0xA98397

                Exclusive, Channel 23

002274: Nov 16 11:19:28.914: ISDN Se0/0/0:23 Q931: RX <- PROGRESS pd = 8  callref = 0xE223

        Progress Ind i = 0x8A81 - Call not end-to-end ISDN, may have in-band info

002275: Nov 16 11:19:30.686: ISDN Se0/0/2:23 Q931: RX <- DISCONNECT pd = 8  callref = 0x01FA

        Cause i = 0x8A90 - Normal call clearing

002276: Nov 16 11:19:30.718: ISDN Se0/0/2:23 Q931: TX -> RELEASE pd = 8  callref = 0x81FA

002277: Nov 16 11:19:30.730: ISDN Se0/0/2:23 Q931: RX <- RELEASE_COMP pd = 8  callref = 0x01FA

002278: Nov 16 11:19:35.043: ISDN Se0/0/2:23 Q931: TX -> CONNECT pd = 8  callref = 0x81FB

002279: Nov 16 11:19:35.071: ISDN Se0/0/2:23 Q931: RX <- CONNECT_ACK pd = 8  callref = 0x01FB

002280: Nov 16 11:19:35.735: ISDN Se0/0/0:23 Q931: RX <- CONNECT pd = 8  callref = 0xE223

002281: Nov 16 11:19:35.739: ISDN Se0/0/0:23 Q931: TX -> CONNECT_ACK pd = 8  callref = 0x6223

002282: Nov 16 11:19:50.275: ISDN Se0/0/0:23 Q931: RX <- STATUS_ENQ pd = 8  callref = 0xE1FD

002283: Nov 16 11:19:50.279: ISDN Se0/0/0:23 Q931: TX -> STATUS pd = 8  callref = 0x61FD

I also did a show controllers T1 during the test calls and this was the output:

Test Call 1

T1 0/0/0 is up.

  Applique type is Channelized T1

  Cablelength is long 0db

  Description: <=36.IPZD.104732.1.DC=>

  No alarms detected.

  alarm-trigger is not set

  Soaking time: 3, Clearance time: 10

  AIS State:Clear  LOS State:Clear  LOF State:Clear

  Version info FPGA Rev: 08121917, FPGA Type: PRK4

  Framing is ESF, Line Code is B8ZS, Clock Source is Line.

  CRC Threshold is 320. Reported from firmware  is 320.

  Data in current interval (201 seconds elapsed):

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Total Data (last 24 hours)

     0 Line Code Violations, 0 Path Code Violations,

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

Test Call 2

T1 0/0/0 is up.

  Applique type is Channelized T1

  Cablelength is long 0db

  Description: <=36.IPZD.104732.1.DC=>

  No alarms detected.

  alarm-trigger is not set

  Soaking time: 3, Clearance time: 10

  AIS State:Clear  LOS State:Clear  LOF State:Clear

  Version info FPGA Rev: 08121917, FPGA Type: PRK4

  Framing is ESF, Line Code is B8ZS, Clock Source is Line.

  CRC Threshold is 320. Reported from firmware  is 320.

  Data in current interval (97 seconds elapsed):

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Total Data (last 24 hours)

     0 Line Code Violations, 0 Path Code Violations,

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

Please let me know what you think. Again, the half-duplex issues only comes up when the users place an outbound call to external numbers.

[buggs2006]

Does anyone have any recommendations on this one-way audio issue? Based on my research, one-way audio is usually caused by routing issues, but we've been going through our routes here for 3 weeks now and everything seems correct. We can ping and traceroute from the phones to the VGWs and vice versa, and there is an any any rule on our firewall allowing traffic from their particular subnet. I'm able to determine from a show rtp statistics command that RTP packets are being transmitted from the VGW to the remote phone, but are not receiving RTP packets from the phone in the opposite direction.

Can anyone provide any ideas on how we could solve this? We've had this issue for nearly a month now.

[nilotpal.jyotideka@sadara.com]

It seems like some UDP ports are not allowed in the firewall. Check whether full range is allowed (16384 - 32767 / UDP)