11-10-2016 11:19 AM - edited 03-17-2019 08:38 AM
Hi everyone,
Having a strange issue. Last week, we deployed a few 9951 Cisco phones to a remote site in Miami. The phones register to the Call Manager here at HQ over a DMVPN link and all call processing to and from the Miami phones happens locally at HQ. The voice gateways are also located here at HQ. The phones are successfully registered and they can place and receive internal and external calls no problem, but whenever they place an outbound call to an external number the audio is half-duplex. The user placing the call can hear everything but the person on the other end can't hear anything. I want to point out that this is an intermittent issue; when they place an outbound call to an external number the audio is half-duplex 90% of the time. But when they receive a call from an outside number the audio is half-duplex only about 10% of the time. I'm having a really hard time tracking the source of this issue.
The following are some of the things we've already troubleshot.
-All SIP and RTP ports are being allowed on the firewall here at HQ. This was also verified on the Comcast modem at the Miami site.
-After running a debug isdn q931 command, we are able to verify that the Miami phones have communication with the voice gateways here at HQ. Also, the voice gateways are running MGCP.
-The phones are fully registered to the Call Manager here at HQ.
Any help or insight on this would be greatly appreciated!
-Meekael
11-11-2016 06:30 PM
Hi,
1. Configure syslog server on the VG, enable the debugs and make a note of time and the calling and called party numbers. Check if the working and the non-working calls always use a specific channel/ channel range.
2. Collect the output of the command show controller T1 and look for any slip errors.
3. What is the clocking source you have configured?
Aseem
11-14-2016 06:42 AM
Hi Aseem,
Thanks for responding!
1. Currently the remote phones are configured to use a voice gateway here at HQ that only has one T1 PRI configured, which is sufficient for the minimal amount of users at the remote site. Comcast is due to swap out the modem in the remote site this morning because the particular model they're using right now has had a history of issues with voip in the past. After the modem has been swapped I will conduct further testing with debugs to see if there are a particular range of channels that are affected.
2. Since there is only one PRI in use on this particular gateway, only one port shows as being up but there are no slip errors.
3. The clocking source we have configured is Line.
11-28-2016 11:12 AM
So one-way audio is almost always, 'something blocking something'; it doesn't always have to be routing though. You'll want to note, the media (RTP) and signaling (SIP) traffic do not take the same paths in the network.
In order for the phone to register, it needs only to communicate with the Communications Manager server. However, in a centralized media deployment model for PSTN access as you describe, the media stream would be negotiated between the phone and the gateway interface that CUCM has advertised in the call setup (signaling).
IP phone to IP phone media streams simply negotiate between the phones, therefore if your IP Phone subnets can communicate with one another, station to station calling should work fine, as you've indicated.
When this issue does happen you're saying the IP phone can hear the PSTN but the PSTN cannot hear the IP phone; but it does not happen 10% of the time. Are all possible dynamic VPN connections between the phone subnet(s) in the Miami office and the network of the gateway interface at HQ being allowed?
Additionally, during one of these 'bad' calls where the PSTN cannot hear the IP phone, does the output of show call active voice on the voice gateway show both TransmitPackets and ReceivePackets for that call?
11-28-2016 01:05 PM
Hi Ryan,
Thanks for your insight!
Based on all of our troubleshooting up until this point, it certainly seems like the RTP packets are being blocked or dropped somewhere along the path. The strange thing is this: when we first deployed the phones nearly a month ago everything worked fine with outbound PSTN calls. Few days went by and the problem surfaced, but like I mentioned, happened about 90% of the time. Now, three weeks in, the one-way audio occurs every time one of the Miami users makes an outbound call to the PSTN. This issue does not occur when they receive PSTN calls on their phones.
To your point about media and signaling taking different paths in the network, how would I go about tracing the path the RTP packets are taking? This would help us isolate the devices we're running packet captures on and save us a ton of time in the near future.
In reference to your question about all possible dynamic VPN connections between the phone subnet in the Miami office and the network of the gateway interface at HQ, we have verified that there is a rule on the firewall allowing ANY traffic from the data and voice vlan subnets in Miami. However, I just recently learned that there is no rule for traffic coming from the subnet that the actual DMVPN tunnel interfaces reside on, but my network lead does not seem to think this matters. What are your thoughts?
Finally, I will perform more test calls this evening after hours to provide the output of the show call active voice command you mentioned. I believe the output I received from the show rtp statistics command will displays something similar in regards to the transmit and receive packets. See the attached image; these statistics are from an active call where the one-way audio was occurring. As you can see, the RTP packets are being transmitted from the gateway but are not being received in return.
11-28-2016 01:49 PM
there are a few things you can do,
you can use Wireshark and span ports and cap[ture for RTP, following an upstream or down stream direction, or alternatively set up permit rtp ACL in a downstream or upstream direction, while logging these ACLs, until you find a point in the path where you stop/start seeing RTP.
11-28-2016 03:51 PM
Hi Dennis,
Thanks for your input on this issue. As I mentioned above, I am currently in the process of performing packet captures on every device along the path. Hopefully we can isolate exactly where the packet drops are taking place.
11-28-2016 08:26 PM
what ever you find, make sure you share it with us if you can, well worth knowing what you find
11-28-2016 02:24 PM
Well, your screenshot confirms any doubt about an RTP path issue; if the gateway isn't receiving audio from the phone, it obviously can't put it on a bearer channel and send it to the carrier (i.e PSTN can't hear but IP Phone can hear).
Regarding the DMVPN interface question; as long as the local networks on both sides of the tunnel (the remote network on the opposite side) are exempted from NAT (which I'd assume they are), then the actual tunnel interface wouldn't be the source address of the packet and therefore shouldn't need to be included in any ACLs.
I would traceroute the gateway interface from the IP phone network, then as Dennis talks about, SPAN each hop interface during a 'bad' call and see where the RTP flow breaks.
Let me know if I can be of any more help.
11-28-2016 03:49 PM
Hi Ryan,
Thanks again for your ideas. I am in the middle of performing packet captures on every hop in the traceroute path and will let you know what I find. Thanks for making yourself available.
This has turned out to be a real pain
11-16-2016 10:35 AM
Hi Aseem,
Below are the logs I've collected from the voice gateway during one of the call sessions. I conducted two tests, both are outbound calls from one of the effected phones to my cell phone:
Test Call 1
002121: Nov 16 11:06:00.439: ISDN Se0/0/0:23 Q931: TX -> SETUP pd = 8 callref = 0x6210
Bearer Capability i = 0x8090A2
Standard = CCITT
Transfer Capability = Speech
Transfer Mode = Circuit
Transfer Rate = 64 kbit/s
Channel ID i = 0xA9838E
Exclusive, Channel 14
Calling Party Number i = 0x2181, '2023570953'
Plan:ISDN, Type:National
Called Party Number i = 0xA1, '912408329196'
Plan:ISDN, Type:National
002122: Nov 16 11:06:00.599: ISDN Se0/0/0:23 Q931: RX <- CALL_PROC pd = 8 callref = 0xE210
Channel ID i = 0xA9838E
Exclusive, Channel 14
002123: Nov 16 11:06:03.303: ISDN Se0/0/0:23 Q931: RX <- PROGRESS pd = 8 callref = 0xE210
Progress Ind i = 0x8A81 - Call not end-to-end ISDN, may have in-band info
002124: Nov 16 11:06:09.747: ISDN Se0/0/0:23 Q931: RX <- CONNECT pd = 8 callref = 0xE210
002125: Nov 16 11:06:09.755: ISDN Se0/0/0:23 Q931: TX -> CONNECT_ACK pd = 8 callref = 0x6210
Test Call 2
002272: Nov 16 11:19:26.422: ISDN Se0/0/0:23 Q931: TX -> SETUP pd = 8 callref = 0x6223
Bearer Capability i = 0x8090A2
Standard = CCITT
Transfer Capability = Speech
Transfer Mode = Circuit
Transfer Rate = 64 kbit/s
Channel ID i = 0xA98397
Exclusive, Channel 23
Calling Party Number i = 0x2181, '2023570953'
Plan:ISDN, Type:National
Called Party Number i = 0xA1, '912408329196'
Plan:ISDN, Type:National
002273: Nov 16 11:19:26.514: ISDN Se0/0/0:23 Q931: RX <- CALL_PROC pd = 8 callref = 0xE223
Channel ID i = 0xA98397
Exclusive, Channel 23
002274: Nov 16 11:19:28.914: ISDN Se0/0/0:23 Q931: RX <- PROGRESS pd = 8 callref = 0xE223
Progress Ind i = 0x8A81 - Call not end-to-end ISDN, may have in-band info
002275: Nov 16 11:19:30.686: ISDN Se0/0/2:23 Q931: RX <- DISCONNECT pd = 8 callref = 0x01FA
Cause i = 0x8A90 - Normal call clearing
002276: Nov 16 11:19:30.718: ISDN Se0/0/2:23 Q931: TX -> RELEASE pd = 8 callref = 0x81FA
002277: Nov 16 11:19:30.730: ISDN Se0/0/2:23 Q931: RX <- RELEASE_COMP pd = 8 callref = 0x01FA
002278: Nov 16 11:19:35.043: ISDN Se0/0/2:23 Q931: TX -> CONNECT pd = 8 callref = 0x81FB
002279: Nov 16 11:19:35.071: ISDN Se0/0/2:23 Q931: RX <- CONNECT_ACK pd = 8 callref = 0x01FB
002280: Nov 16 11:19:35.735: ISDN Se0/0/0:23 Q931: RX <- CONNECT pd = 8 callref = 0xE223
002281: Nov 16 11:19:35.739: ISDN Se0/0/0:23 Q931: TX -> CONNECT_ACK pd = 8 callref = 0x6223
002282: Nov 16 11:19:50.275: ISDN Se0/0/0:23 Q931: RX <- STATUS_ENQ pd = 8 callref = 0xE1FD
002283: Nov 16 11:19:50.279: ISDN Se0/0/0:23 Q931: TX -> STATUS pd = 8 callref = 0x61FD
I also did a show controllers T1 during the test calls and this was the output:
Test Call 1
T1 0/0/0 is up.
Applique type is Channelized T1
Cablelength is long 0db
Description: <=36.IPZD.104732.1.DC=>
No alarms detected.
alarm-trigger is not set
Soaking time: 3, Clearance time: 10
AIS State:Clear LOS State:Clear LOF State:Clear
Version info FPGA Rev: 08121917, FPGA Type: PRK4
Framing is ESF, Line Code is B8ZS, Clock Source is Line.
CRC Threshold is 320. Reported from firmware is 320.
Data in current interval (201 seconds elapsed):
0 Line Code Violations, 0 Path Code Violations
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Total Data (last 24 hours)
0 Line Code Violations, 0 Path Code Violations,
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Test Call 2
T1 0/0/0 is up.
Applique type is Channelized T1
Cablelength is long 0db
Description: <=36.IPZD.104732.1.DC=>
No alarms detected.
alarm-trigger is not set
Soaking time: 3, Clearance time: 10
AIS State:Clear LOS State:Clear LOF State:Clear
Version info FPGA Rev: 08121917, FPGA Type: PRK4
Framing is ESF, Line Code is B8ZS, Clock Source is Line.
CRC Threshold is 320. Reported from firmware is 320.
Data in current interval (97 seconds elapsed):
0 Line Code Violations, 0 Path Code Violations
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Total Data (last 24 hours)
0 Line Code Violations, 0 Path Code Violations,
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
Please let me know what you think. Again, the half-duplex issues only comes up when the users place an outbound call to external numbers.
11-28-2016 10:33 AM
Does anyone have any recommendations on this one-way audio issue? Based on my research, one-way audio is usually caused by routing issues, but we've been going through our routes here for 3 weeks now and everything seems correct. We can ping and traceroute from the phones to the VGWs and vice versa, and there is an any any rule on our firewall allowing traffic from their particular subnet. I'm able to determine from a show rtp statistics command that RTP packets are being transmitted from the VGW to the remote phone, but are not receiving RTP packets from the phone in the opposite direction.
Can anyone provide any ideas on how we could solve this? We've had this issue for nearly a month now.
04-06-2020 12:14 AM
It seems like some UDP ports are not allowed in the firewall. Check whether full range is allowed (16384 - 32767 / UDP)
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide