No audio or static on incoming SIP calls

[computer]

Call flows is as follows.

PSTN--->Verizon SIP trunk (MPLS Ethernet)--->3750X--->CUBE (2921)--->CUCM (8.0.3 running on 1 pub and 2 subs)---->Remote routers (28xx and 29xx) -->remote switches (3xxx and 2xxx models)--->IP Phone (79xx)

CUBE, CUCM, and 3750X are at a Verizon data center. The remote locations are connected to the data center via Verizon SES/TLS fiber network. We have a real-time EVC and general data EVC provisioned on all of the SES circuits. The routers are configured with QoS settings to prioritize voice traffic.

All locations - Intermittent issues with incoming calls from outside numbers where we cannot hear the caller at all immediately after picking thes handset. Caller will hang up and call again multiple times before we can finally hear them. Less than 10% of calls are having this problem. 

One location on specific phones - We can hear the caller but there is a very noticeable "static" or crackling noise on the call. 

I have worked Verizon on this and they are not reporting any circuit issues both on the MPLS circuit and on the SES circuits. I have an open Cisco TAC case but so far they have not been able to see anything unusual on the CUCM traces I've sent them or on the CUBE debugs I am collecting on a syslog server. They have asked me to set up a Wireshark trace and span the switchport the CUBE is attached to. The problem is we have data and voice VLANs at the data center and all of our servers are on the data VLAN and the CUBE is on the voice VLAN so I don't know how we can do this.

The only real lead I have is that Verizon said they aren't seeing any packets hitting the real-time EVC's. They stated that the voice packets will get dropped if they don't go into that EVC. I'm thinking maybe we aren't tagging the voice packets appropriately? Has anyone else seen this kind of problem?

Matt

[Dennis Mink]

Re. wireshark, if you run a SPAN port you will capture everything disregarding what VLAN the traffic is in.  I would use some sort of capture filter to only grab the voice traffic.

Can you set up a test call and run  "show call act voice compact" and "show call active voice brief" to see if you have any TX/RX on your RTP stream. do this both inbound and outbound.

This should at least tell you if your RTP streams hit your CUBE ingress and egress.

[computer]

I don't recall if we ever actually "resolved" this problem. However one thing I didn't mention before is that our remote office circuits are using Verizon TLS/SES. This is a layer 2 circuit not Layer 3 like MPLS. We cannot set up end-to-end QoS even though we have EVCs set up with Verizon to prioritize RT (real-time) traffic. It's very likely we are dropping some voice packets.

[wenqianyu]

Wonder if you have this issue fixed.

We have the same problem. Intermittently incoming calls get strong static noise. We do call recording for all calls and we could hear those calls with strong noise. Most if not all are incoming calls, and noise is one direction, from outside in.

Thanks,

Wenqian

[CaSToR]

This is the exact same issue I am having. We don't record, but we have voicemails that contain the "static". Did anyone ever resolve this issue?