911 calls to Verizon are dropped

[gp1200x]

   I am a network person working with our VOIP technician trying to solve a problem. We have a Cisco VOIP install - one CUCM pub and two subscribers with a SIP at two sites (redundancy and both online simultaneously to split traffic calls to specific sites) with 3945 CUBES at both SIP sites. This has been running for a few years with no issues. Recently all 911, 8911 and 9911 calls (all are supposed to dial 911 out to Verizon) are failing. Failure is with all sites over both SIPs. On the phone if we dial 911 we see 911 and then after a 5-10 second delay we see 11 pop up and a message (sounds like a Verizon message) stating the number is not reachable. We restored the CER-PUB and Sub to an old date (when we knew 911 was working) and it did not resolve the problem. We also took some of the failing ELIN numbers (that were used by the CER when dialing 911) and placed them on a phone and the calls go out successfully to Verizon when we dial outside line numbers. We have several companies look at the trace I attached and they seem to feel the problem is with Verizon rejecting the calls. We spoke to Verizon and they stated that the 11 was being displayed by our system and they would not be blocking those calls. They refused to do a trace with us to verify what they were seeing at there end and told us we should be looking inside at our CUCM again. We have two separate SIPs with the same issue. Anyone able to read the SIP trace and definitely concur it is on our end?  or Verizon?   I am going to suggest that we first reboot the CUCMs and then secondly possibly take the VOIP CUCMs back to an older date to verify but it will be a very disruptive testing time. Any suggestions?   Multiple people have looked into our CUCM and no one has seen anything that they can put their finger on as to the cause.   Thanks

I added another attachment with the debug commands you requested.  The calling number is the ELIN# of 8454753401 to 911 from internal phone 845-486-2465 which I don't think you would see. I have a 911 call right around 10:01:00 AND 10:01:10

I had a few more but cut the size of the trace...I thought it only allowed 256KB uploads - not 256MB...let me know if it is insufficient'

Thanks for your help. The fact that it falls through the first three dial peers may be the main issue because the first two dial peers I believe are for our VARS connection to Verizon that was setup 6 months ago or so. The latter two dial peers may have been wrong from the beginning when the CUBE was configured years ago??   I am looking up the diagrams for the setup that Verizon did when we switches to VARS last year.

I really appreciate your help. 

One more piece of info....I compared the CUBE 3945 at OCIS (one we are using in the debug) to a config from six months ago. The only differences are the ntp statements, the ssh statements and two other oddities......that may not matter.....under   sccm ccm group 10     the order of the associate profile 10 and associate profile 12 were reversed in the older config (but the statement commands are identical) and also the    sccp ccm 172.22.97.3   and sccp ccm 172.22.196.3 indentifier statements were in reversed order however the statements themselves were identical.  There are no other differences between the configs from the one taken today and the one dated 6/2013.  

Failure only started 2 weeks ago.

[Manish Prasad]

Please send across "debug ccsip message" and "debug voice ccapi inout" only.

Thanks

Manish

[Manish Prasad]

First its very hard to read the logs in such format in which you attached. But whatever i can find out i will put it here....

When an incoming request for 911 came from 8454753401 , it matches four outgoing dial-peer respectively.

OutgoingDial-peer=500 - Called Number=911

Outgoing Dial-peer=501 - Called Number=911

Outgoing Dial-peer=102 - Called Number=11

and Outgoing Dial-peer=302 - Called Number=11

Now what happens is the first three dial-peer could not process it because resource unavailable error (Disconnect Cause=47) and when the last  dial-peer process its request it strip 9 from the digit . So when call went to ITSP its look like this ...INVITE sip:11@172.31.101.17:5077 SIP/2.0. This is the reason your Telco is unable to process this request.

Now we need to look at your router configuration then we can further help you finding why digits were stripped. Attach your router config.

And also collect debug messages is proper format in txt. Do this when you collect those two debugs.

conf t:

service sequence-numbers
service timestamps debug datetime localtime msec
logging buffered 10000000 debug
no logging console
no logging monitor
default logging rate-limit
default logging queue-limit

Then. enable debugs

debug ccsip meesages

debug voip ccapi inout

Enable session capture to txt file in terminal program.> (such as Putty)

then do the ff:

terminal length 0
show logging

Thanks

Manish

[gp1200x]

Thanks for all of your help!!!!!!!!....our 911,8911 and 9911 calls are all going through as 911 calls  to the Verizon VARs sites now on both CUBES over their respective SIPs without any failures. Our  failing dial peers of 500 and 501 appear to be working now with the voice-class codec 1 command. 

I believe this problem was initiated over a weekend so Verizon probably made some code or config changes at their VARs sites that kicked off the failure since our CUBES worked this way for 3 years without any 911 issues. 

I also read your other discussion you submitted about no outbound dial-peer with no codecs assigned.   

We also have another issue that started at the same time which is probably related to Verizon too but I will post that separately since it involves DMTF.

THANKS AGAIN!!  

[Manish Prasad]

Keep the issues coming,  we are here to help and we believe in Learning through Helping.

[Manish Prasad · Accepted solution]

Put "voice class codec 1" or "voice class code 2" under dial-peer 500 and 501 and then test.

[gp1200x]

I added voice-class codec 1 to both dial-peers and a 911 call worked!!!  You fixed it.

We looked at the configs from 6/2013 to verify and that statement was not in those dial-peers back them so I don't know why it has been working all this time without any failures. We saw the codec messages in the trace but didn't want to make changes and make it worse. 

I will have our Voice technician read up on it and so will I.  I also notice there are other dial-peers with no codecs set like 102  - Should I set that it there too ?  

I really appreciate your help. You solved a major issue we had for the last two weeks. !!!!

[Manish Prasad]

Yes you can put voice class under dial-peer 102 also. It was look like issue with the resource (codec) but i will confirm you by tomorrow(its midnight in India ) after looking at the logs in detail.

And thanks for the nice rating.