Re: SIP 404 Not Found calling internal numbers

JasunKAR · ‎08-23-2019

Having a strange problem on CUCM 12.5 where calls to existing numbers return 404 not found.

History. We had a pilot number that was removed from the system. The number lives in an avaya system outside of Cisco. The call should hit the route pattern and cube then route to Avaya. Instead after removing it we hear a fast busy. Using a packet capture I see a SIP 404 not found message. The call never hits the gateway per debug ccsip messages command.

This morning we had reports of another number returning a fast busy. Same scenario it was once a hunt pilot and it was removed in favor of the number staying in Avaya. Same behavior 404 not found, call never hits cube, and we hear a fast busy.

Dialed number analyzer states it should be going to the gateway per the route list for our SIP trunks. In my packet capture the call manager is the one returning the 404 not found.

What's even more fun is when I put the number back into CUCM as a valid extension I still get the 404 not found. I put the number on my hard phone as a shared line then use jabber to call the shared line number. Dialed number analyzer even validates the call should be going to my hard phone. Calling search spaces and partitions are fine. This is a pretty new install of 12.5 so there isn't much in the way of CSS and PTs to get in the way of troubleshooting.

I have used multiple phones to call these two numbers with the same results. Multiple phones that do not have issues making calls. No route patterns or translation patterns in the way either. I've run several combinations of these numbers through route plan report to see if something is hiding.

Anyone have a hint on a direction to look in?

Gregory Brunn · ‎08-23-2019

What you describing seems like very buggy behavior. Without screen shots of the configs are debugs we really can not confirm for ourselves what is happening.

I would first check DB replication status to ensure your not hitting some odd replication issue, then I would probably open a tac case on this one as like I said this sounds like very odd and buggy behavior.

Hari Prasad · ‎08-23-2019

Hello JasunKar,

Please make test call and pull CUCM traces for 5 min to see what is happening with the call.

You can upload the traces into Translatorx tool and filter the call using calling and called number with call ID to check the issue.

If you face issue in reading traces , then upload the traces file only in which you are getting that call.

Please rate helpful posts
Thanks, Hari Prasad

TONY SMITH · ‎08-23-2019

Have you reset any gateways and trunks involved?

JasunKAR · ‎08-23-2019

Thanks for the replies!

I do agree it sounds like i'm facing a bug. I was hoping someone had been in a similar situation already :(

I do have a TAC case opened. Just provided more logs on second number. I have not reset any SIP gateways as I have felt that isn't my challenge. The numbers are defined internally and the gateways should not be involved in the call. I only mention the gateways because when removing the numbers from call manager (even out of directory number), route patterns are not obeyed to send the calls to the gateway. I don't even see setup messages on the gateways for the call and CUCM continues to return 404 in wireshark just as if the phone was defined in CUCM.

Gregory Brunn · ‎08-23-2019

Tony is correct is calling this out I would schedule a reset of the trunk just to be sure. +5

JasunKAR · ‎08-27-2019

Just to update in case others run into this. Cisco TAC verified CUCM still thinks the number is a defined pilot even though it isn't. Their only work around is rebooting call manager service for now. Am doing that tonight.

You can define a translation pattern to work around this issue to get calls in :) It seems to work fine as a translation pattern but nothing else.

JasunKAR · ‎08-28-2019

This is resolved. We rebooted the cluster per TACs recommendation. Below is part of our logs that pointed to the challenge:

++++++++++++++SINCE EARLIER THIS WAS THE PART OF THE HUNT PILOT SO IT WAS STILL SEARCHING THIS NUMBER IN THE IMBD DATA ++++++++++

70539360.000 |09:32:41.301 |SdlSig |SsInterceptInd |wait |QueueControl(2,100,153,2) |Cdcc(2,100,39,93410) |2,100,251,742740.2208^10.254.67.214^SEPB000B4BB2AF8 |[R:N-H:0,N:2,L:0,V:0,Z:0,D:0] Type=33554472 Key=0InterceptPriority=3761481008 NodeId=2 Party=39421820 DevId=(0,0,0) dn=:tn=0npi=1ti=1nd=13177064126User=13177064126Host=P-CMPSUB1-101.domain.comPort=5060PassWord=Madder=Transport=4mDisplayName=RawUrl=sip:13177064126@P-CMPSUB1-101.domain.com;user=phoneOrigPort=0pi=0si1

NeXT Action Plan:

================

>>Sometimes IMDBB data gets corrupted so to troubleshoot this issue we need to restart the CCM services .<<

>> We agreed To restart the CCM service in the off production hours of all the nodes using the cli command "util system restart" .

Gregory Brunn · ‎08-28-2019

Thanks for posting. Happy TAC was able to resolve this for you and that a restart was needed to fix the odd behavior.

I have seem similar behavior with items that can't be deleted due to dependencies yet the GUI shows no dependencies.

The right SQL statement typically gets the issue flushed out.

JasunKAR · ‎08-28-2019

Yes sir I was feverishly looking for any informix queries to see if I can "see" these numbers still defined. I'm great with informix searching CDR. Not so much on the other stuff. If you have some good learning resources I'm all ears :) Others may be too. I'm all about learning to fish.