09-22-2014 06:28 AM - last edited on 03-25-2019 08:31 PM by ciscomoderator
We had someone dial 911 and it went out a secondary MGCP gateway which caused the call to get routed to a incorrect PSAP.
I've checked the logs on this gateway and the PRI never went down during the time of the call. I also tested the next day and the call went out the correct gateway multiple times without any changes on my end.
The only thing I can think of is that this particular gateway lost it's MGCP registration during the time of the call and the 911 call got routed to the next available MGCP gateway. However, there is nothing in the logs that shows MGCP was down during the 911 call.
I have CDR logs showing the 911 being routed to the other gateway, but it doesn't give a reason why it did so.
Any thoughts on how I can track this call more in depth to figure out why it went out the secondary route group member / gateway?
Solved! Go to Solution.
09-22-2014 07:20 AM
Did you check distribution algorithm on the route group? Top Down vs. Circular.
09-23-2014 11:42 AM
It hit your 9.@ Route Pattern in the BI_IA_Unrestricted partition:
05745660.001 |03:23:27.031 |AppInfo |Digit Analysis: star_DaReq: daReq.partitionSearchSpace(:0820ca35-41ef-4b13-8d5e-bd76bb146425), filteredPartitionSearchSpaceString(BA_Internal:BA_IA_Speeddials:BA_Translation:BA_IA_Local:BA_IA_Unrestricted:BA_IA_Gateways), partitionSearchSpaceString(BA_Internal:BA_IA_Speeddials:BA_Translation:BA_IA_Local:BA_IA_Unrestricted:BA_IA_Gateways)
05745660.002 |03:23:27.031 |AppInfo |Digit Analysis: star_DaReq: Matching Legacy Numeric, digits=9911
05745660.003 |03:23:27.032 |AppInfo |Digit Analysis: getDaRes data: daRes.ssType=[0] Intercept DAMR.sstype=[0], TPcount=[0], DAMR.NotifyCount=[0], DaRes.NotifyCount=[0]
05745660.004 |03:23:27.032 |AppInfo |DbMobility: getMatchedRemDest starts: cnumber = 911
05745660.005 |03:23:27.032 |AppInfo |DbMobility: getMatchedRemDest: full match case
05745660.006 |03:23:27.032 |AppInfo |DbMobility SelectByDestination: no matching RemDestDynamic record exists for remdest [911]
05745660.007 |03:23:27.032 |AppInfo |DbMobility: can't find remdest 911 in map
05745660.008 |03:23:27.032 |AppInfo |Digit analysis: match(pi="2", fqcn="5159564404", cn="4404",plv="5", pss="BA_Internal:BA_IA_Speeddials:BA_Translation:BA_IA_Local:BA_IA_Unrestricted:BA_IA_Gateways", TodFilteredPss="BA_Internal:BA_IA_Speeddials:BA_Translation:BA_IA_Local:BA_IA_Unrestricted:BA_IA_Gateways", dd="9911",dac="0")
05745660.009 |03:23:27.032 |AppInfo |Digit analysis: analysis results
05745660.010 |03:23:27.032 |AppInfo ||PretransformCallingPartyNumber=4404
|CallingPartyNumber=5159564404
|DialingPartition=BA_IA_Unrestricted
|DialingPattern=9.@
|FullyQualifiedCalledPartyNumber=9911
|DialingPatternRegularExpression=(9)(911)
|DialingWhere=(AREA-CODE == 800) OR (AREA-CODE == 888) OR (AREA-CODE == 866) OR (AREA-CODE == 877) OR (SERVICE == 911) OR (AREA-CODE == 855)
|PatternType=National
|PotentialMatches=ForegoPotentialMatches
|DialingSdlProcessId=(0,0,0)
|PretransformDigitString=9911
|PretransformTagsList=ACCESS-CODE:SERVICE
|PretransformPositionalMatchList=9:911
|CollectedDigits=911
|UnconsumedDigits=
|TagsList=SERVICE
|PositionalMatchList=911
|VoiceMailbox=9911
|VoiceMailCallingSearchSpace=BA_Internal
|VoiceMailPilotNumber=1900
|RouteBlockFlag=RouteThisPattern
|RouteBlockCause=0
|AlertingName=
|UnicodeDisplayName=
|DisplayNameLocale=1
|OverlapSendingFlagEnabled=0
|WithTags=
|WithValues=
|CallingPartyNumberPi=NotSelected
|ConnectedPartyNumberPi=NotSelected
|CallingPartyNamePi=NotSelected
|ConnectedPartyNamePi=NotSelected
|CallManagerDeviceType=NoDeviceType
|PatternPrecedenceLevel=Routine
|CallableEndPointName=[b3a0f041-f16b-45e9-90de-19bdf999dd59]
|PatternNodeId=[2fb5b40c-c243-485a-98cf-490507dd713d]
|AARNeighborhood=[]
|AARDestinationMask=[]
|AARKeepCallHistory=true
|AARVoiceMailEnabled=false
|NetworkLocation=OffNet
|Calling Party Number Type=Cisco Unified CallManager
|Calling Party Numbering Plan=Cisco Unified CallManager
|Called Party Number Type=Cisco Unified CallManager
|Called Party Numbering Plan=Cisco Unified CallManager
|ProvideOutsideDialtone=true
|AllowDeviceOverride=false
|AlternateMatches= Information Not Available
|TranslationPatternDetails= Information Not Available
|ResourcePriorityNamespace=
|PatternRouteClass=RouteClassDefault
Which goes to this Route List:
05745672.001 |03:23:27.032 |AppInfo |RouteListControl::idle_CcSetupReq - RouteList(RL_IA_IL_Local), numberSetup=0 numberMember=13 vmEnabled=0
09-22-2014 07:20 AM
Did you check distribution algorithm on the route group? Top Down vs. Circular.
09-22-2014 07:32 AM
Yes, route group is set to top down.
09-22-2014 07:45 AM
You would need to pull out the CUCM traces for that kind of analysis.
09-22-2014 07:51 AM
I'm assuming I'll probably have to enable some additional trace options in CUCM. Are the traces you are referring to done in RTMT? Also any chance I'll be able to find out what happened a few days back now? Thanks all.
09-22-2014 07:55 AM
For troubleshooting we usually set them to detailed.
How much detail, and whether they'll contain something related to the root cause, will depend on the level to which they were set when the problem happened.
How many days you can trace back in them, depends on your call volume.
09-22-2014 07:59 PM
Ok looks like detailed debugging was already enabled and the records go back to last Friday. Here is what I pulled out of RTMT so far:
The 911 call happened at approximately 3:22am on 9/19. You can see from the trace that it's going to the secondary MGCP gateway in the route group. Strange thing is, it doesn't even try to go to the primary gateway VG0A and just sends the 911 call to VG0B.
2014/09/19 02:38:08.109|CC|RELEASE|22263806|22263807|16
2014/09/19 03:03:55.007|CC|SETUP|22263808|22263809|4453|4440|4440
2014/09/19 03:03:55.012|CC|OFFERED|22263808|22263809|4453|4440|4440|SEP00131ADC4823|SEP00152B8F3FAE
2014/09/19 03:04:25.399|CC|RELEASE|22263808|22263809|16
2014/09/19 03:22:40.987|CC|SETUP|22263811|22263812|4440|4404|4404
2014/09/19 03:22:40.993|CC|OFFERED|22263811|22263812|4440|4404|4404|SEP00152B8F3FAE|SEP000AF4BB77D3
2014/09/19 03:23:21.579|CC|RELEASE|22263811|22263812|16
2014/09/19 03:23:27.032|CC|SETUP|22263814|22263815|5159564404|911|911
2014/09/19 03:23:27.036|CC|OFFERED|22263814|22263815|5159564404|911|911|SEP000AF4BB77D3|S0/SU0/DS1-0@VG0B.xxxx.net
2014/09/19 03:24:30.803|CC|RELEASE|22263814|22263815|16
2014/09/19 03:40:29.172|CC|RELEASE|22263816|22263817|16
2014/09/19 03:40:42.267|CC|RELEASE|22263818|22263819|16
2014/09/19 03:40:59.753|CC|REJECT|22263820|22263821|4432|||1
2014/09/19 03:41:13.940|CC|REJECT|22263823|22263824|4432|9515|9515|1
2014/09/19 03:41:20.507|CC|SETUP|22263825|22263826|5159564440|5159889|5159889
2014/09/19 03:41:20.515|CC|OFFERED|22263825|22263826|5159564440|5159889|5159889|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:41:26.805|CC|RELEASE|22263825|22263826|16
2014/09/19 03:41:33.767|CC|SETUP|22263828|22263829|5159564440|5159889|5159889
2014/09/19 03:41:33.788|CC|OFFERED|22263828|22263829|5159564440|5159889|5159889|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:41:43.749|CC|RELEASE|22263828|22263829|16
2014/09/19 03:43:53.995|CC|RELEASE|22263830|22263831|16
2014/09/19 03:44:07.358|CC|SETUP|22263832|22263833|5159564440|6019515|6019515
2014/09/19 03:44:07.364|CC|OFFERED|22263832|22263833|5159564440|6019515|6019515|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:44:14.550|CC|RELEASE|22263832|22263833|16
2014/09/19 03:44:23.736|CC|RELEASE|22263834|22263835|16
2014/09/19 03:44:29.218|CC|SETUP|22263838|22263839|5159564440|9887020|9887020
2014/09/19 03:44:29.270|CC|OFFERED|22263838|22263839|5159564440|9887020|9887020|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:44:32.158|CC|RELEASE|22263838|22263839|16
2014/09/19 03:44:38.368|CC|RELEASE|22263836|22263837|16
2014/09/19 03:44:39.476|CC|REJECT|22263840|22263841|4440|9|9|1
2014/09/19 03:44:43.146|CC|RELEASE|22263842|22263843|16
2014/09/19 03:45:02.286|CC|SETUP|22263845|22263846|5159564440|6019988|6019988
2014/09/19 03:45:02.303|CC|OFFERED|22263845|22263846|5159564440|6019988|6019988|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:45:06.126|CC|RELEASE|22263847|22263848|16
2014/09/19 03:45:12.108|CC|RELEASE|22263845|22263846|16
2014/09/19 03:45:18.042|CC|SETUP|22263849|22263850|5159564432|01139083526015159889020|01139083526015159889020
2014/09/19 03:45:18.049|CC|OFFERED|22263849|22263850|5159564432|01139083526015159889020|01139083526015159889020|SEP00131A8C37F3|S0/SU0/DS1-0@VG0B.xxxx.net
2014/09/19 03:45:20.163|CC|RELEASE|22263851|22263852|16
2014/09/19 03:45:22.299|CC|RELEASE|22263849|22263850|16
2014/09/19 03:45:28.913|CC|REJECT|22263853|22263854|4440|9|9|1
2014/09/19 03:45:34.672|CC|RELEASE|22263858|22263859|16
2014/09/19 03:45:38.893|CC|RELEASE|22263855|22263856|3
2014/09/19 03:45:56.258|CC|SETUP|22263860|22263861|5159564432|01139083526015159889020|01139083526015159889020
2014/09/19 03:45:56.262|CC|OFFERED|22263860|22263861|5159564432|01139083526015159889020|01139083526015159889020|SEP00131A8C37F3|S0/SU0/DS1-0@VG0B.xxxx.net
2014/09/19 03:45:58.204|CC|RELEASE|22263860|22263861|16
2014/09/19 03:46:13.692|CC|RELEASE|22263862|22263863|16
2014/09/19 03:46:28.364|CC|SETUP|22263864|22263865|4432|9889020|9889020
2014/09/19 03:46:28.372|CC|OFFERED|22263864|22263865|4432|9889020|9889020|SEP00131A8C37F3|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:46:34.861|CC|RELEASE|22263864|22263865|16
2014/09/19 03:46:48.460|CC|RELEASE|22263866|22263867|16
2014/09/19 03:46:53.857|CC|SETUP|22263868|22263869|5159564440|5159889|5159889
2014/09/19 03:46:53.864|CC|OFFERED|22263868|22263869|5159564440|5159889|5159889|SEP00152B8F3FAE|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:47:01.798|CC|RELEASE|22263868|22263869|16
2014/09/19 03:47:02.979|CC|RELEASE|22263870|22263871|3
2014/09/19 03:47:03.417|CC|RELEASE|22263872|22263873|16
2014/09/19 03:47:05.188|CC|RELEASE|22263874|22263875|16
2014/09/19 03:47:17.918|CC|SETUP|22263876|22263877|4406|18775766676|18775766676
2014/09/19 03:47:17.925|CC|OFFERED|22263876|22263877|4406|18775766676|18775766676|SEP000AB7C6B04E|S0/SU0/DS1-0@VG0A.xxxx.net
2014/09/19 03:47:33.761|CC|RELEASE|22263876|22263877|16
2014/09/19 03:48:10.213|CC|RELEASE|22263878|22263879|16
2014/09/19 03:50:40.131|CC|SETUP|22263880|22263881|4439|4404|4404
2014/09/19 03:50:40.135|CC|OFFERED|22263880|22263881|4439|4404|4404|SEP0012D9EDDAB0|SEP000AF4BB77D3
2014/09/19 03:51:27.848|CC|RELEASE|22263880|22263881|16
2014/09/19 03:54:35.794|CC|SETUP|22263883|22263884|4404|4494|4494
2014/09/19 03:54:35.798|CC|OFFERED|22263883|22263884|4404|4494|4494|SEP000AF4BB77D3|VGC127f83095a17
2014/09/19 03:55:12.905|CC|RELEASE|22263883|22263884|16
2014/09/19 03:55:44.958|CC|RELEASE|22263885|22263886|16
2014/09/19 03:57:08.368|CC|SETUP|22263887|22263888|5641|918775766676|918775766676
2014/09/19 03:57:08.375|CC|OFFERED|22263887|22263888|5641|918775766676|918775766676|SEP001A6D3FC473|S0/SU0/DS1-0@VG0C.xxxx.net
2014/09/19 03:57:36.528|CC|RELEASE|22263887|22263888|16
2014/09/19 03:59:56.396|CC|SETUP|22263889|22263890|5637|5639|5639
2014/09/19 03:59:56.401|CC|OFFERED|22263889|22263890|5637|5639|5639|SEP001A6D3FC40B|SEP001A6D3FC38A
2014/09/19 04:03:55.084|CC|RELEASE|22263889|22263890|16
2014/09/19 04:12:00.909|CC|SETUP|22263892|22263893|5155209852|4404|4404
2014/09/19 04:12:00.912|CC|OFFERED|22263892|22263893|5155209852|4404|4404|S0/SU0/DS1-0@VG0A.xxxx.net|SEP000AF4BB77D3
2014/09/19 04:12:37.302|CC|RELEASE|22263892|22263893|16
2014/09/19 04:12:54.447|CC|SETUP|22263895|22263896|4404|4494|4494
2014/09/19 04:12:54.453|CC|OFFERED|22263895|22263896|4404|4494|4494|SEP000AF4BB77D3|VGC127f83095a17
2014/09/19 04:12:56.397|CC|RELEASE|22263895|22263896|16
2014/09/19 04:13:04.427|CC|SETUP|22263897|22263898|5159564404|15155209852|15155209852
2014/09/19 04:13:04.434|CC|OFFERED|22263897|22263898|5159564404|15155209852|15155209852|SEP000AF4BB77D3|S0/SU1/DS1-0@VG0A.xxxx.net
09-23-2014 08:29 AM
That's just the call logs. We need the SDL files.
09-23-2014 10:04 AM
09-23-2014 10:16 AM
So it went out this PRI:
05745680.003 |03:23:27.033 |AppInfo |SMDMSharedData::findLocalDevice - Name=S0/SU0/DS1-0@VG036015.xxxx.net Key=91f798b2-c835-a160-97b7-d6d56d295a13 isActvie=1 Pid=(1,143,38) found
I'm guessing that's the incorrect one? Which GW/PRI should it have gone out? Can you share a screenshot of your route group?
09-23-2014 10:28 AM
Yes, it went out S0/SU0/DS1-0@VG036015.xxxx.net and it should have gone out S0/SU0/DS1-0@VG074010.xxxx.net
attached
09-23-2014 10:39 AM
The traces you provided are from the publisher? Is that the node the PRI shows registered to? If not, we'll need the SDL traces from that node.
09-23-2014 10:49 AM
Yes, and both nodes in question are registered to the pub. Thanks for taking a look.
09-23-2014 11:04 AM
I don't see anything about that PRI being down when it tries to route the call.
I see a successful call out the S0/SU0/DS1-0@VG074010.xxxx.net PRI later at 3:41:20 but that call used the "RL_IA_Local" route list.
The 911 call used the "RL_IA_IL_Local" route list which seems to have S0/SU0/DS1-0@VG036015.xxxx.net listed as the first endpoint. Can you share a screenshot of the route groups configured on the "RL_IA_IL_Local" route list?
09-23-2014 11:33 AM
Looks like you found the problem RG_IA_IL_Local was set to circular which was the first one on that RL's list to go out. Strange thing about this is all my test calls that I made went out RG_IA_Local which was already set to top down and had the correct gateway at the top.
The reason why I thought this was for this reason:
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide