AlarmName = RouteListExhausted, Reason=41,No more Routes in RouteListN

[DavidImmanuelSamDaniel8838]

Hello all, 

I've 2 CUBEs in production that are fine, however recently installed 2 new CUBEs with a new SIP service provider trunk. I have 2 different call manager groups, inbound and outbound works with a 1st call manager group. But if I change to the other CM group on the device pool, outbound calls are not working. Calls are not hitting the CUBE. This issue occurs when phone is registered to one particular CUCM server. I've pulled the RTMT log and found the below error on the SDL log.  

Note: CUCM version is 14.

43085828.001 |14:23:07.238 |AppInfo |RouteListCdrc::null0_CcSetupReq check vipr call mViprReroute=0 mViprAlreadyAttempt=0 CI=123700095 BRANCH=0
43085828.002 |14:23:07.238 |AppInfo |RouteListCdrc::null0_CcSetupReq - Terminating a call after the RouteListCdrc cannot find any more device.
43085828.003 |14:23:07.238 |AppInfo |RouteListCdrc::terminateCall - No more Routes in RouteListName = RL_SIP_PSTN_Gamma. Rejecting the call
43085828.004 |14:23:07.238 |AppInfo |RouteListCdrc::terminateCall - Sending CcRejInd, with the cause code (1459617833), to RouteListControl because all devices are busy/stopped.
43085828.005 |14:23:07.238 |AppInfo |GenAlarm: AlarmName = RouteListExhausted, subFac = CALLMANAGERKeyParam = , severity = 4, AlarmMsg = RouteListName : RL_SIP_PSTN_Gamma, Reason=41, RouteGroups(RG_SIP_PSTN_Gamma)
CallingPartyNumber :
CallingDeviceName : TCTSAMXXXX
CalledPartyNumber : 90745XXXXX48
CalledPartition : a151343f-81a6-f376-cc52-5cd54c12da76
CalledPattern : 9.0[123578]XXXXXXXXX
AppID : Cisco CallManager
ClusterID : ABC-Cluster
NodeID : ABC-UCMSUB2.XYZ.uk

We do have similar discussions on the forum, please refer to the below URL. But that's the trunk between clusters. However, I've tried that solution as well. On the route pattern, removed the route list and mapped the trunk directly. but no luck. 

Also there are some bug IDs for RouteListExhausted error. I've tried that work around too. rebooted the affected server, deleted the trunk, route list, route group and re-created everything. No luck. 

https://community.cisco.com/t5/ip-telephony-and-phones/intercluster-trunk-rejecting-calls-rtmt-routelistexhausted/td-p/2217650/page/2

Has anyone faced this issue before? can you please help me? 

[Maren Mahoney]

I am presuming that other calls egressing via the new CUBEs are successful? (So it's not the CUCM dial plan or CUBE route plan config.)

Do you have "Run on All Active Nodes" checked on the Route List and the SIP Trunk? And, if so, do you have all of the CUCM nodes listed in the Allow list on the second pair of CUBEs? (In particular, the one server where the non-successful phone is registered.)

Maren

[DavidImmanuelSamDaniel8838]

Hi Maren, Yes, other egressing calls are good if I use a different CM group. I am sure it's not a config issue. "Run on All Active Nodes" checked on the Route List and the SIP Trunk. The CUBE's trust list has all the CUCM servers. 

[Maren Mahoney]

I am re-reading your original post and I have another question. What servers are in the 'working' CMG and what servers are in the 'non-working' CMG. Is the server where the 'non-working' phones in either CMG?

Maren

[DavidImmanuelSamDaniel8838]

Hi Maren, thanks for looking into this. 3 sub servers in both the CMG. But the order is different. Ex: in the below order. CMG2 with 10.10.10.201 as the primary has the issue. We have this issue when the phone registered with 10.10.10.201

CMG1:

10.10.10.202

10.10.10.203

10.10.10.201

CMG2: 

10.10.10.201

10.10.10.202

10.10.10.203

[Maren Mahoney]

How interesting! The next thing I'd be looking at would be how the dial-peers are configured in the CUBEs.

• Are you using a 'server-group' to identify your CUCMs, and if so what does that list look like?
• Or are you declaring separate dial-peers invoking via a dial-peer group, and if so is there a specific dial-peer for the .201 server?
• What does this SIP binding look like?

I'm shooting blind, of course, because I don't see your config. It might be worthwhile to read through the following to see if any of it applies to your environment:

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/voice/cube/configuration/cube-book/multiple-server-groups.html#concept_6345721E8F314CAF864157BBEA0159B8 

Maren

[DavidImmanuelSamDaniel8838]

Hi Maren, I am using server-group. Single dial-peer to CUCM with server-group. Please find the config below. The server group has .201. 

voice class server-group 2
ipv4 10.10.10.202 preference 1
ipv4 10.10.10.201 preference 2
ipv4 10.10.10.203 preference 3
description *** Defined CUCM cluster - round robin **
!

dial-peer voice 4001 voip
description Outgoing from CUBE to CUCM
preference 1
destination-pattern 0T
session protocol sipv2
session transport udp
session server-group 2
destination uri-via ITSP
voice-class codec 1
voice-class sip rel1xx supported "100rel"
voice-class sip bind control source-interface GigabitEthernet0/0/1
voice-class sip bind media source-interface GigabitEthernet0/0/1
dtmf-relay sip-kpml rtp-nte
ip qos dscp cs3 signaling
no vad
!

[Maren Mahoney]

To no one's surprise (least of all you) that all looks fine. Hmmmm.....  

Do you have SIP Options PING enabled on the trunk, and if so does the trunk show in full service? Is the Options exchange successful for each of the servers (and especially for .201)?

Can I assume that the Local Route Group for the Device Pool of the phones registered to .201 is the same LRG as other device pools (and is the Route Group that has the new CUBEs in it)?

I'm wondering about the interplay between the CMG in the trunk's DP, and the CMG identified on the Route List itself. If you configure both the Trunk's DP with CMG2 and also set the Route List's CMG to CMG2 does that make a difference? 

I can't think of anything else that would be server-specific influencing outgoing messaging out to a SIP Gateway. There are a bunch of really smart folks on this site that may have more ideas. 

Maren

[Roger Kallberg]

Not related to your question as such, but the description in your server group states round robin, but the configuration you have would not result in round robin usage of the servers defined in the group. If that what you truly want you’d need to change your server group configuration to this.

voice class server-group 2
ipv4 10.10.10.202 preference 1
ipv4 10.10.10.201 preference 1
ipv4 10.10.10.203 preference 1
description *** Defined CUCM cluster - round robin **

Or remove the preference from the server lines all together as that would result in all having preference 0.

On your issue, if you share your complete configuration and screenshots if your SIP trunk and route list configuration in CM we might be able to identify the issue. Also output from debug ccsip message and debug voip ccapi inout for a working and failing call would help to identify your issue. Please collect the asked for information and post it in attached files, one with the configuration and another one with the debug output.

[DavidImmanuelSamDaniel8838 · Accepted solution]

Thanks Roger and Maren for the response. It looks like a network issue, SIP port is not accessible from the affected server. I've engaged the networking team. I will update once the issue is fixed. 

admin:utils network connectivity 192.168.XX.2 5060

Ncat: Version 7.50 ( https://nmap.org/ncat )

Ncat: Connection timed out.

Connect to the port (tcp) failed: Connection refused

Service not accessible

[DavidImmanuelSamDaniel8838]

Thanks @Maren Mahoney  and @Roger Kallberg for your time. This issue has been fixed now. It's a network issue.