Some device trunk or gateway is causing all MTP resources to lockup
::Problem Description:
Calls are not working
Error Message: Fast busy tone
CUCM Full Version: 9.1.2.11900-12
IOS Version: 15.1(4)M1
In the CUCM Traces I could see "Error: Endpoint failed to open receive channel"
Call got dropped with the following error message: [Error: Endpoint failed to open receive channel]
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>
:: Truncated CUCM output ::
61295853.019 |14:09:14.595 |AppInfo |MediaTerminationPointControl(13)::logResourceStatusinTrace -- Device Name=MTP-GW03 ResourceAvailable=110 ResourceUsed=190
61295854.000 |14:09:14.595 |SdlSig-O |AllocateMtpResourceRes |NA RemoteSignal |MediaResourceCdpc(2,100,132,58845) |MediaTerminationPointControl(9,100,130,13) |3,100,13,1.55459^*^* |[R:N-H:0,N:0,L:0,V:0,Z:0,D:0] CI=45851241 ConvId=151250403 Caps[5] = cap=258(0),cap=12(60),cap=257(30),cap=259(30),cap=261(30) Pid=100,9,130,13 Type=0 Region=SYS-REG Device Capability= [0x139 DETECT_2833 PT_2833 PT_CAP PORT_CAP MM_CAP] Count=1
61295855.000 |14:09:14.596 |SdlSig-I |MXAgenaOpenLogicalChannel |waiting |MediaTerminationPointControl(9,100,130,13) |H225Cdpc(2,100,189,87581) |3,100,13,1.55459^*^* |[R:N-H:0,N:0,L:0,V:0,Z:0,D:0] codec=12 CI=0 mediaType=0 confId=151250403 streamFlag=F partyId=33701630 PT=F PTId=16777216 dyn payloadTypeNum=0 MediaEncrAlgo=0 ConnType=0 ReqIpAddrType=0 ipAddrType=0 ipv4=0.0.0.0:172639067 v150MER=F T38MER=F
61295855.001 |14:09:14.596 |AppInfo |MediaTerminationPointControl(13)::star_MediaExchangeAgenaOpenLogicalChannel - PartyId = 33701630
61295855.002 |14:09:14.596 |AppInfo |MediaTerminationPointControl(13)::fillAudioPartyIDtoMediaPidTable - PartyId = 33701630, OLCRecd=1, StartTlkRecd=0
61295856.000 |14:09:14.601 |SdlSig |StationOutputOpenReceiveChannel |waiting |MediaTerminationPointControl(9,100,130,13) |MediaTerminationPointControl(9,100,130,13) |3,100,13,1.55459^*^* |[R:N-H:0,N:0,L:0,V:0,Z:0,D:0] ConfId=151250403 CI=0 PacketSize=20 PayloadType=12 EchoCancellation=0 G723BitRate=0 PartyId=0x2023efe SPId=0x1000000 RFC2833PayloadType=0 mixingMode=0 partyDir=0 srcIP=.type=0 .addr=0x{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}(0.0.0.0) Port=172639067 requestedIpAddrType=0
61295856.001 |14:09:14.601 |AppInfo |MediaTerminationPointControl(13)::star_StationOutputOpenReceiveChannel - TCPPid = [9.100.13.10692] myIP: 0x2a03010a (10.1.3.42) ConferenceID: 151250403, MediaPartyId: 33701630, msecPacketSize: 20 compressionType: 12
61295857.000 |14:09:14.601 |SdlSig |SdlDataInd |wait |StationInit(9,100,57,1) |SdlTCPConnection(9,100,13,10692) |9,100,13,10692.88709^10.1.3.42^MTP-GW03 |*TraceFlagOverrode
61295857.001 |14:09:14.601 |AppInfo |StationInit: (0000013) OpenReceiveChannelAck Status=2 (Error: Endpoint failed to open receive channel), IpAddr=IpAddr.type:0 ipAddr:0x0000000000000000c6243d6650100f86(0.0.0.0), Port=0, PartyID=33701630
61295858.000 |14:09:14.601 |SdlSig |StationOpenReceiveChannelAck |waiting |MediaTerminationPointControl(9,100,130,13) |StationInit(9,100,57,1) |9,100,13,10692.88709^10.1.3.42^MTP-GW03 |[R:N-H:0,N:0,L:0,V:0,Z:0,D:0] ORCStatus = 2 ipAddr=.type=0 .addr=0x{0,0,0,0,0,0,0,0,c6,24,3d,66,50,10,f,86}(0.0.0.0) Port=0 CI=0 PartyId=0x2023efe
61295858.001 |14:09:14.601 |AppInfo |MediaTerminationPointControl(13)::incMTPResourceCannotOpenPortCounter Count=1
61295859.000 |14:09:14.601 |SdlSig-O |MXAgenaOpenLogicalChannelAck |NA RemoteSignal |H225Cdpc(2,100,189,87581) |MediaTerminationPointControl(9,100,130,13) |9,100,13,10692.88709^10.1.3.42^MTP-GW03 |[R:N-H:0,N:0,L:0,V:0,Z:0,D:0] rc=2 isMultiMedia=T LCN=0 partyId=33701630 port=0 ipAddrType=0 ipv4=0.0.0.0
61295860.000 |14:09:14.605 |SdlSig-I |MXAgenaCloseLogicalChannel |waiting |MediaTerminationPointControl(9,100,130,13) |H225Cdpc(2,100,189,87581) |9,100,13,10692.88709^10.1.3.42^MTP-GW03 |[R:N-H:0,N:1,L:0,V:0,Z:0,D:0] ConferenceID= 151250403 PartyId= 33701630 streamFlag=F ClearType= 0 PortHandling=0 ci=0 lcn=0 PT=F PTId=0 switchingToFax=F ConnType=0
61295860.001 |14:09:14.605 |AppInfo |MediaTerminationPointControl(13)::star_MediaExchangeAgenaCloseLogicalChannel - PartyId = 33701630
61295860.002 |14:09:14.605 |AppInfo |MediaTerminationPointControl(13)::star_MediaExchangeAgenaCloseLogicalChannel - Sending Close Channel for Party id =33701630
61295860.003 |14:09:14.605 |AppInfo |MediaTerminationPointControl(13)::removeAudioPartyIdtoMediaPidEntry - Close Channel and Stop Talking both received for PartyId = 33701630. Removing it from our tables.
61295860.004 |14:09:14.605 |AppInfo |MediaTerminationPointControl(13)::star_MediaExchangeAgenaCloseLogicalChannel -- Could not delete. PortReq not found in the list
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>
323VGW#sh sccp conn !! Truncated output
sess_id conn_id stype mode codec sport rport ripaddr conn_id_tx
35407173 34038128 mtp recvonly g729a 21034 0 0.0.0.0
35488329 34518530 mtp recvonly g729a 18072 0 0.0.0.0
35499218 34582954 mtp recvonly g729a 28516 0 0.0.0.0
35521700 34713899 mtp recvonly g729a 23768 0 0.0.0.0
35548015 34866058 mtp recvonly g729a 30860 0 0.0.0.0
35549301 34872433 mtp recvonly g729a 18394 0 0.0.0.0
35599089 35170820 mtp recvonly g729a 22136 0 0.0.0.0
... ... ... ... ... ...
... ... ... ... ... ...
... ... ... ... ... ...
151227115 33627471 mtp recvonly g729a 26526 0 0.0.0.0
151228338 33630742 mtp recvonly g729a 18740 0 0.0.0.0
151249182 33691703 mtp recvonly g729a 17376 0 0.0.0.0
151249198 33691758 mtp recvonly g729a 31962 0 0.0.0.0
151249949 33697852 mtp recvonly g729a 29518 0 0.0.0.0
Total number of active session(s) 273, and connection(s) 273
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>
323VGW#show call active voice br !! Truncated output
... ... ... ... ... ...
Telephony call-legs: 1
SIP call-legs: 0
H323 call-legs: 1
Call agent controlled call-legs: 0
SCCP call-legs: 273
Multicast call-legs: 0
Total call-legs: 275
0 : 4024018 14:50:39.531 EDT Mon Jun 23 2014.1 +0 pid:0 Originate connecting
dur 1w0d tx:0/0 rx:0/0
IP 0.0.0.0:0 SRTP: off rtt:0ms pl:0/0ms lost:0/0/0 delay:0/0/0ms g729ar8 TextRelay: off
media inactive detected:n media contrl rcvd:n/a timestamp:n/a
long duration call detected:n long duration call duration:n/a timestamp:n/a
0 : 4368045 12:49:31.931 EDT Wed Jul 16 2014.1 +0 pid:0 Originate connecting
dur 4w5d tx:0/0 rx:3/60
IP 0.0.0.0:0 SRTP: off rtt:0ms pl:0/0ms lost:0/0/0 delay:0/0/0ms g729ar8 TextRelay: off
media inactive detected:n media contrl rcvd:n/a timestamp:n/a
long duration call detected:n long duration call duration:n/a timestamp:n/a
0 : 4415474 16:23:13.371 EDT Fri Jul 18 2014.1 +0 pid:0 Originate connecting
dur 4w3d tx:0/0 rx:2/40
IP 0.0.0.0:0 SRTP: off rtt:0ms pl:0/0ms lost:0/0/0 delay:0/0/0ms g729ar8 TextRelay: off
media inactive detected:n media contrl rcvd:n/a timestamp:n/a
long duration call detected:n long duration call duration:n/a timestamp:n/a
0 : 4510301 09:49:51.631 EDT Fri Jul 25 2014.1 +0 pid:0 Originate connecting
dur 3w4d tx:0/0 rx:0/0
IP 0.0.0.0:0 SRTP: off rtt:0ms pl:0/0ms lost:0/0/0 delay:0/0/0ms g729ar8 TextRelay: off
media inactive detected:n media contrl rcvd:n/a timestamp:n/a
long duration call detected:n long duration call duration:n/a timestamp:n/a
... ... ... ... ... ...
... ... ... ... ... ...
... ... ... ... ... ...
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>
:: Troubleshooting ::
: CUCM Traces :
+
: Following show & debug commands can help :
FYI !! before hitting any command make sure you understand the network/performance impact !!
- show sccp all
- show dspfarm all
- show voice dsp group all
- show sccp connection
- show call active voice brief
- show sccp connections detail
- show sccp statistics
- show sccp ccm group [id]
- show dspfarm profile [id]
- show media resource status
- show dspfarm dsp all
- show rtpspi statistics
- show call history voice brief
- show voip rtp connection
- show sccp connections internal
- show dspfarm dsp active
- debug ip tcp transaction
- debug sccp messages
- debug sccp errors
- debug sccp events
- debug dsp-resource-manager flex dspfarm
- debug dsp-resource-manager flex dspstats
- debug dsp-resource-manager flex error
- debug dspfarm events
- debug dspfarm error
- debug media resource provisioning all
- debug voip xcodemsp !!! transcoder
********************************************************************************************
********************************************************************************************
:: Resolution ::
"Some device trunk or gateway is causing all MTP resources to lockup"
"35407173 34038128 mtp recvonly g729a 21034 0 0.0.0.0"
conf t
no sccp
sccp
or
shut - no shut (on the dspfarm profile) released the mtp.
You may contact me on mogrover@cisco.com for further information on this or contact Cisco TAC
