MGCP Gateway: ISDN E1 losing clocking, Error Message

CitrusCollab85908 · ‎03-06-2022

Hi all,

Thank you for your time and help, want to ask if you know what this particular ISDN Q.921 debug message means.

Problem Symptom: ISDN E1 circuit frequently drops. We see Line Code Violations, clock slips, all the symptoms with either Layer-2 or Layer-1, but for the most part, don't have enough call traffic to notice the error in a call, but we have had instances where the called party (reception team) say they start hearing garbled voice, before the call could disconnect (no way voice first).

What does this mean:

Mar 7 04:55:48.744: ISDN Se0/1/0:15 **ERROR**: handle_l2d_srq_mail: Message tossed, state = 0 spoofing = 0

So we got advice to use 'debug isdn q921' to find out what's happening:

ar 7 04:54:33.423: ISDN Se0/1/0:15 Q921: User RX <- RRf sapi=0 tei=0 nr=60
Mar 7 04:54:37.613: ISDN Se0/1/1:15 Q921: User RX <- RRp sapi=0 tei=0 nr=45
Mar 7 04:54:37.613: ISDN Se0/1/1:15 Q921: User TX -> RRp sapi=0 tei=0 nr=44
Mar 7 04:54:37.613: ISDN Se0/1/1:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:54:37.624: ISDN Se0/1/1:15 Q921: User RX <- RRf sapi=0 tei=0 nr=45
Mar 7 04:54:42.727: ISDN Se0/1/0:15 Q921: User RX <- INFO sapi=0 tei=0, ns=41 nr=60
Mar 7 04:54:42.727: ISDN Se0/1/0:15 Q921: User TX -> RR sapi=0 tei=0 nr=42
Mar 7 04:54:42.732: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=60 nr=42
Mar 7 04:54:42.736: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=61 nr=42
Mar 7 04:54:42.745: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=61
Mar 7 04:54:42.747: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=62
Mar 7 04:54:42.767: ISDN Se0/1/0:15 Q921: User RX <- INFO sapi=0 tei=0, ns=42 nr=62
Mar 7 04:54:42.768: ISDN Se0/1/0:15 Q921: User TX -> RR sapi=0 tei=0 nr=43
Mar 7 04:54:42.808: ISDN Se0/1/0:15 Q921: User RX <- INFO sapi=0 tei=0, ns=43 nr=62
Mar 7 04:54:42.808: ISDN Se0/1/0:15 Q921: User TX -> RR sapi=0 tei=0 nr=44
Mar 7 04:54:42.812: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=62 nr=44
Mar 7 04:54:42.824: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=63
Mar 7 04:54:47.612: ISDN Se0/1/1:15 Q921: User RX <- RRp sapi=0 tei=0 nr=45
Mar 7 04:54:47.612: ISDN Se0/1/1:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:54:52.812: ISDN Se0/1/0:15 Q921: User RX <- RRp sapi=0 tei=0 nr=63
Mar 7 04:54:52.812: ISDN Se0/1/0:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:54:57.612: ISDN Se0/1/1:15 Q921: User TX -> RRp sapi=0 tei=0 nr=44
Mar 7 04:54:57.612: ISDN Se0/1/1:15 Q921: User RX <- RRp sapi=0 tei=0 nr=45
Mar 7 04:54:57.613: ISDN Se0/1/1:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:54:57.623: ISDN Se0/1/1:15 Q921: User RX <- RRf sapi=0 tei=0 nr=45
Mar 7 04:55:02.812: ISDN Se0/1/0:15 Q921: User RX <- RRp sapi=0 tei=0 nr=63
Mar 7 04:55:02.812: ISDN Se0/1/0:15 Q921: User TX -> RRp sapi=0 tei=0 nr=44
Mar 7 04:55:02.812: ISDN Se0/1/0:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:55:02.823: ISDN Se0/1/0:15 Q921: User RX <- RRf sapi=0 tei=0 nr=63
Mar 7 04:55:07.612: ISDN Se0/1/1:15 Q921: User RX <- RRp sapi=0 tei=0 nr=45
Mar 7 04:55:07.612: ISDN Se0/1/1:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:55:12.812: ISDN Se0/1/0:15 Q921: User RX <- RRp sapi=0 tei=0 nr=63
Mar 7 04:55:12.812: ISDN Se0/1/0:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:55:13.311: ISDN Se0/1/0:15 Q921: User RX <- INFO sapi=0 tei=0, ns=44 nr=63
Mar 7 04:55:13.311: ISDN Se0/1/0:15 Q921: User TX -> RR sapi=0 tei=0 nr=45
Mar 7 04:55:13.324: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=63 nr=45
Mar 7 04:55:13.325: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=64 nr=45
Mar 7 04:55:13.337: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=64
Mar 7 04:55:13.339: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=65
Mar 7 04:55:17.612: ISDN Se0/1/1:15 Q921: User RX <- RRp sapi=0 tei=0 nr=45
Mar 7 04:55:17.613: ISDN Se0/1/1:15 Q921: User TX -> RRp sapi=0 tei=0 nr=44
Mar 7 04:55:17.613: ISDN Se0/1/1:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:55:17.624: ISDN Se0/1/1:15 Q921: User RX <- RRf sapi=0 tei=0 nr=45
Mar 7 04:55:20.509: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=65 nr=45
Mar 7 04:55:20.521: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=66
Mar 7 04:55:20.551: ISDN Se0/1/0:15 Q921: User RX <- INFO sapi=0 tei=0, ns=45 nr=66
Mar 7 04:55:20.551: ISDN Se0/1/0:15 Q921: User TX -> RR sapi=0 tei=0 nr=46
Mar 7 04:55:27.612: ISDN Se0/1/1:15 Q921: User RX <- RRp sapi=0 tei=0 nr=45
Mar 7 04:55:27.612: ISDN Se0/1/1:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:55:30.412: ISDN Se0/1/0:15 Q921: User RX <- RRp sapi=0 tei=0 nr=66
Mar 7 04:55:30.412: ISDN Se0/1/0:15 Q921: User TX -> RRf sapi=0 tei=0 nr=46
Mar 7 04:55:37.612: ISDN Se0/1/1:15 Q921: User TX -> RRp sapi=0 tei=0 nr=44
Mar 7 04:55:37.612: ISDN Se0/1/1:15 Q921: User RX <- RRp sapi=0 tei=0 nr=45
Mar 7 04:55:37.612: ISDN Se0/1/1:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:55:37.623: ISDN Se0/1/1:15 Q921: User RX <- RRf sapi=0 tei=0 nr=45
Mar 7 04:55:38.310: ISDN Se0/1/0:15 Q921: User RX <- INFO sapi=0 tei=0, ns=46 nr=66
Mar 7 04:55:38.311: ISDN Se0/1/0:15 Q921: User TX -> RR sapi=0 tei=0 nr=47
Mar 7 04:55:38.324: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=66 nr=47
Mar 7 04:55:38.328: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=67 nr=47
Mar 7 04:55:38.337: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=67
Mar 7 04:55:38.345: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=68
Mar 7 04:55:38.410: ISDN Se0/1/0:15 Q921: User RX <- INFO sapi=0 tei=0, ns=47 nr=68
Mar 7 04:55:38.411: ISDN Se0/1/0:15 Q921: User TX -> RR sapi=0 tei=0 nr=48
Mar 7 04:55:40.063: ISDN Se0/1/0:15 Q921: User RX <- INFO sapi=0 tei=0, ns=48 nr=68
Mar 7 04:55:40.064: ISDN Se0/1/0:15 Q921: User TX -> RR sapi=0 tei=0 nr=49
Mar 7 04:55:40.067: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=68 nr=49
Mar 7 04:55:40.079: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=69
Mar 7 04:55:41.178: ISDN Se0/1/0:15 Q921: User RX <- INFO sapi=0 tei=0, ns=49 nr=69
Mar 7 04:55:41.179: ISDN Se0/1/0:15 Q921: User TX -> RR sapi=0 tei=0 nr=50
Mar 7 04:55:41.183: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=69 nr=50
Mar 7 04:55:41.183: ISDN Se0/1/0:15 Q921: User TX -> INFO sapi=0 tei=0, ns=70 nr=50
Mar 7 04:55:41.195: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=70
Mar 7 04:55:41.196: ISDN Se0/1/0:15 Q921: User RX <- RR sapi=0 tei=0 nr=71
Mar 7 04:55:42.213: ISDN Se0/1/0:15 Q921: User RX <- INFOp sapi=0 tei=0, ns=50 nr=71
Mar 7 04:55:42.213: ISDN Se0/1/0:15 Q921: User TX -> RRf sapi=0 tei=0 nr=51
Mar 7 04:55:47.612: ISDN Se0/1/1:15 Q921: User RX <- RRp sapi=0 tei=0 nr=45
Mar 7 04:55:47.612: ISDN Se0/1/1:15 Q921: User TX -> RRf sapi=0 tei=0 nr=44
Mar 7 04:55:48.693: %CONTROLLER-5-UPDOWN: Controller E1 0/1/0, changed state to down
Mar 7 04:55:48.744: ISDN Se0/1/0:15 **ERROR**: handle_l2d_srq_mail: Message tossed, state = 0 spoofing = 0
Mar 7 04:55:48.749: %ISDN-6-LAYER2DOWN: Layer 2 for Interface Se0/1/0:15, TEI 0 changed to down
Mar 7 04:55:49.017: %NETCLK-6-SRC_UPD: Synchronization source E1 0/1/0 status (Signal Failure) is posted to all selection process.
Mar 7 04:55:49.018: %NETCLK-6-SEL_CLOCK_SRC: Synchronization source Internal is selected by selection process(T0) to synchronize the system and generate T0 output.
Mar 7 04:55:49.018: %NETCLK-6-ENTER_HOLDOVER: System Internal synchronization source is selected.
Mar 7 04:55:49.689: %CONTROLLER-5-UPDOWN: Controller E1 0/1/0, changed state to up
Mar 7 04:55:49.728: ISDN Se0/1/0:15 Q921: User TX -> SABMEp sapi=0 tei=0
Mar 7 04:55:50.728: ISDN Se0/1/0:15 Q921: User TX -> SABMEp sapi=0 tei=0
Mar 7 04:55:50.739: ISDN Se0/1/0:15 Q921: User RX <- UAf sapi=0 tei=0
Mar 7 04:55:50.739: %ISDN-6-LAYER2UP: Layer 2 for Interface Se0/1/0:15, TEI 0 changed to up

The equipment is ISR4321, with NIM-2CE1T1-PRI.

I have 'show tech' and other bits of information, but don't want to bomb the group.

Roger Kallberg · ‎03-06-2022

Please share your running configuration.

CitrusCollab85908 · ‎03-07-2022

Hello,

Sorry I have to blank out a lot of the customer information.

Router ISR is currently running

service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
!
hostname routerABC
!
boot-start-marker
boot-end-marker
!
!
vrf definition Mgmt-intf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
card type e1 0 1
enable secret 5 blah
!
aaa new-model
!
!
!
!
!
!
!
aaa session-id common
!
!
!
!
!
!
!
!
!

no ip domain lookup
ip domain name abc.xyz.com.au

!
!
!
!
!
!
!
!
!
!
subscriber templating
multilink bundle-name authenticated
!
!

isdn switch-type primary-5ess
!
!
!
!
!
voice service voip
allow-connections h323 to h323
allow-connections h323 to sip
fax protocol t38 version 0 ls-redundancy 0 hs-redundancy 0 fallback none
modem passthrough nse codec g711alaw
!
!
!
!
!
!
!
voice-card 0/1
dsp services dspfarm
no watchdog
!
license udi pid ISR4321/K9 sn xxxxxxxx
spanning-tree extend system-id
!
username blah secret 5 blah
username blah privilege 15 secret 5 blah
!
redundancy
mode none
!
controller E1 0/1/0
pri-group timeslots 1-31 service mgcp
!
controller E1 0/1/1
pri-group timeslots 1-31 service mgcp
!
!
vlan internal allocation policy ascending
!
ip ftp username voipftp
ip ftp password xxxxxxxxxx
ip tftp source-interface GigabitEthernet0
ip ssh time-out 60
ip ssh authentication-retries 5
ip ssh rsa keypair-name mopt_key
ip ssh version 2
!
!
!
!
!
interface GigabitEthernet0/0/0
no ip address
negotiation auto
!
interface GigabitEthernet0/0/0.3
encapsulation dot1Q 3
ip address x.x.x.x y.y.y.y
no cdp enable
!
interface GigabitEthernet0/0/1
no ip address
shutdown
negotiation auto
!
interface Service-Engine0/1/0
!
interface Serial0/1/0:15
no ip address
encapsulation hdlc
isdn switch-type primary-net5
isdn incoming-voice voice
isdn supp-service mcid
isdn bind-l3 ccm-manager
isdn bchan-number-order ascending
no cdp enable
!
interface Serial0/1/1:15
no ip address
encapsulation hdlc
isdn switch-type primary-net5
isdn incoming-voice voice
isdn supp-service mcid
isdn bind-l3 ccm-manager
no cdp enable
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
no ip address
shutdown
negotiation auto
!
interface Vlan1
no ip address
shutdown
!
interface Vlan100
no ip address
!
ip forward-protocol nd
no ip http server
no ip http secure-server
ip route 0.0.0.0 0.0.0.0 x.x.x.x
!
!
!
!
!
!
!
control-plane
!
!
voice-port 0/1/0:15
echo-cancel coverage 64
cptone AU
!
voice-port 0/1/1:15
echo-cancel coverage 64
cptone AU
!
!
!
!
!
mgcp
mgcp call-agent x.x.x.x 2427 service-type mgcp version 0.1
mgcp dtmf-relay voip codec all mode out-of-band
mgcp rtp unreachable timeout 1000 action notify
mgcp modem passthrough voip mode nse
mgcp ip qos dscp cs3 signaling
mgcp package-capability rtp-package
mgcp package-capability sst-package
mgcp package-capability pre-package
no mgcp package-capability fxr-package
no mgcp timer receive-rtcp
mgcp sdp simple
mgcp fax t38 ecm
mgcp fax t38 inhibit
mgcp rtp payload-type g726r16 static
mgcp behavior rsip-range tgcp-only
mgcp behavior comedia-role none
mgcp behavior comedia-check-media-src disable
mgcp behavior comedia-sdp-force disable
!
mgcp profile default
!
sccp ccm x.x.x.x identifier 2 version 7.0
sccp ccm y.y.y.y identifier 1 version 7.0
!
sccp ccm group 1
associate ccm 1 priority 1
associate ccm 2 priority 2
associate profile 10 register CFB002545d426d0
!
ccm-manager music-on-hold
!
ccm-manager fallback-mgcp
ccm-manager redundant-host x.x.x.x
ccm-manager mgcp
no ccm-manager fax protocol cisco
ccm-manager config server x.x.x.x
!
dspfarm profile 10 conference
codec g729br8
codec g729r8
codec g729abr8
codec g729ar8
codec g711alaw
codec g711ulaw
maximum sessions 2
associate application SCCP
shutdown
!
dial-peer voice 10 pots
service mgcp
direct-inward-dial
port 0/1/0:15
forward-digits all
!
dial-peer voice 20 pots
service mgcp
direct-inward-dial
port 0/1/1:15
forward-digits all
!
!
!
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0 4
privilege level 15
transport input ssh
!
network-clock synchronization automatic
!
end

Roger Kallberg · ‎03-07-2022

For your clocking issue and in general ISDN configuration I would recommend that you make these changes.

network-clock synchronization automatic
network-clock input-source 1 controller E1 0/1/0
network-clock input-source 2 controller E1 0/1/1

controller E1 0/1/0
 clock source line primary

controller E1 0/1/1
 clock source line secondary

isdn switch-type primary-net5

There are additional alterations that I would recommend that you'll make as the configuration has no use or are down-right unneeded, but I'll circle back to that later.

Roger Kallberg · ‎03-08-2022

These would be what I would recommend you to change, apart from what already shared prior.

dial-peer voice 10 pots
 no direct-inward-dial
 no forward-digits all

dial-peer voice 20 pots
 no direct-inward-dial
 no forward-digits all
!These commands has no use for MGCP

dspfarm profile 10 conference
 no max sessions
 no codec g729br8
 no codec g729abr8
 maximum sessions ?  !set it to what is required, important to only use even numbers as CFB always uses the full module, even if the max session is set as an odd number. Thus it would waste DSPs that can not be used for anything else.
 no shut
!Turns off any codec that is enabled for VAD. This frees up DSP resources as these codecs have a higher impact on resource use. 

ip tftp source-interface GigabitEthernet0/0/0.3
!Tie the source interface to what is truly used on the gateway.

voice service voip
 no allow-connections h323 to h323
 no allow-connections h323 to sip
!These are Cube commands and has no relevance to MGCP.

b.winter · ‎03-06-2022

What's on the other end of the E1 connection?

A provider or a PBX?

Who is the "master" for the clock? Is it your router or the other end? Depending on that question, you have to configure the clocking accordingly.

CitrusCollab85908 · ‎03-07-2022

Service provider, is Optus, and we/customer have had this working for some time. Clocking is from provider/network, and it should be getting it via E1 0/1/0

The site gets busy period of 2 weeks of the year, and they have UCCX acting as IVR and queueing for calls during these two weeks.

Rest of the time is quiet, thus rare for them to even get a problem report.

Ratheesh Kumar · ‎03-07-2022

hi there

Whats the framing set ? trying changing that in MGCP gateway

Hope this Helps

Cheers
Rath!

***Please rate helpful posts and if applicable mark "Accept as a Solution"***

CitrusCollab85908 · ‎03-07-2022

We tried changing framing, it drops the circuit. thanks.

Ratheesh Kumar · ‎03-07-2022

hi there

Whats the framing set ? trying changing that in MGCP gateway

Hope this Helps

Cheers
Rath!

***Please rate helpful posts and if applicable mark "Accept as a Solution"***