Understanding Call flow through CUCM to PBX

[RAustin70]

I am having a problem and I need help understanding something.

We have a CUCM 11.5.1 system installed on our site here.  Voice Gateways are 2 ISR 4431's with NIM-4MFT-T1/E1 & NIM-2MFT-T1/E1 installed in each.  2 ports go out to DSN, 4 go out to Local, 2 go out to Long Distance, and 4 are going to the Sites old PBX that we are replacing.  It is a CS1000 system.

The PBX has no trunks out to anything but our ISR's (0/1/2 & 0/1/3 on each ISR, specifically) and all calls pass through us.

My question is how does traffic flow through the ISR's to get to/from that PBX?  I am having difficulty troubleshooting an issue where STE and vIPer devices on the PBX can pass voice traffic just fine, but as soon as they try to go secure, the calls fail.  I am at a total loss at this point. 

We put a local T1 back on the PBX, set up an access trunk, and if the secure phones make calls over that trunk the secure calls succeed.  But if they try and go through the ISR's or a call comes in from the ISR, the secure calls fail.

Pulling my hair out, and I don't have much.

Rob

[R0g22]

I see you are still stuck with getting your STE phones working. I can understand your frustration. Can you tell me how and where does your CUCM get involved when you connect the PBX to the ISR ? What is the call flow like ?

[RAustin70]

HaHa, yes, extremely frustrating. 

When the CUCM was installed in May, the engineer created MGCP gateways for the ISR's.  On those gateways, there are 2 ports on each gateway (0/1/2 & 0/1/3) that are configured as PBX lines.

Call flow to my knowledge, goes like this: an Off-net call comes in from Verizon.  CUCM checks it against our DNs.  If it is not one of ours, it sends the call out the PBX T1's to be handled by that system (Nortel CS1000 Rel 5.0)

On the PBX, they have it set up that if a user goes off hook and dials a DN that is not on it's own system, it sends it down one of those 4 T1's for the CUCM to handle.

I am looking at a ridiculous # of Slips on those 4 T1's and we spent all day trying to get those down to near-zero.  But I don't even know if that is the real issue, or if those slips have been happening all along and something else is missing/misconfigured

[R0g22]

Slips can be a culprit. Let's get them away first. Can you attach "show run" and "show controller t1" output please ?

[RAustin70]

Great minds think alike.  Here are some parts of the run config I was putting together.  Let me know if you need any more of it.

controller T1 0/1/0
 framing esf
 linecode b8zs
 cablelength long 0db
 pri-group timeslots 1-24 service mgcp
 description DSN
!
controller T1 0/1/1
 framing esf
 clock source line secondary
 linecode b8zs
 cablelength long 0db
 pri-group timeslots 1-24 service mgcp
 description Local
!
controller T1 0/1/2
 framing esf
 linecode b8zs
 cablelength long 0db
 pri-group timeslots 1-24 service mgcp
 description PBX
!
controller T1 0/1/3
 framing esf
 linecode b8zs
 cablelength long 0db
 pri-group timeslots 1-24 service mgcp
 description PBX
!
controller T1 0/2/0
 framing esf
 linecode b8zs
 cablelength long 0db
 pri-group timeslots 1-24 service mgcp
 description Long Distance
!
controller T1 0/2/1
 framing esf
 clock source line primary
 linecode b8zs
 cablelength long 0db
 pri-group timeslots 1-24 service mgcp
 description Local
!
interface Serial0/1/0:23
 description DSN
 no ip address
 encapsulation hdlc
 isdn switch-type primary-ni
 isdn incoming-voice voice
 isdn bind-l3 ccm-manager
 no cdp enable
!
interface Serial0/1/1:23
 description LOCAL
 no ip address
 encapsulation hdlc
 isdn switch-type primary-ni
 isdn incoming-voice voice
 isdn bind-l3 ccm-manager
 no cdp enable
!
interface Serial0/1/2:23
 description PBX
 no ip address
 encapsulation hdlc
 isdn switch-type primary-5ess
 isdn protocol-emulate network
 isdn incoming-voice voice
 isdn bind-l3 ccm-manager
 no cdp enable
!
interface Serial0/1/3:23
 no ip address
 encapsulation hdlc
 isdn switch-type primary-ni
 isdn incoming-voice voice
 isdn bchan-number-order ascending
 no cdp enable
!
interface Serial0/2/0:23
 description LONG DISTANCE
 no ip address
 encapsulation hdlc
 isdn switch-type primary-dms100
 isdn incoming-voice voice
 isdn bind-l3 ccm-manager
 no cdp enable
!
interface Serial0/2/1:23
 description LOCAL
 no ip address
 encapsulation hdlc
 isdn switch-type primary-ni
 isdn incoming-voice voice
 isdn bind-l3 ccm-manager
 no cdp enable
!
!
voice-port 0/1/0:23
!
voice-port 0/2/0:23
!
voice-port 0/1/1:23
!
voice-port 0/2/1:23
!
voice-port 0/1/2:23
!
voice-port 0/1/3:23
!
!
network-clock synchronization automatic
network-clock input-source 1 controller T1 0/2/1
network-clock input-source 1 controller T1 0/1/1

Here is show controller t1

T1 0/1/0 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: DSN
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (614 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 1 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/1/1 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: Local
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line Secondary.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (614 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 1 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/1/2 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: PBX
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (614 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     38 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     38 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 1 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     49 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     49 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/1/3 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: PBX
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (614 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     38 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     38 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 1 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     51 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     51 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/2/0 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: Long Distance
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (614 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 1 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/2/1 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: Local
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line Primary.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (614 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 1 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

all my controllers out to verizon are clean.  Either I am missing something, or the PBX guy needs to change something.  I tried to set my clocking to internal on the PBX links, but it will not allow it with a pri-group assigned.

[R0g22]

A couple of things -

1. With ISR-4k, you don't want to use the default clocking. Remove "network-clock synchronization participate <slot/subslot" command for each T1 NIM card.
2. From my experience I can vouch for the fact that PBX's cannot provide clocking. The ISR needs to provide clocking to them. On the controller's that connect to the PBX, add "clock source network" command. Do NOT user internal if you have to provide clocking to the line. This used to work in ISR-G2. With 4k's, you need to use network parameter.

Once the changes are done, do a "clear counter" and monitor.

[RAustin70]

Dang it!!  I have a touch screen, and hit solved by accident.  Sorry.

so I am clear, you are telling me to:

conf t

no network-clock synchronization automatic

no network-clock input-source 1 controller T1 0/2/1

no network-clock input-source 1 controller T1 0/1/1

controller t1 0/1/2

clock source network

controller t1 0/1/3

clock source network

end

clear controller all

So, the ISR-4k knows to pull clocking from the "clock source line primary/secondary" entries on controllers 0/1/1 & 0/2/1?

[R0g22]

1. You need to have "network-clock synchronization automatic" command. This is not there by default.
2. The "network-clock synchronization participate <slot/subslot>" command is present by default for every T1/E1 NIM card. You need to remove it. If you do a "show run all" you will see this command.
3. You need to have "clock source network" configured on the controllers that connect to the PBX.
4. Remove the input source commands as well. I believe the controller being referenced through input source are your PBX facing controllers. If yes, remove them.
5. Do a "clear counters" and monitor with "show controller t1 x/y/z" command where x,y,z relate to slot/subslot/port.

[RAustin70]

1. Got it.  Leaving it in

2. see below output, don't seem to have those entries in either ISR

OA-2-ER1A-4431-1#sho run all | i network-c
 network-clock wait-to-restore 300
 network-clock hold-off 300
 network-clock source quality-level - tx
 network-clock source quality-level - rx
 network-clock wait-to-restore 300
 network-clock hold-off 300
 network-clock source quality-level - tx
 network-clock source quality-level - rx
no network-clock revertive
network-clock synchronization automatic
network-clock synchronization ssm option 1
network-clock synchronization input-threshold -
network-clock synchronization squelch-threshold QL-SEC
network-clock eec 1
network-clock hold-off 300 global
network-clock input-source 1 controller T1 0/2/1
network-clock input-source 1 controller T1 0/1/1
network-clock wait-to-restore 300 global

OA-2-ER1A-4431-2#sho run all | i network-c
 network-clock wait-to-restore 300
 network-clock hold-off 300
 network-clock source quality-level - tx
 network-clock source quality-level - rx
 network-clock wait-to-restore 300
 network-clock hold-off 300
 network-clock source quality-level - tx
 network-clock source quality-level - rx
no network-clock revertive
network-clock synchronization automatic
network-clock synchronization ssm option 1
network-clock synchronization input-threshold -
network-clock synchronization squelch-threshold QL-SEC
network-clock eec 1
network-clock hold-off 300 global
network-clock input-source 1 controller T1 0/2/1
network-clock input-source 1 controller T1 0/1/1
network-clock wait-to-restore 300 global

3.  Done

4. The input source commands used to be pointing to my PBX that is true.  But when we moved the Local Lines from the PBX to the ISRs, i changed them to the Local lines to get clocking from verizon.  Do I still pull them?

5. did clear controller all, and have to go home now or the family will not let me in the door.

Thank you.
Rob

[R0g22]

2. I believe you are on a recent 16.X IOS. The code must have been changed to accomodate the changes requested in the following defect -
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvb01800/?reffering_site=dumpcr

I would still want you to do that for every TI/E1 NIM you are using.

4. You would only remove them when you connect them to the PBX. If you are connected to Verizon and running clean, you need not remove them. When you test the controllers connected to the PBX, remove them during the testing.

[RAustin70]

Came in this morning, this is the output on the first ISR

T1 0/1/0 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: DSN
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (147 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 54 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     2 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     2 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/1/1 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: Local
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line Secondary.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (147 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 54 15 minute intervals):
     7457 Line Code Violations, 4 Path Code Violations,
     1 Slip Secs, 1 Fr Loss Secs, 2 Line Err Secs, 0 Degraded Mins,
     1 Errored Secs, 1 Bursty Err Secs, 0 Severely Err Secs, 11 Unavail Secs
T1 0/1/2 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: PBX
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Network.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (147 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     8 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     8 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 54 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     2961 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     2961 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/1/3 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: PBX
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Network.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (147 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     8 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     8 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 54 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     2959 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     2959 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/2/0 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: Long Distance
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (158 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 54 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
T1 0/2/1 is up
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  Description: Local
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line Primary.
  BER thresholds:  SF = 10e-3  SD = 10e-6
  Data in current interval (158 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
  Total Data (last 54 15 minute intervals):
     0 Line Code Violations, 0 Path Code Violations,
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

We are looking at nearly 3k slips per PBX interface.  The odd thing to me now is the Line code violations and other errors on 0/1/1 which is one of the local lines to the Demarc(verizon)

[RAustin70]

Yes we are on IOS XE 16.6.2.  I have found a command 'no network-clock participate slot/subslot.  Says it Enable/Disables a slot from participating in network-clocking.

I ran a show network clocks synchronization:

OA-2-ER1A-4431-1#sho network-clocks synchronization
Symbols:     En - Enable, Dis - Disable, Adis - Admin Disable
             NA - Not Applicable
             *  - Synchronization source selected
             #  - Synchronization source force selected
             &  - Synchronization source manually switched

Automatic selection process : Enable
Equipment Clock : 2048 (EEC-Option1)
Clock Mode : QL-Disable
ESMC : Disabled
SSM Option : 1
T0 : T1 0/2/1
Hold-off (global) : 300 ms
Wait-to-restore (global) : 300 sec
Tsm Delay : 180 ms
Revertive : No

Nominated Interfaces

 Interface            SigType     Mode/QL      Prio  QL_IN  ESMC Tx  ESMC Rx
 Internal             NA          NA/Dis       251   -         -         -      
*T1 0/2/1             NA          NA/Dis       1     -         -         -      
 T1 0/1/1             NA          NA/Dis       1     -         -         -    

[R0g22]

That's the same command that I asked you to remove for every NIM card configured on the router.

[RAustin70]

Thank you again Nipun!

     I think I am tracking what you're saying now.

I am to run:

no network-clock synchronization participate 0/1

no network-clock synchronization participate 0/2 to disable participation in the backplane clock.

Currently when I run these commands I get "Slot 0 subslot 1(2) is the source of secondary(primary) clock in use. Please unconfigure the source before disabling the slot. Command Aborted."

So, in controller T1 0/1/1 & controller T1 0/2/1, I am to remove the "clock source line primary | secondary" entries, reload, then re-run the command.

I cannot do this right now, I have to wait until after 6PM EST so I have time to ponder.

If I do not have those clock source line primary|secondary commands loaded, I am assuming that the commands that get clocking from Verizon are the network-clock input-source entries that are pointing to the Verizon ports?  Or just having the network-clock synchronization automatic is enough?

Rob

[R0g22]

So you would only remove those commands when you have the PRI connected to the PBX. You basically remove the those commands since you do not the NIM to participate in the router backplane clock. They are in their own clocking domain.

Next, you would need to remove the input source before you can remove the participation. There is no need to reload the router for this but since the router is in production I don't know if changing over to PBX during production will affect calls or not.