Unable to make external Calls

[JBrav0]

First off im not an expert on VoIP.

So recently in the last 2 weeks, our phone system has been unable to make external calls and external calls are unable to come in. 

There is no busy signal, it's just silence.  Now, this isn't happening all day it seems to happen every day at around 3pm-4pm. 

The first thing I do is check that my gateway T1 line is running. 

everything is powered on, CPU utilization doesn't even hit 10% T1 line is up, the system has been up for 5 years and there are some errors Ill list them at the end. 

I always call the phone company to look at their stuff and they always say they didn't find anything (of course by the time they get there the next day things work after 1 or 2 hours of the phones being down). 

Nothing has been changed in the system and I'm not the one that's set it up. We can make internal calls to each other. 

In my mind, if it's a configuration problem on our end the system should have been down entirely for a long time already instead of it being recent. There is one T1 line and I did think of restarting the system, however, I got paranoid because it's old, and I've experienced restarting systems restarting and the battery dying off so before I do that I went and ordered identical backups just in case. I don't know why we already didn't have backups. 

Has anyone experienced this issue? or is there another form of troubleshooting that I may be missing?

here are the errors 

CCC-ISR4331-VG2#show controller t1

T1 0/1/0 is up.

  Applique type is Channelized T1

  Cablelength is long gain36 0db

  No alarms detected.

  alarm-trigger is not set

  Soaking time: 3, Clearance time: 10

  AIS State:Clear  LOS State:Clear  LOF State:Clear

  Framing is ESF, Line Code is B8ZS, Clock Source is Line Primary.

  Data in current interval (249 seconds elapsed):

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Total Data (last 24 hours)

     90642 Line Code Violations, 235 Path Code Violations,

     61 Slip Secs, 224 Fr Loss Secs, 40 Line Err Secs, 0 Degraded Mins,

     91 Errored Secs, 13 Bursty Err Secs, 0 Severely Err Secs, 382 Unavail Secs

The config for the T1 line:

Model: cisco ISR4331/K9 

controller T1 0/1/0
framing esf
clock source line primary
linecode b8zs
cablelength long 0db
pri-group timeslots 1-24 service mgcp

[Leonardo Santana]

HI,

Whats is the call flow?

OUTBOUND

IP PHONE -> CUCM -> MGCP GATEWAY-> T1 PRI -> PSTN

INBOUND:

PSTN -> T1 PRI->MGCP GATEWAY -> CUCM -> IP PHONE

When the problem start I recommend that you collect the debug:

debug voice ccapi inout

Show mgcp

show ccm-manager

Your gateway is located on the same site of your Cucm?

Regards

Leonardo Santana

[Leonardo Santana]

Just adding some links that might helps you:

https://www.cisco.com/c/en/us/support/docs/wan/t1-e1-t3-e3/14159-e1-flchrt-alarm-1.html?referring_site=bodynav 

https://www.cisco.com/c/pt_br/support/docs/wan/t1-e1-t3-e3/9319-E1-error.html 

[Nithin Eluvathingal]

I can see error on your PRI.

Total Data (last 24 hours)

     90642 Line Code Violations, 235 Path Code Violations,

     61 Slip Secs, 224 Fr Loss Secs, 40 Line Err Secs, 0 Degraded Mins,

     91 Errored Secs, 13 Bursty Err Secs, 0 Severely Err Secs, 382 Unavail Secs

Have you configured network-Clock commands. Check if you have  below metnioned commands on your configuration. If not add the below commands and clear the counts and see show controllers T1. If you see errors, contact your ISP.

network-clock synchronization automatic

network-clock input-source 1 controller T1 0/1/0

[JBrav0]

Yes, both of those have been added. As I mentioned previously this system has been running great for the last 5 years with the last 2 years no issues have ever come up. Nothing has been changed and all of a sudden these odd interruptions have started.

[Roger Kallberg]

If you have not done any changes on your end the suspicion would be on your service provider. Have you contacted them to discuss the issue you have?

[JBrav0]

Yes, every time and they keep saying everything is okay. They say if there are slip code violations it's due to the clock not being set to the T1 line, but I verified this, and it's set to the T1 line. At this point, I believe it's the cable or something in between and they are not doing a good job at checking. However, as I stated earlier this isn't a field I've experienced with so I don't feel confidant in ruling my end out.

[Leonardo Santana]

I understood you concern.

You can try to change your cable that connects to the TELCO modem just to test but i think the best path to follow is to contact their support and escalate this case.

You have the data from your counter to show them, is not just the slip code.

Regards

Leonardo Santana

[Nithin Eluvathingal]

In my region, isp always says everything fine from their side. Until  unless we escalate, they don’t work on our issues. 
Most of my similar issues where due to ISP problem.

so I don’t trust ISP.

I am not sure about your region. If you trust ISP, as others mentioned try a different NIM card.

[Leonardo Santana]

Hi,

Did you check this link:

https://www.cisco.com/c/pt_br/support/docs/wan/t1-e1-t3-e3/9319-E1-error.html 

Here you have the explanation of the counters and what you need to do if they are increasing.

Your counters:

Total Data (last 24 hours)

     90642 Line Code Violations, 235 Path Code Violations,

     61 Slip Secs, 224 Fr Loss Secs, 40 Line Err Secs, 0 Degraded Mins,

     91 Errored Secs, 13 Bursty Err Secs, 0 Severely Err Secs, 382 Unavail Secs

Has @Roger Kallberg and @Nithin Eluvathingal said yours counters are showing that you have errors on your link and you need to contact your TELCO.

Lets check one by one accordingly with the link:

Line Code Violations Increasing

Ensure the line coding configured on the port matches the line coding of the line. Look for Line Code is HDB3 in the show controller e1 output.

If Line Code Violations keep increasing, contact your Service Provider to check the E1 line as Line Code Violations can also be caused by physical line problems.

Path Code Violations Increasing

Ensure the framing format configured on the port matches the framig format of the line. Look for Framing is {crc4|no-crc4} in the show controller e1 output.

If Path Code Violations keep increasing, contact your Service Provider to check the E1 line as Path Code Violations can also be caused by physical line problems.

Framing Loss Seconds Increasing

Follow these instructions when dealing with a framing loss seconds increase.

1. Ensure the framing format configured on the port matches the framing format of the line. Look for Framing is {crc4|no-crc4} in the show controller e1 output.
2. To change the framing format, use the framing {crc4|no-crc4} command in the controller configuration mode. For example:

1. bru-nas-03(config-controller)#framing crc4

Contact your Service Provider and consult the T1/E1 Controller Commands documentation for details on build-out settings.

Slip Secs Counter Increasing

If slips are present on the E1 line, there is a clocking problem. The Customer Premises Equipment (CPE) will need to synchronize to the clocking from the E1 provider (telco). Complete the following steps to correct this problem:

1. Ensure the clock source is derived from the network. In the show controller e1 EXEC command output, look for Clock Source is Line Primary.

Note: If there are multiple E1s into an access server, only one can be the primary source. The other E1s derive the clock from the primary source. If there are multiple E1s, ensure the E1 line designated as the primary clock source is configured correctly. You may also configure a second E1 line to provide clocking in case the primary source goes down. To do this, use the clock source line secondary command from controller configuration mode.

1. Set the E1 clock source from controller configuration mode. For example:

bru-nas-03(config-controller)#clock source line

So contact your TELCO to check your T1 connection.

Regards

Leonardo Santana

[Nithin Eluvathingal]

If you have the network clock command which I have mentioned, Contact your ISP.  errors in PRI are  due to missing network clock commands or ISP side issues.'

Your PRI line is not stable.

[TONY SMITH]

I've seen intermittent problems on E1 lines caused by a faulty VWIC in the router.  Might be worth following up if you have hardware maintenance cover and the telco has repeatedly said there is no problem on the line.

[TONY SMITH]

The way I look at it, after the telco has claimed they have tested a few times, it is sort of your turn to prove your hardware is OK.  Even though 99 times out of 100 it's the telco, I have occasionally had what appeared to be telco problems actually caused by hardware.  If the kit's on maintenance I would be pressing for a replacement part.