T1 circuits stay Up/Down after carrier repair until router is rebooted

[ChuckHaynes]

Greetings,

We have several locations throughout the country that intermittently experience the same symptom and I was hoping to determine the cause and a solution (other than rebooting the router).

The majority of our sites are in very isolated areas and the only service available is T1 circuits. We typically have several (2-5) of them in a multilink bundle. The LEC is normally Frontier and the service provider is always AT&T. These locations are on AT&T's AVPN (MPLS) network. Due to the aging infrastructure and the number of sites, we generally see circuit issues daily. Routinely, after Frontier/AT&T have resolved a circuit problem, they will inform us that they are testing clean to the smartjack... however, we will still see the circuit being Up/Down. We can reboot the router and fix the issue, but obviously that takes down the entire site. I am trying to understand why this is happening and figure out another way to resolve it.

I have shut/no shut the interface. I've clear counters. The output of show interface doesn't show the router as being looped. Other ports on the T1 card are still working fine (with other circuits), so it isn't the entire card. Any ideas or suggestions would be greatly appreciated. The routers are typically 2911s running IOS 15.0-15.2.

Thanks

[Georg Pauwen]

Hello,

you are (obviously) running very old IOS software on very old routers. One thing you could do is to configure an EEM script that reacts to a syslog message and then bounces just that one interface that is down.

The script would look something like below. The matching pattern needs to be what you see in your router log when the circuit goes down:

event manager applet BOUNCE_T1

event syslog pattern "serialx/x/x up, line protocol down"

action 1.0 cli command "enable"

action 2.0 cli command "config t"

action 3.0 cli command "interface serial0/0/0:15"

action 4.0 cli command "shut"

action 5.0 cli command "no shut"

action 6.0 cli command "end"

action 7.0 end

[ChuckHaynes]

I meant that shutting and no shutting the port does not resolve the issue.

[Giuseppe Larosa · Accepted solution]

Hello @ChuckHaynes ,

each T1 / E1 has an associated controller.

You should try to shut / unshut the associated controller this should be enough to avoid to reload the router.

As suggested by @Georg Pauwen  the shut/ no shut action can be performed by an EEM script triggered by a syslog message.

controller t1 x/y

shut

! wait 20 seconds

no shut

Hope to help

Giuseppe

[ChuckHaynes]

I will try this and report back. Thanks.

[ChuckHaynes]

I tested this today and it worked. Thanks!