Cisco 2851 crashes every night. Help!

[j.allnutt]

We have a Cisco 2851 router that crashes every night. Below is the 'show log' output. The provider is telling us that it is our equipment. We have replaced the router and still have the same problem. Is it our equipment or the provider? Please help.

Feb 15 19:29:43: %BGP-5-ADJCHANGE: neighbor x.x.x.x Down Interface flap

Feb 15 19:29:43: %BGP_SESSION-5-ADJCHANGE: neighbor x.x.x.x IPv4 Unicast topology base removed from session  Interface flap

Feb 15 19:29:44: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0:0, changed state to down

Feb 15 22:34:13: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to down (LOS detected)

Feb 15 22:34:15: %LINK-3-UPDOWN: Interface Serial0/0/0:0, changed state to down

[mavespig]

Hi,

not so much info in the logs but I have a few questions:

- does the router crash right after the serial goes down?

- is the timing correct? BGP goes down at 19:29, but the serial goes down 3 hours later

- does that happen every night? always at the same time?

- can you check if the router writes a crashinfo file in flash/disk, when it crashes?

- what the CPU and memory utilization at the time of crash? (approx)

Cheers

Marco

[j.allnutt]

Thanks for the reply. This is a remote router in PR, so we don't have a lot of information. Below are the answers.

-  does the router crash right after the serial goes down? I wouldn't say it is a 'crash' per say. The serial goes down and we lose connectivity. A reboot sometimes fixes the problem.

- is the timing correct? BGP goes down at 19:29, but the serial goes down 3 hours later? That is what we saw last night.

- does that happen every night? always at the same time? Yes, it has been happening a little after 4pm EST every day.

- can you check if the router writes a crashinfo file in flash/disk, when it crashes? No crash files on the router.

- what the CPU and memory utilization at the time of crash? (approx) Both were under 10% last night.

[Joseph Razionale]

Hi,

The provided router log is reporting a Loss Of Signal (LOS) on the T1 controller.

"show log | i T1"

Configure "loop-detect" under the T1 controller, and request that the provider intrusively test the T1 facility during off-hours, specifically when the trouble occurs.

[Leo Laohoo]

If the router crashes every night, then there's got to be crashlogs file/subdirectory.  If you do, can you please post the LAST FOUR (4) latest one?

If the router crashes every night, have you tried upgrading to another IOS version?

[Mandlenkosi Nkiwane]

Looking at the logs provided it seems there is a connection / circuit problem,  not a router crashing perse..

Like someone has already suggested, you can have the ISP to do intrusive testing on the circuit.

[Leo Laohoo]

-  does the router crash right after the serial goes down? I wouldn't say it is a 'crash' per say. The serial goes down and we lose connectivity. A reboot sometimes fixes the problem. 
- does that happen every night? always at the same time? Yes, it has been happening a little after 4pm EST every day.

Sounds like an ISP issue alright.  I've heard this before.  I've seen a case where the WAN link from the ISP turns to mush after 5pm daily and during the weekend.  Because the issue happens outside business hours no one bothered and the ISP couldn't replicate it.

Anyway when the calls got louder about how the network degrades after office hours, the ISP finally (or grudgingly) put a few 6-hour BER test machines at the customer premises.  This is where they found the complaints to be true.  They investigated and found out it's happening because ever time the place (where the nodes are) closes for the day, they turned off the power to the lights.  There was a short circuit (or something) which caused the circuits to go absolutely nuts.