So I had a strange incident this month and can't attribute it to anything yet, and TAC is too busy pointing fingers to other groups for me to get a straight answer out of them....
Cisco 2821 - H.323 voice gateway (had MGCP running on it in the past that was never fully shut off).
Jun 2 17:22:37.529 CDT: ISDN Se0/1/0:23 Q931: RX <- PROGRESS pd = 8 callref =0x9A6D
Cause i = 0x829F - Normal, unspecified
Progress Ind i = 0x8281 - Call not end-to-end ISDN, may have in-band info
Jun 2 17:22:37.553 CDT: ISDN Se0/1/0:23 Q931: TX -> DISCONNECT pd = 8 callref= 0x1A6D
Cause i = 0x80E6 - Recovery on timer expiry
Jun 2 17:22:37.565 CDT: ISDN Se0/1/0:23 Q931: RX <- RELEASE pd = 8 callref = 0x9A6D
Jun 2 17:22:37.569 CDT: ISDN Se0/1/0:23 Q931: TX -> RELEASE_COMP pd = 8 callref = 0x1A6D
CMD: 'controller T1 0/1/0 ' 03:32:01 CDT Tue Jun 3 2014
CMD: ' shutdown ' 03:32:01 CDT Tue Jun 3 2014
Jun 3 03:32:01.788 CDT: %CONTROLLER-5-UPDOWN: Controller T1 0/1/0, changed state to administratively down
CMD: 'end' 03:32:01 CDT Tue Jun 3 2014
Jun 3 03:32:01.792 CDT: %SYS-5-CONFIG_I: Configured from console by console
Jun 3 03:32:01.796 CDT: ISDN Se0/1/0:23 Q931: L3_ShutDown: Shutting down ISDN Layer 3
Jun 3 03:32:01.804 CDT: ISDN Se0/1/0:23 Q931: Ux_DLRelInd: DL_REL_IND receivedfrom L2
Jun 3 03:32:01.944 CDT: %MARS_NETCLK-3-HOLDOVER: Entering Holdover for Controller T1 0/1/0
Jun 3 03:32:03.788 CDT: %LINK-3-UPDOWN: Interface Serial0/1/0:23, changed state to down
Jun 3 03:32:12.196 CDT: %MARS_NETCLK-3-HOLDOVER_TRANS: Holdover timer exceeded for Controller T1 0/1/0
Jun 3 03:32:12.196 CDT: %MARS_NETCLK-3-CLK_TRANS: Network clock source transitioned from priority 1 to priority 10
1) This is the only PRI port that is currently active on my CUCM. Whatever happened ignored all of those and chose to shut down this specific port.
2) This all happened within 1 second so it was no a manual user intervention (automated by something)
3) There is no console connection on this device, nobody was in the DataCenter during this time.
4) This device runs AAA and there are no ACS logs during this timeframe to indicate that anyone was logged into the device remotely.
On the router itself, i show the following:
Current configuration : 8475 bytes
!
! Last configuration change at 03:32:28 CDT Tue Jun 3 2014
! NVRAM config last updated at 11:12:54 CST Tue Mar 4 2014 by jmad
! NVRAM config last updated at 11:12:54 CST Tue Mar 4 2014 by jmad
Note there is no "by %username%" for "Last configuration".
At this point I'm at a complete loss. The only thing i can think of that might have done this is CUCM or possibly the router itself shutting down a port that was acting up (which I believe should have given a different terminal message).
Ideas?
