HSRP 5 STATECHANGE

[wade.jae1]

I have been tasked with troubleshooting the cause of an HSRP 5 STATECHANGE that is happening intermittently between two production routers:

Feb 13 22:50:15.961: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Standby -> Active
Feb 13 22:50:18.517: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Active -> Speak
Feb 13 22:50:29.701: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Speak -> Standby
Feb 13 22:56:55.231: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Standby -> Active
Feb 13 22:57:08.795: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Active -> Speak
Feb 13 22:57:18.863: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Speak -> Standby
Feb 13 23:18:23.989: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Standby -> Active
Feb 13 23:18:26.721: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Active -> Speak
Feb 13 23:18:38.089: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Speak -> Standby
Feb 13 23:38:27.186: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Standby -> Active
Feb 13 23:38:41.322: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Active -> Speak
Feb 13 23:38:51.714: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Speak -> Standby

It will stop for as many as two days and then re-appear. 

Running config of the interfaces involved:

Current configuration : 376 bytes
!
interface GigabitEthernet0/0
description pal-tos-rt001c_ge-0/0 & pal-tos-sw001c_ge-0/1
ip address 10.33.120.1 255.255.255.0 secondary
ip address 10.33.244.1 255.255.255.240
ip nbar protocol-discovery
ip nat inside
ip virtual-reassembly
duplex full
speed 1000
standby 0 ip 10.33.244.3
standby 0 priority 250
standby 0 preempt
standby 0 track 1 decrement 200
end

Current configuration : 291 bytes
!
interface GigabitEthernet0/0
description pal-tos-rt002c_ge-0/0 & pal-tos-sw002c_ge-0/1
ip address 10.33.120.2 255.255.255.0 secondary
ip address 10.33.244.2 255.255.255.240
ip nat inside
ip virtual-reassembly
duplex full
speed 1000
standby 0 ip 10.33.244.3
standby 0 preempt
end

I woluld love some help. If anymore information is required let me know. Thanks!

[brselzer]

Hello,

This usually indicates we lost HSRP hellos. What kind of switches are there? Do we see high cpu at the time of the flapping? Do we see output or input drops on the interfaces?

These would be the things I check for as the standby will become active once we stop receiving hellos from the active.

Hope that helps!

[wade.jae1]

Thank you for the quick reply!

They are both Cisco 2901's running Version 15.0.

This is something that I have just started investigating today so I have not seen the issue happen in real time yet therefore I cannot answer whether or not there was high CPU at the time of the flapping. 

Perhaps the below command output can shed light:

#sh processes cpu history

pal-tos-rt002c 07:36:29 PM Tuesday Feb 20 2018 UTC

33311111111111111111111111111111111111 11111 22
100
90
80
70
60
50
40
30
20
10
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)

1
222222222332382622232323222322222222222222322222222322222222
100
90
80
70
60
50
40
30
20 *
10 * *
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

122122213222222121212211212211222222211222121221122122222212121212212121
919454832173318389823999183538812654338230818409840972509278939920893889
100
90
80
70
60
50
40
30 * * * * * * * * * * * ** * * * * * * *
20 *** *** ******* *** ******** ******** ************************** *******
10 ************************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

I also see no output drops on either interface. 

See interface stats:

sh int gi0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is 30e4.db67.3350 (bia 30e4.db67.3350)
Description: pal-tos-rt001c_ge-0/0 & pal-tos-sw001c_ge-0/1
Internet address is 10.33.244.1/28
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 33000 bits/sec, 56 packets/sec
5 minute output rate 33000 bits/sec, 54 packets/sec
1416898873 packets input, 2977589606 bytes, 0 no buffer
Received 66757760 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 64856198 multicast, 0 pause input
0 input packets with dribble condition detected
1430217084 packets output, 3598441869 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
1264190 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
1 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

sh int gi0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is 30e4.db67.3398 (bia 30e4.db67.3398)
Description: pal-tos-rt002c_ge-0/0 & pal-tos-sw002c_ge-0/1
Internet address is 10.33.244.2/28
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 21w5d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 32000 bits/sec, 53 packets/sec
5 minute output rate 32000 bits/sec, 53 packets/sec
368616844 packets input, 1344980744 bytes, 0 no buffer
Received 24628216 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 22731983 multicast, 0 pause input
0 input packets with dribble condition detected
355640655 packets output, 2770755081 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
438148 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

If I can provide anything else let me know.

[brselzer]

Has the issue happened in the last 72 hours? If so, the CPU seems normal at the time.

What does the topology look like? Are the 2901s directly connected or do they run through some L2 switches in order to exchange HSRP hellos? If so, you might want to check those devices as well to see if you see drops/high cpu/logs/etc.

Hope that helps!

[wade.jae1]

Wow, 

I am a first time user of these forums and I am already very impressed. I have to thank you for your time.

This issue last happened on 02/16/2018 @ 01:35 GMT - 1:45 GMT so not quite within the last 72 hours.

There is a L2 C2960 in between both HSRP devices. The routers connect to Gi0/1 and Gi0/2.

I notice the CPU is slightly higher on the L2 device:

#sh processes cpu history

11111
1111155555555554444455555555554444444444444444444444444444
100
90
80
70
60
50
40
30
20
10 *************** **********
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

1111111111111111 11111 1 111111111111111 131111 111111111
1110011111111110910111518101111111111110891711119111111011
100
90
80
70
60
50
40 *
30 *
20 *
10 *#****#*#**********************************#**#***********
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

3323222433233232232224326633437254242323362242214442333422542222224233
7348356345874395489865279954600015033139047408683018380361202673657451
100
90
80
70 ** *
60 ** * *
50 * ** * * ** * * *
40 * * * * * * * * *** * * ** * * * * *** * * ** * *
30 ** * ******** ** ************** ** * * **** *** ********* ** ** *** **
20 **********************************************************************
10 **********************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

I see no input or output drops on Gi0/1 or Gi0/2:

 #sh int gi0/2
GigabitEthernet0/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 64d9.89e4.bd9a (bia 64d9.89e4.bd9a)
Description: pal-tos-sw001c_ge-0/2 & pal-tos-sw002c_ge-0/2
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:20, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/0 (size/max)
30 second input rate 35000 bits/sec, 56 packets/sec
30 second output rate 37000 bits/sec, 59 packets/sec
2527094547 packets input, 1404762080104 bytes, 0 no buffer
Received 52245626 broadcasts (50898019 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 50898019 multicast, 0 pause input
0 input packets with dribble condition detected
2509317991 packets output, 188488823873 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

#sh int gi0/1
GigabitEthernet0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 64d9.89e4.bd99 (bia 64d9.89e4.bd99)
Description: pal-tos-sw001c_ge-0/1 & pal-tos-rt001c_ge-0/0
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:44, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 32000 bits/sec, 52 packets/sec
5 minute output rate 33000 bits/sec, 53 packets/sec
1091574579 packets input, 152943787838 bytes, 0 no buffer
Received 52157263 broadcasts (52083922 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 52083922 multicast, 0 pause input
0 input packets with dribble condition detected
1242155064 packets output, 96288891793 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Is there a debug I could run to hopefully pin point this?

[brselzer · Accepted solution]

Hello,

Happy to help. The L2 switch interfaces look fine. CPU on the L2 switch "shouldn't" be a factor as packets that are switched shouldn't hit the cpu under normal circumstances. 

I can look into debugs but I wouldn't advise turning them on until we have confirmed if we see high cpu or not because the debugs could compound your problem. I would advise checking CPU on all three switches the next time the issue happens. 

If you see high CPU, that changes are problem completely. HSRP is just a victim at that point and the real problem is the sudden spike in CPU on the router.

If we do not see high cpu, you could revisit debugs to see if there is another reason for the flaps.

The last thing I would recommend if you have the resources (a spare laptop running wireshark), is to setup a rolling capture off the 2960 towards the HSRP Standby. This will tell us a few things:

1) Are we really losing HSRP packets

2) If there was high cpu was it due to a lot of broadcast

3) Was there any other abnormal traffic on the link at the time

Hope that helps. Thanks!

[wade.jae1]

Hello,

I will be sure to check the CPU usage next time around.

Unfortunately, I do not have the luxury of Wireshark for this particular site.

I will let you know what I find.

Thanks again!

[Richard Burts]

I believe that Will has identified an important aspect of this situation. Can you share with us the details of configuration for tracking this object. Also can you post log messages from the other router in the HSRP group. I would be especially interested in whether they might include mention of a coup, which would confirm that the HSRP change was the result of changing priority due to tracking.

HTH

Rick

[wade.jae1]

Hello,

I was out yesterday. I am picking this back up.

Here is the config on the primary HSRP router that tracks the object:

#sh run int gigabitEthernet 0/0
Building configuration...

Current configuration : 376 bytes
!
interface GigabitEthernet0/0
 description pal-tos-rt001c_ge-0/0 & pal-tos-sw001c_ge-0/1
 ip address 10.33.120.1 255.255.255.0 secondary
 ip address 10.33.244.1 255.255.255.240
 ip nbar protocol-discovery
 ip nat inside
 ip virtual-reassembly
 duplex full
 speed 1000
 standby 0 ip 10.33.244.3
 standby 0 priority 250
 standby 0 preempt
 standby 0 track 1 decrement 200
end

Here is the running config for the tracked object:

#sh run | sec track 1
track 1 interface Serial0/0/0:0 line-protocol
 standby 0 track 1 decrement 200

Runing config of Se0/0/0:0:

#sh run int se0/0/0:0
Building configuration...

Current configuration : 227 bytes
!
interface Serial0/0/0:0
 description pal-tos-rt001c_se-0/0/0:0 & rvs-tos-rt001c_se-0/1/0:0 (XO PID: SF/HCFS/791615/ /TQW /) IPL-S.RVS.2775
 ip address 10.33.245.114 255.255.255.252
 ip nat outside
 ip virtual-reassembly
end

Se0/0/0:0 interface stats:

#sh int se0/0/0:0
Serial0/0/0:0 is up, line protocol is up
  Hardware is DSX1
  Description: pal-tos-rt001c_se-0/0/0:0 & rvs-tos-rt001c_se-0/1/0:0 (XO PID: SF/HCFS/791615/ /TQW /) IPL-S.RVS.2775
  Internet address is 10.33.245.114/30
  MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
     reliability 255/255, txload 4/255, rxload 4/255
  Encapsulation HDLC, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/664 (size/max/drops/flushes); Total output drops: 7214
  Queueing strategy: weighted fair
  Output queue: 0/1000/64/2019 (size/max total/threshold/drops)
     Conversations  0/11/256 (active/max active/max total)
     Reserved Conversations 0/0 (allocated/max allocated)
     Available Bandwidth 1152 kilobits/sec
  5 minute input rate 28000 bits/sec, 53 packets/sec
  5 minute output rate 28000 bits/sec, 55 packets/sec
     3902431903 packets input, 2912378440 bytes, 2019 no buffer
     Received 4439954 broadcasts, 2698 runts, 7 giants, 0 throttles
     10003 input errors, 4803 CRC, 0 frame, 5155 overrun, 0 ignored, 1057 abort
     3853982975 packets output, 3538179374 bytes, 0 underruns
     0 output errors, 0 collisions, 49035 interface resets
     27 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     191 carrier transitions
  Timeslot(s) Used:1-24, SCC: 0, Transmitter delay is 0 flags

Most recent logs from the configured Active router:

Feb 20 20:26:35.170: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to down (AIS detected)
Feb 20 20:26:35.390: %TRACKING-5-STATE: 1 interface Se0/0/0:0 line-protocol Up->Down
Feb 20 20:26:35.462: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Active -> Speak
Feb 20 20:26:37.170: %LINK-3-UPDOWN: Interface Serial0/0/0:0, changed state to down
.Feb 20 20:26:37.170: %OSPF-5-ADJCHG: Process 1, Nbr 10.33.249.7 on Serial0/0/0:0 from FULL to DOWN, Neighbor Down: Interface down or detached
.Feb 20 20:26:38.170: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0:0, changed state to down
.Feb 20 20:26:42.170: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to up
.Feb 20 20:26:43.170: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to down (RAI detected)
.Feb 20 20:26:47.190: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Speak -> Standby
.Feb 20 20:26:53.170: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to up
.Feb 20 20:26:55.171: %LINK-3-UPDOWN: Interface Serial0/0/0:0, changed state to up
.Feb 20 20:26:55.171: %TRACKING-5-STATE: 1 interface Se0/0/0:0 line-protocol Down->Up
.Feb 20 20:26:56.171: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to down (AIS detected)
.Feb 20 20:26:56.391: %TRACKING-5-STATE: 1 interface Se0/0/0:0 line-protocol Up->Down
.Feb 20 20:26:58.171: %LINK-3-UPDOWN: Interface Serial0/0/0:0, changed state to down
.Feb 20 20:27:10.171: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to up
.Feb 20 20:27:12.171: %LINK-3-UPDOWN: Interface Serial0/0/0:0, changed state to up
.Feb 20 20:27:12.171: %TRACKING-5-STATE: 1 interface Se0/0/0:0 line-protocol Down->Up
.Feb 20 20:27:13.171: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0:0, changed state to up
.Feb 20 20:27:13.463: %OSPF-5-ADJCHG: Process 1, Nbr 10.33.249.7 on Serial0/0/0:0 from LOADING to FULL, Loading Done
.Feb 20 20:27:14.375: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Standby -> Active

Most recent logs from the other (configured standby) HSRP router:

Feb 20 20:26:35.459: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Standby -> Active
Feb 20 20:27:14.376: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Active -> Speak
Feb 20 20:27:25.440: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Speak -> Standby

The engineer who designed this is no longer with us so I am unsure of whether or not this tracked object is necessary.

I am in the process now of troubleshooting the tracked object since it is a very dirty interface.

[brselzer]

Hello Wade,

The tracked object tracks the line protocol of the serial interface and if it goes down, it lowers its priority. The reason for this is, you don't want a router that doesn't have wan connectivity to be the active router for HSRP. 

It looks like that link has some issues. I would guess those interface resets are the cause of the problem. Once the interface goes down. HSRP will flap. It looks like there might be some physical issue with that connections if the CRCs and input errors are increasing. 

If it were me, I would change your priority so that the other router is active all the time to stop the HSRP flaps then start troubleshooting L1 on that serial connection. It also might be worth clearing the counters on that interface and see what is increasing. A lot of those counters could be old and unrelated to this problem.

Hope that helps!

[Richard Burts]

Thanks for the additional information. It does make clear that there is a relationship between the HSRP state changes and the issues with the serial interface. If you want to verify that you would turn on debug standby terse and look for messages about resign and coup.

In his explanation Bradley says "if it goes down, it lowers its priority". There is something implied there and I would like to make it explicit. When the active router lowers its priority to a value less than the priority of the HSRP standby, and if the standby is configured with preempt, then the standby takes over as active router. And that certainly looks like what is happening here.

When I was looking at the interface statistics for the serial interface I noticed the number of drops. If the packets being dropped include HDLC keepalive messages then might be a reason for the instability of the serial interface and perhaps that is something to look into.

But then I noticed these messages in the log output

CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to down (AIS detected)

and

CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to down (RAI detected)

These certainly indicate that there are issues on the serial link. Perhaps the output of show controller for the T1 controller might have helpful information.

Until you get this resolved I would agree with the suggestion from Bradley about changing the configuration to make the other HSRP router the active HSRP router.

HTH

Rick

[willwetherman]

Hi,

I noticed in your configs that the primary router is referencing a tracked object. If the tracked object fails then the router will generate the same HSRP state change messages that you are seeing. Have you checked that this tracked object is stable?

Will

[wade.jae1]

Hello,

Well after I read your reply I checked my logs again and the issue happened. It correlates directly to the tracked object (Se0/0/0:0 in this case) going up and then down:

Feb 20 20:26:35.170: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to down (AIS detected)
Feb 20 20:26:35.390: %TRACKING-5-STATE: 1 interface Se0/0/0:0 line-protocol Up->Down
Feb 20 20:26:35.462: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Active -> Speak
Feb 20 20:26:37.170: %LINK-3-UPDOWN: Interface Serial0/0/0:0, changed state to down
.Feb 20 20:26:37.170: %OSPF-5-ADJCHG: Process 1, Nbr 10.33.249.7 on Serial0/0/0:0 from FULL to DOWN, Neighbor Down: Interface down or detached
.Feb 20 20:26:38.170: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0:0, changed state to down
.Feb 20 20:26:42.170: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to up
.Feb 20 20:26:43.170: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to down (RAI detected)
.Feb 20 20:26:47.190: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Speak -> Standby
.Feb 20 20:26:53.170: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to up
.Feb 20 20:26:55.171: %LINK-3-UPDOWN: Interface Serial0/0/0:0, changed state to up
.Feb 20 20:26:55.171: %TRACKING-5-STATE: 1 interface Se0/0/0:0 line-protocol Down->Up
.Feb 20 20:26:56.171: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to down (AIS detected)
.Feb 20 20:26:56.391: %TRACKING-5-STATE: 1 interface Se0/0/0:0 line-protocol Up->Down
.Feb 20 20:26:58.171: %LINK-3-UPDOWN: Interface Serial0/0/0:0, changed state to down
.Feb 20 20:27:10.171: %CONTROLLER-5-UPDOWN: Controller T1 0/0/0, changed state to up
.Feb 20 20:27:12.171: %LINK-3-UPDOWN: Interface Serial0/0/0:0, changed state to up
.Feb 20 20:27:12.171: %TRACKING-5-STATE: 1 interface Se0/0/0:0 line-protocol Down->Up
.Feb 20 20:27:13.171: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0:0, changed state to up
.Feb 20 20:27:13.463: %OSPF-5-ADJCHG: Process 1, Nbr 10.33.249.7 on Serial0/0/0:0 from LOADING to FULL, Loading Done
.Feb 20 20:27:14.375: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Standby -> Active

This then causes the backup router to take over and back again:

Feb 20 20:26:35.459: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Standby -> Active
Feb 20 20:27:14.376: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Active -> Speak
Feb 20 20:27:25.440: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 0 state Speak -> Standby

I am working with my engineering team now.

Work continues.

Thanks for everyone's help!!!!