IP SLA process

[ubsaccount]

Mar 6 16:24:27.548 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Down->Up
Mar 6 16:29:17.573 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Up->Down
Mar 6 16:30:27.587 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Down->Up
Mar 6 17:31:17.958 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Up->Down
Mar 6 17:32:27.975 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Down->Up
Mar 6 17:57:18.102 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Up->Down
Mar 6 17:58:28.119 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Down->Up
Mar 6 18:13:18.211 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Up->Down
Mar 6 18:15:28.234 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Down->Up
Mar 6 18:20:18.259 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Up->Down
Mar 6 18:21:28.272 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Down->Up
Mar 6 18:30:18.322 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Up->Down
Mar 6 18:31:28.335 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Down->Up

So we get this message every day with a service provider that has lousy service.  

The Timeout is set at 30 seconds which is long, but as I watch the IP SLA count down to 0 Seconds I would think IP SLA is preparing to switch over and this happens a lot.  But doesn't seem like the IP SLA really switch routes because the ICMP-ECHO recovers.  I am trying to understand is IP SLA removing the track routes from the RIB then adding it back to the RIB when it recovers causing a flap? Also the 1641 changes is that the number of times IP SLA attempted to switch over?

IP SLA 1 reachability
Reachability is Down, delayed Up (5 secs remaining)
1641 changes, last change 00:01:04
Delay up 60 secs, down 20 secs
Latest operation return code: Timeout
Tracked by:
STATIC-IP-ROUTING 0
Router 1#sho track 1
Track 1
IP SLA 1 reachability
Reachability is Down, delayed Up (1 sec remaining)
1641 changes, last change 00:01:09
Delay up 60 secs, down 20 secs
Latest operation return code: Timeout
Tracked by:
STATIC-IP-ROUTING 0

This is what pinging to google looks like.

Sending 500, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!.!!!!.!!!!!!.!!!.!!!!!!!.!!!.!!!!!!!!!!.!

[Philip D'Ath]

Could you share your actual sla configuration?

[ubsaccount]

Something like this.  The threshold is default 5000

ip sla 1
icmp-echo 8.8.8.8 source-interface Gi0/0
timeout 30000
ip sla schedule 1 start-time now life forever

track 10 ip sla 1 reachability

delay down 30 up 60

ip route 0.0.0.0 0.0.0.0 45.66.23.9 track 10

ip route 0.0.0.0 0.0.0.0 67.23.45.2 10

[Philip D'Ath]

Try adding the below as well.  You want to guarantee your test traffic is going out the interface being tested.  Simply specifying the source interface is not sufficient.

ip route 8.8.8.8 255.255.255.255 45.66.23.9 permanent

[ubsaccount]

What I am asking when the message shows 

Mar 6 16:29:17.573 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Up->Down
Mar 6 16:30:27.587 UTC: %TRACKING-5-STATE: 1 ip sla 1 reachability Down->Up

Is the default route being removed from RIB.  I kept refreshing to see the changes , but cannot.    I'm thinking if we send an ICMP-echo to a different source our internet will not flap.  Even though the ISP has crappy service

[Philip D'Ath]

Correct, the default route will be removed and then re-added.