IP SLA and Track stopped working

[qasimkhans8]

HI all,

         I am using track and IP sla on my 3845 router. suddenly it stopped working. i cannot ping any internet IP address. even my ISP gateway. here is my IP sla and track.

track 10 ip sla 1 reachability
delay down 10 up 10
!
track 20 ip sla 2 reachability
delay down 10 up 10

ip sla 1
icmp-echo 50.x.x.x source-interface FastEthernet0/0/1
threshold 1000
timeout 3000
frequency 3
ip sla schedule 1 life forever start-time now

ip sla 2
icmp-echo 162.x.x.x source-interface FastEthernet0/1/0
threshold 2000
timeout 3000
frequency 3
ip sla schedule 2 life forever start-time now

route-map PBR permit 10
match ip address acl_natisp1
set ip next-hop verify-availability 50.x.x.x 1 track 10
!
route-map PBR permit 20
match ip address acl_natisp2
set ip next-hop verify-availability 162.x.x.x 2 track 20
!
route-map ISP2 permit 20
match ip address acl_internet
match interface FastEthernet0/1/0
!
route-map ISP1 permit 10
match ip address acl_internet
match interface FastEthernet0/0/1

Router#sh ip sla statistics
IPSLAs Latest Operation Statistics

IPSLA operation id: 1
Latest RTT: NoConnection/Busy/Timeout
Latest operation start time: 09:57:09 EST Mon Aug 28 2017
Latest operation return code: Timeout
Number of successes: 0
Number of failures: 266
Operation time to live: Forever

IPSLA operation id: 2
Latest RTT: NoConnection/Busy/Timeout
Latest operation start time: 09:57:09 EST Mon Aug 28 2017
Latest operation return code: Timeout
Number of successes: 0
Number of failures: 259
Operation time to live: Forever

Router#show track
Track 10
IP SLA 1 reachability
Reachability is Down
15 changes, last change 1d21h
Delay up 10 secs, down 10 secs
Latest operation return code: Timeout
Tracked by:
ROUTE-MAP 0
STATIC-IP-ROUTING 0
Track 20
IP SLA 2 reachability
Reachability is Down
237 changes, last change 1d21h
Delay up 10 secs, down 10 secs
Latest operation return code: Timeout
Tracked by:
ROUTE-MAP 0

currently 2nd ISP is working. even with these output my 2nd ISP is down,

all my users are on 2nd ISP. 1st ISP is not working.

If i directly connect PC to 1st ISP. its working. 

Thanks.

[Francesco Molino]

Hi 

Can you share the full config please?

You posted some PBR but we don't know where those PBR are applied.

Also if you take 1 of your IP sla, can you test from the router a ping to 162.x.x.x from interface FastEthernet0/1/0.

Remove any PBR first to make sure we capture the right output for the test.

Thanks

PS: Please don't forget to rate and select as validated answer if this answered your question

[qasimkhans8]

here is my complete config 

1. track 10 ip sla 1 reachability
2.  delay down 10 up 10
3. !
4. track 20 ip sla 2 reachability
5.  delay down 10 up 10
7. interface FastEthernet0/0/1
8.  ip address 50.x.x.10 255.255.255.248
9.  ip nat outside
10.  ip virtual-reassembly in
11.  duplex auto
12.  speed 100
14. !
15. interface FastEthernet0/1/0
16.  ip address 162.x.x.34 255.255.255.224
17.  ip nat outside
18.  ip virtual-reassembly in
19.  duplex auto
20.  speed auto
21. !
24. ip nat inside source route-map ISP1 interface FastEthernet0/0/1 overload
25. ip nat inside source route-map ISP2 interface FastEthernet0/1/0 overload
27. ip route 0.0.0.0 0.0.0.0 50.x.x.9 track 10
28. ip route 0.0.0.0 0.0.0.0 162.x.x.62 200
30. ip access-list extended acl_internet
31.  deny   ip 192.168.0.0 0.0.255.255 10.10.10.0 0.0.0.255
32.  deny   ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
33.  permit ip 192.168.0.0 0.0.255.255 any
35. ip access-list extended acl_natisp1
36.  deny   ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
37.  permit ip 192.168.0.0 0.0.255.255 any
39. ip access-list extended acl_natisp2
40.  deny   ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
41.  permit ip 192.168.0.0 0.0.255.255 any
45. ip sla 1
46.  icmp-echo 50.x.x.9 source-interface FastEthernet0/0/1
47.  threshold 1000
48.  timeout 3000
49.  frequency 3
50. ip sla schedule 1 life forever start-time now
52. ip sla 2
53.  icmp-echo 162.x.x.62 source-interface FastEthernet0/1/0
54.  threshold 2000
55.  timeout 3000
56.  frequency 3
57. ip sla schedule 2 life forever start-time now
59. route-map PBR permit 10
60.  match ip address acl_natisp1
61.  set ip next-hop verify-availability 50.x.x.9 1 track 10
62. !
63. route-map PBR permit 20
64.  match ip address acl_natisp2
65.  set ip next-hop verify-availability 162.x.x.62 2 track 20
66. !
67. route-map ISP2 permit 20
68.  match ip address acl_internet
69.  match interface FastEthernet0/1/0
70. !
71. route-map ISP1 permit 10
72.  match ip address acl_internet
73.  match interface FastEthernet0/0/1
77. Router#sh ip sla statistics
78. IPSLAs Latest Operation Statistics
80. IPSLA operation id: 1
81.         Latest RTT: NoConnection/Busy/Timeout
82. Latest operation start time: 09:57:09 EST Mon Aug 28 2017
83. Latest operation return code: Timeout
84. Number of successes: 0
85. Number of failures: 266
86. Operation time to live: Forever
88. IPSLA operation id: 2
89.         Latest RTT: NoConnection/Busy/Timeout
90. Latest operation start time: 09:57:09 EST Mon Aug 28 2017
91. Latest operation return code: Timeout
92. Number of successes: 0
93. Number of failures: 259
94. Operation time to live: Forever
99. Router#show track
100. Track 10
101.   IP SLA 1 reachability
102.   Reachability is Down
103.     15 changes, last change 1d21h
104.   Delay up 10 secs, down 10 secs
105.   Latest operation return code: Timeout
106.   Tracked by:
107.     ROUTE-MAP 0
108.     STATIC-IP-ROUTING 0
109. Track 20
110.   IP SLA 2 reachability
111.   Reachability is Down
112.     237 changes, last change 1d21h
113.   Delay up 10 secs, down 10 secs
114.   Latest operation return code: Timeout
115.   Tracked by:
116.     ROUTE-MAP 0

i cannot ping to 162.x.x.62 from interface FastEthernet0/1/0. even i cannot ping any public ip from my router including my 1st and 2nd ISP's gateway ip address.

[Francesco Molino]

If you can't ping your next hop from your interface that explain why your IP SLA are down.

Can you share the show ip int brie ?

Are they directly connected or physical cabling is passing through a switch ?

Thanks

PS: Please don't forget to rate and select as validated answer if this answered your question

[qasimkhans8]

my both router interfaces are directly connected to ISP's devices.

Interface                     IP-Address              OK? Method Status Protocol
GigabitEthernet0/0      unassigned              YES NVRAM up up
GigabitEthernet0/1      unassigned              YES NVRAM up up
FastEthernet0/0/0        unassigned             YES NVRAM administratively down down
FastEthernet0/0/1        50. x.x.10               YES NVRAM up up
FastEthernet0/1/0        162.x.x.34              YES NVRAM up up

currently my 2nd ISP is working. but show ip sla stat    shows its down.

1. IPSLA operation id: 1
2.         Latest RTT: NoConnection/Busy/Timeout
3. Latest operation start time: 09:57:09 EST Mon Aug 28 2017
4. Latest operation return code: Timeout
5. Number of successes: 0
6. Number of failures: 266
7. Operation time to live: Forever
9. IPSLA operation id: 2
10.         Latest RTT: NoConnection/Busy/Timeout
11. Latest operation start time: 09:57:09 EST Mon Aug 28 2017
12. Latest operation return code: Timeout
13. Number of successes: 0
14. Number of failures: 259
15. Operation time to live: Forever

[Francesco Molino]

Can you please do : ping 162.x.x.62 source FastEthernet0/1/0 and paste the output

also can you stop your IPSLA: no ip sla schedule 2 life forever start-time now
and start it again: ip sla schedule 2 life forever start-time now

PAste the output of show track and sh ip sla statis.

can you do debug track and debug ip sla and paste the output as well.

thanks

[qasimkhans8]

ping 162.x.x.62 source fastEthernet 0/1/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 162.x.x.62, timeout is 2 seconds:
Packet sent with a source address of 162.x.x.34
.....
Success rate is 0 percent (0/5)

i did 

no ip sla schedule 2 life forever start-time now

ip sla schedule 2 life forever start-time now

sh track
Track 10
IP SLA 1 reachability
Reachability is Down
5 changes, last change 20:33:25
Delay up 10 secs, down 10 secs
Latest operation return code: Timeout
Tracked by:
ROUTE-MAP 0
STATIC-IP-ROUTING 0
Track 20
IP SLA 2 reachability
Reachability is Down
3 changes, last change 3d21h
Delay up 10 secs, down 10 secs
Latest operation return code: Timeout
Tracked by:
ROUTE-MAP 0

ITC-R1-R3845#sh ip sla statistics
IPSLAs Latest Operation Statistics

IPSLA operation id: 1
Latest RTT: NoConnection/Busy/Timeout
Latest operation start time: 13:59:58 EST Tue Aug 29 2017
Latest operation return code: Timeout
Number of successes: 0
Number of failures: 24679
Operation time to live: Forever

IPSLA operation id: 2
Latest RTT: NoConnection/Busy/Timeout
Latest operation start time: 10:33:43 EST Wed Aug 30 2017
Latest operation return code: Timeout
Number of successes: 0
Number of failures: 29
Operation time to live: Forever

there is no output when i do debug track and debug ip sla

[Francesco Molino]

Ok don't to check why IP SLA is down anymore. If you can't ping your gateway, that's normal it's down.

Are you directly attached to your ISP router?
You need to figure out first why your ISP router isn't pinging.
If you connect a laptop directly to your ISP router (as you did if I understood) to test, can you ping the ISP interface IP 162.x.x.62 ?

Thanks

[qasimkhans8]

Are you directly attached to your ISP router?

Yes,

You need to figure out first why your ISP router isn't pinging.

I can ping ISP gateway if i connect my laptop direct to ISP device.

If you connect a laptop directly to your ISP router (as you did if I understood) to test, can you ping the ISP interface IP 162.x.x.62 ?

yes i can ping ISP gateway.

i was seach about this problem and came to know following bug "Timeout issue on IP SLA". is it ture?

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCso46681/?referring_site=bugquickviewredir

[Francesco Molino]

Maybe you're facing that bug but before we need to be sure that your router ping its gateway and then we will be able to see what's going on with IP SLA.

Was it working before? What did you change? Can you test by changing the speed and duplex just in case!

Right now you've a weird issue not really related to IP SLA. On your point to point you can't ping the remote router.

Ok stop your IP SLA.

Shut your ISP1. Shut/no shut your ISP2.

Test it again.

If not pinging yet, :

a. can you paste the output of show interface FastEthernet0/1/0

b. change the cable just to be sure or test it

c. reboot your router

Let me know when you did that if you can ping the router.

Thanks

[qasimkhans8]

Was it working before? What did you change? Can you test by changing the speed and duplex just in case!

yes everything was working perfectly. i did not change anything. suddenly it stopped working, i restart the router and 2nd ISP satarted working. while IP SLA shows timeout.

Right now you've a weird issue not really related to IP SLA. On your point to point you can't ping the remote router.

I dont think so it is cable issue. because 2nd ISP is working.

Ok stop your IP SLA.

Shut your ISP1. Shut/no shut your ISP2.

Test it again.

i shut down ISP 1 interface and sh/no sh ISP 2, still cannot ping ISPs gateway. but ISP 2 internet is working.

If not pinging yet, :

a. can you paste the output of show interface FastEthernet0/1/0

sh interfaces fastEthernet 0/1/0
FastEthernet0/1/0 is up, line protocol is up
Hardware is FastEthernet, address is fc99.4792.693e (bia fc99.4792.693e)
Internet address is 162.x.x.34/27
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 31/255, rxload 50/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 1/75/0/5 (size/max/drops/flushes); Total output drops: 2549
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 19904000 bits/sec, 2271 packets/sec
5 minute output rate 12296000 bits/sec, 2203 packets/sec
312616934 packets input, 2949511771 bytes
Received 4524 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
376449590 packets output, 2019245671 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

b. change the cable just to be sure or test it

I think cable is good

c. reboot your router

i reboot the router when this issue occurred.

[Francesco Molino]

I understand that your isp2 works with a laptop connected directly to isp router.
However, from your internal router you're not able to ping your isp gateway and that's not normal.
Then we should troubleshot all physical connectivity and reboot it to make sure it's not stuck somewhere.

[qasimkhans8]

my ISP 2 working through my "router" (where IP SLA is not working).

i am surprised IP SLA is timeout to both ISPs. i cannot ping both ISPs gateways. but ISP 2 is still working on my "router". i sh/no sh both interfaces. disable / enable IP SLAs as you suggested. but both IP SLAs are still timeout. 

[qasimkhans8]

Hi,

     any thought on my question.

[Francesco Molino]

Hi,

Are you able now to ping your ISP from your router?
have you tried pinging ISP2 from a laptop directly connected? Maybe that's just because icmp is dropped by ISP router. In that case you will need to ask them or to use another method of ip sla like tcp-connect.