Tracked routes start to flap

[mcgiga]

Hello,

I have three ISPs configured in Cisco ASA. Route 1 and 2 are tracked. If ISP1 goes down it will switch to ISP2. If ISP2 goes down too it will switch to ISP3.
Yesterday everything looked good and worked. Today I noticed in the ASDM firewall log that ASA is deleting and creating routes every ~5 second.

I can successfully ping every destination (8.8.8.8, 1.1.1.1) from every ISP interface from the ASA.

The track and sla configuration looks like this:

sla monitor 150
type echo protocol ipIcmpEcho 8.8.8.8 interface WAN1
threshold 1000
frequency 10
sla monitor schedule 150 life forever start-time now

sla monitor 160
type echo protocol ipIcmpEcho 1.1.1.1 interface WAN2
threshold 1000
frequency 10
sla monitor schedule 160 life forever start-time now

In sla debug I get:

IP SLA Monitor(160) Scheduler: Starting an operation
IP SLA Monitor(160) echo operation: Sending an echo operation
IP SLA Monitor(150) Scheduler: Starting an operation
IP SLA Monitor(150) echo operation: Sending an echo operation
IP SLA Monitor(150) echo operation: RTT=10
IP SLA Monitor(150) Scheduler: Updating result
IP SLA Monitor(160) echo operation: Timeout
IP SLA Monitor(160) Scheduler: Updating result
IP SLA Monitor(150) Scheduler: Starting an operation
IP SLA Monitor(150) echo operation: Sending an echo operation
IP SLA Monitor(160) Scheduler: Starting an operation
IP SLA Monitor(160) echo operation: Sending an echo operation
IP SLA Monitor(160) echo operation: RTT=10
IP SLA Monitor(160) Scheduler: Updating result

What could that be? Is this somehow related to two trackings?

[balaji.bandi]

what ASA  code running on this device ?

we only see sla monitor, can you show us more information what is the action you doing.

some test i doing the Lab for dual ISP should be same for Tipple ISP :

https://www.balajibandi.com/?p=1932

[mcgiga]

ASA is running on 9.20(3).

track 1 rtr 150 reachability
!
track 2 rtr 160 reachability

route WAN1 0.0.0.0 0.0.0.0 10.10.10.10 1 track 1 (public ISP ip)
route WAN2 0.0.0.0 0.0.0.0 20.20.20.20 2 track 2 (public ISP ip)
route WAN3 0.0.0.0 0.0.0.0 192.168.150.1 3

[balaji.bandi]

Can post sla output and show route output ?

[mcgiga]

asa/pri/act# sh sla mon conf
SA Agent, Infrastructure Engine-II
Entry number: 150
Owner:
Tag:
Type of operation to perform: echo
Target address: 8.8.8.8
Interface: WAN1
Number of packets: 1
Request size (ARR data portion): 28
Operation timeout (milliseconds): 5000
Type Of Service parameters: 0x0
Verify data: No
Operation frequency (seconds): 10
Next Scheduled Start Time: Start Time already passed
Group Scheduled : FALSE
Life (seconds): Forever
Entry Ageout (seconds): never
Recurring (Starting Everyday): FALSE
Status of entry (SNMP RowStatus): Active
Enhanced History:

Entry number: 160
Owner:
Tag:
Type of operation to perform: echo
Target address: 1.1.1.1
Interface: WAN2
Number of packets: 1
Request size (ARR data portion): 28
Operation timeout (milliseconds): 5000
Type Of Service parameters: 0x0
Verify data: No
Operation frequency (seconds): 10
Next Scheduled Start Time: Start Time already passed
Group Scheduled : FALSE
Life (seconds): Forever
Entry Ageout (seconds): never
Recurring (Starting Everyday): FALSE
Status of entry (SNMP RowStatus): Active
Enhanced History:

---

asa/pri/act# sh sla mon oper
Entry number: 150
Modification time: 19:16:17.640 CEST Fri Sep 27 2024
Number of Octets Used by this Entry: 2056
Number of operations attempted: 1152
Number of operations skipped: 0
Current seconds left in Life: Forever
Operational state of entry: Active
Last time this entry was reset: Never
Connection loss occurred: FALSE
Timeout occurred: FALSE
Over thresholds occurred: FALSE
Latest RTT (milliseconds): 10
Latest operation start time: 22:28:07.641 CEST Fri Sep 27 2024
Latest operation return code: OK
RTT Values:
RTTAvg: 10 RTTMin: 10 RTTMax: 10
NumOfRTT: 1 RTTSum: 10 RTTSum2: 100

Entry number: 160
Modification time: 20:57:11.732 CEST Fri Sep 27 2024
Number of Octets Used by this Entry: 2272
Number of operations attempted: 546
Number of operations skipped: 0
Current seconds left in Life: Forever
Operational state of entry: Active
Last time this entry was reset: Never
Connection loss occurred: FALSE
Timeout occurred: FALSE
Over thresholds occurred: FALSE
Latest RTT (milliseconds): 10
Latest operation start time: 22:28:41.732 CEST Fri Sep 27 2024
Latest operation return code: OK
RTT Values:
RTTAvg: 10 RTTMin: 10 RTTMax: 10
NumOfRTT: 1 RTTSum: 10 RTTSum2: 100

---

Gateway of last resort is 10.10.10.10 to network 0.0.0.0

S* 0.0.0.0 0.0.0.0 [1/0] via 10.10.10.10, WAN1
C 10.10.10.9 255.255.255.248 is directly connected, WAN1
L 10.10.10.11 255.255.255.255 is directly connected, WAN1
C 20.20.20.19 255.255.255.248 is directly connected, WAN2
L 20.20.20.21 255.255.255.255 is directly connected, WAN2
C 192.168.150.0 255.255.255.248 is directly connected, WAN3
L 192.168.150.2 255.255.255.255 is directly connected, WAN3

---

I have removed internal routes from output.

After I have disabled the SLA160 some several minutes an enabled it later again, the issue is currently not present. But I have to make sure that it doesn't happen again.

[balaji.bandi]

After I have disabled the SLA160 some several minutes an enabled it later again, the issue is currently not present. But I have to make sure that it doesn't happen again.

i do not see high level any issue. make sure device have latest IOS Code running and keep Monitor and let us know how that goes.

also you can simulate remove the WAN1 and WAN2 all working as expected as part of testing, when no peak time ?

[mcgiga]

Today after I have booted both firewalls the issue is present again. This setup is not running in a productiv environement yet, so it was turned off.

It's adding and removing routes continously. I have no clue what's causing this. Every ISP-interface on the ASA can ping it's gateway and a random destination on WAN, like 1.1.1.1.

asa/pri/act# sh sla mon oper
Entry number: 150
Modification time: 16:33:25.276 CEST Mon Sep 30 2024
Number of Octets Used by this Entry: 2056
Number of operations attempted: 326
Number of operations skipped: 0
Current seconds left in Life: Forever
Operational state of entry: Active
Last time this entry was reset: Never
Connection loss occurred: FALSE
Timeout occurred: FALSE
Over thresholds occurred: FALSE
Latest RTT (milliseconds): 10
Latest operation start time: 17:27:25.277 CEST Mon Sep 30 2024
Latest operation return code: OK
RTT Values:
RTTAvg: 10 RTTMin: 10 RTTMax: 10
NumOfRTT: 1 RTTSum: 10 RTTSum2: 100

Entry number: 160
Modification time: 16:33:25.277 CEST Mon Sep 30 2024
Number of Octets Used by this Entry: 2056
Number of operations attempted: 326
Number of operations skipped: 0
Current seconds left in Life: Forever
Operational state of entry: Active
Last time this entry was reset: Never
Connection loss occurred: FALSE
Timeout occurred: TRUE
Over thresholds occurred: FALSE
Latest RTT (milliseconds): NoConnection/Busy/Timeout
Latest operation start time: 17:27:55.272 CEST Mon Sep 30 2024
Latest operation return code: Timeout
RTT Values:
RTTAvg: 0 RTTMin: 0 RTTMax: 0
NumOfRTT: 0 RTTSum: 0 RTTSum2: 0

[balaji.bandi]

what is the Model of ASA and code running, can you post all your show run removing confidence information.

so i can look and simulate for you.

[mcgiga]

Secure Firewall 3105, ASA is running 9.20(3).

---

asa/pri/act# sh run

ASA Version 9.20(3)
!
hostname asa
enable password
service-module 0 keepalive-timeout 4
service-module 0 keepalive-counter 6
service-module 1 keepalive-timeout 4
service-module 1 keepalive-counter 6
passwd
!
license smart
 feature tier standard
 feature strong-encryption
names
no mac-address auto

!
interface Ethernet1/1
 description LAN/STATE Failover Interface
!
interface Ethernet1/2
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/3
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/4
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/5
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/6
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/7
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/8
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/8.130
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/9
 description isp switch
 channel-group 2 mode active
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/10
 description isp switch
 channel-group 2 mode active
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/11
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/12
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/13
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/14
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/15
 description core switch
 channel-group 1 mode active
 no nameif
 no security-level
 no ip address
!
interface Ethernet1/16
 description core switch
 channel-group 1 mode active
 no nameif
 no security-level
 no ip address
!
interface Management1/1
 management-only
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Port-channel1
 description core switch
 no nameif
 no security-level
 no ip address
!
interface Port-channel1.100
 vlan 100
 nameif Transit-Net
 security-level 90
 ip address 192.168.150.1 255.255.255.254
!
interface Port-channel2
 description isp switch
 no nameif
 no security-level
 no ip address
!
interface Port-channel2.150
 description wan1
 vlan 150
 nameif WAN1
 security-level 0
 ip address 10.10.10.10 255.255.255.248
!
interface Port-channel2.160
 description wan2
 vlan 160
 nameif WAN2
 security-level 0
 ip address 20.20.20.20 255.255.255.248
!
interface Port-channel2.170
 description wan3
 vlan 170
 nameif WAN3
 security-level 0
 ip address 30.30.30.30 255.255.255.248
!
boot system disk0:/cisco-asa-fp3k.9.20.3.SPA
ftp mode passive
clock timezone xxx
dns domain-lookup WAN1
dns domain-lookup WAN2
dns domain-lookup WAN3
dns server-group DefaultDNS
 name-server 1.1.1.1
same-security-traffic permit intra-interface
no object-group-search access-control

<objects removed>

access-group Transit-Net_access_in in interface Transit-Net
access-group WAN1_in_1 in interface WAN1
access-group WAN2_access_in in interface WAN2
access-group WAN3_access_in in interface WAN3

<acls removed>

pager lines 24
logging enable
logging timestamp
logging asdm informational
mtu Management 1500
mtu Transit-Net 1500
mtu WAN1 1500
mtu WAN2 1500
mtu WAN3 1500
failover
failover lan unit primary
failover lan interface Failover-Link Ethernet1/1
failover link Failover-Link Ethernet1/1
failover interface ip Failover-Link 192.168.190.1 255.255.255.252 standby 192.168.190.2
failover ipsec pre-shared-key 8 xxx
no failover wait-disable
no monitor-interface Management
monitor-interface Transit
no monitor-interface WAN1
no monitor-interface WAN2
no monitor-interface WAN3
no monitor-interface service-module
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-openjre-7202.bin
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
arp rate-limit 32768
nat (Transit,WAN1) source dynamic any interface
nat (Transit,WAN2) source dynamic any interface
nat (Transit,WAN3) source dynamic any interface
route WAN1 0.0.0.0 0.0.0.0 10.10.10.10 1 track 1
route WAN2 0.0.0.0 0.0.0.0 20.20.20.20 2 track 2
route WAN3 0.0.0.0 0.0.0.0 30.30.30.30 3

<routes removed>

timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
timeout conn-holddown 0:00:15
timeout igp stale-route 0:01:10
user-identity default-domain LOCAL
aaa authentication ssh console LOCAL
aaa authentication login-history
http server enable
http server idle-timeout 30
no snmp-server location
no snmp-server contact
sla monitor 150
 type echo protocol ipIcmpEcho 8.8.8.8 interface WAN1
 threshold 1000
 frequency 30
sla monitor 160
 type echo protocol ipIcmpEcho 1.1.1.1 interface WAN2
 threshold 1000
 frequency 30
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpoint _SmartCallHome_ServerCA
 no validation-usage
 crl configure
crypto ca trustpoint _SmartCallHome_ServerCA2
 no validation-usage
 crl configure
crypto ca trustpoint ASDM_TrustPoint0
 enrollment terminal
 subject-name CN= xxx
 crl configure
crypto ca trustpoint ASDM_TrustPoint1
 keypair ASDM_TrustPoint1
 crl configure
crypto ca trustpool policy
 auto-import
crypto ca certificate chain _SmartCallHome_ServerCA
!
track 1 rtr 150 reachability
!
track 2 rtr 160 reachability
telnet timeout 30
ssh stricthostkeycheck
ssh timeout 30
ssh key-exchange group dh-group14-sha256
console timeout 0
no vpn-addr-assign aaa
no vpn-addr-assign dhcp
no ipv6-vpn-addr-assign aaa
no ipv6-vpn-addr-assign local
threat-detection basic-threat
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
ntp authentication-key 1 md5 8 xxx
ntp authenticate
ntp trusted-key 1

 enable WAN1
 enable WAN2
 http-headers
  hsts-server
   enable
   max-age 31536000
   include-sub-domains
   no preload
  hsts-client
   enable
  x-content-type-options
  x-xss-protection
  content-security-policy
 anyconnect image disk0:/secure_client/cisco-secure-client-win-5.1.6.103-webdeploy-k9.pkg 1
 anyconnect profiles SecureClient1 disk0:/secure_client/secureclient1.xml
 keepout "."
 cache
  disable
 error-recovery disable
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum client auto
  message-length maximum 512
  no tcp-inspection
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect rsh
  inspect rtsp
  inspect esmtp
  inspect sqlnet
  inspect skinny
  inspect sunrpc
  inspect sip
  inspect netbios
  inspect tftp
  inspect ip-options
 class class-default
  user-statistics accounting
!
service-policy global_policy global
prompt hostname priority state
no call-home reporting anonymous
call-home
 profile License
  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
  destination transport-method http
 profile CiscoTAC-1
  no active
  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
  destination address email callhome@cisco.com
  destination transport-method http
  subscribe-to-alert-group diagnostic
  subscribe-to-alert-group environment
  subscribe-to-alert-group inventory periodic monthly
  subscribe-to-alert-group configuration periodic monthly
  subscribe-to-alert-group telemetry periodic daily
password encryption aes
hpm topN enable
no flow-offload enable

That would be really great if you could test it. My current guess is that the ASA can't handel two tracked routes because there is no boolean link between route 1 and 2.

[balaji.bandi]

You have port-channel sub-interface configured here, what is the other side connected  to Switch, what switch can you post that switch config, and is all the ISP connected to that switch ?

why do you need port-channel you have many interface on device can you not connected directly to ISP to this ports ?

[MHM Cisco World]

you have three ISP?

did you try add different AD to third route and check 

MHM

[mcgiga]

Yes three ISP.

Different AD?

[MHM Cisco World]

Use 8.8.4.4 instead of 1.1.1.1 and check

MHM

[mcgiga]

I changed metric to 254 and 1.1.1.1 to 8.8.4.4. Units are rebooting now because after disabling and enabling SLA again, routes don't flap (same like last week).

I will report back in some minutes.

[MHM Cisco World]

The defualt route toward wan2 must have less AD than third defualt toward wan3 

So make second ad 200

Third make it ad 254 

Note:- dont use ad 255 it not work in ftd/asa

MHM