Tracked routes start to flap

[mcgiga]

Hello,

I have three ISPs configured in Cisco ASA. Route 1 and 2 are tracked. If ISP1 goes down it will switch to ISP2. If ISP2 goes down too it will switch to ISP3.
Yesterday everything looked good and worked. Today I noticed in the ASDM firewall log that ASA is deleting and creating routes every ~5 second.

I can successfully ping every destination (8.8.8.8, 1.1.1.1) from every ISP interface from the ASA.

The track and sla configuration looks like this:

sla monitor 150
type echo protocol ipIcmpEcho 8.8.8.8 interface WAN1
threshold 1000
frequency 10
sla monitor schedule 150 life forever start-time now

sla monitor 160
type echo protocol ipIcmpEcho 1.1.1.1 interface WAN2
threshold 1000
frequency 10
sla monitor schedule 160 life forever start-time now

In sla debug I get:

IP SLA Monitor(160) Scheduler: Starting an operation
IP SLA Monitor(160) echo operation: Sending an echo operation
IP SLA Monitor(150) Scheduler: Starting an operation
IP SLA Monitor(150) echo operation: Sending an echo operation
IP SLA Monitor(150) echo operation: RTT=10
IP SLA Monitor(150) Scheduler: Updating result
IP SLA Monitor(160) echo operation: Timeout
IP SLA Monitor(160) Scheduler: Updating result
IP SLA Monitor(150) Scheduler: Starting an operation
IP SLA Monitor(150) echo operation: Sending an echo operation
IP SLA Monitor(160) Scheduler: Starting an operation
IP SLA Monitor(160) echo operation: Sending an echo operation
IP SLA Monitor(160) echo operation: RTT=10
IP SLA Monitor(160) Scheduler: Updating result

What could that be? Is this somehow related to two trackings?

[mcgiga]

Metrics were:

WAN1 - 1
WAN2 - 2
WAN3 - 3
I have changed WAN3 to 254. But no difference, same issue. Now I have changed 1.1.1.1 to 8.8.4.4 like you mentioned.

Units are rebooting again for another test.

What's your guess why I should change to 8.8.4.4?

[MHM Cisco World]

Yes check 8.8.4.4 I see same issue when one engineer track IP 1.1.1.1.

Goodluck friend 

MHM

[mcgiga]

Issue is still present with 8.8.4.4.

It's weird, when I stop and restart both SLAs issue doesn't seem to exist. After a reboot of both units issue is present again.

[mcgiga]

6	Sep 30 2024	21:43:55	Adding tracked route 0.0.0.0 0.0.0.0 10.10.10.9, distance 2, table default, on interface WAN2
6	Sep 30 2024	21:43:55	Added STATIC route 0.0.0.0 0.0.0.0 via 0.0.0.0 [254/0] on Port-channel2.170 tableid [0]
6	Sep 30 2024	21:43:55	Added STATIC route 0.0.0.0 0.0.0.0 via 0.0.0.0 [2/0] on Port-channel2.160 tableid [0]
6	Sep 30 2024	21:43:54	8.8.8.8	0	10.10.10.10	9558	Teardown ICMP connection for faddr 8.8.8.8/0 gaddr 10.10.10.10/9558 laddr 10.10.10.10/9558 type 8 code 0
6	Sep 30 2024	21:43:52	8.8.4.4	0	20.20.20.20	27712	Teardown ICMP connection for faddr 8.8.4.4/0 gaddr 20.20.20.20/27712 laddr 20.20.20.20/27712 type 8 code 0
3	Sep 30 2024	21:43:52	8.8.8.8		10.10.10.10		Deny inbound icmp src WAN2:8.8.8.8 dst WAN1:10.10.10.10 (type 0, code 0)
6	Sep 30 2024	21:43:52	20.20.20.20	27712	8.8.4.4	0	Built outbound ICMP connection for faddr 8.8.4.4/0 gaddr 20.20.20.20/27712 laddr 20.20.20.20/27712 type 8 code 0
6	Sep 30 2024	21:43:52	10.10.10.10	9558	8.8.8.8	0	Built outbound ICMP connection for faddr 8.8.8.8/0 gaddr 10.10.10.10/9558 laddr 10.10.10.10/9558 type 8 code 0

Here you can see that WAN1 and WAN2 send a ping on their respective interface. Then ping send from WAN2 returns in but not on WAN2 but on WAN1 and gets blocked.

Somethins is wrong I think. Send from wrong interface? Received on wrong interface? I am lost.

[mcgiga]

When I enable:

This is gone:

3	Sep 30 2024	21:43:52	8.8.8.8		10.10.10.10		Deny inbound icmp src WAN2:8.8.8.8 dst WAN1:10.10.10.10 (type 0, code 0)

But routes are still flapping.

Stopping and starting SLAs fixes the issue.

[mcgiga · Accepted solution]

I have found the cause! The router in front of the firewall caused that route flapping.

It looks like icmp traffic originated from the firewall could not be send back to the correct firewall interface. This router handels two ISP provider in the current phase of configuration.

[MHM Cisco World]

Thanks for update us.

Can you more elaborate why there is asymmetric traffic?

MHM