Re: route-map verify-availability with track does not fail over

dal · ‎05-20-2009

Hi!

We have 2 internet gateways (ISP-A and ISP-B), so I have made a configuration using tracks to set Gateway of last resort depending on what ISP is up.

This seems to work perfectly fine.

BUT; In addition to this, only a few of our subnets are "allowed" to use ISP-B, so I have made a route-map:

route-map test permit 10

match ip address 10

set ip next-hop verify-availability 158.38.16.129 100 track 1

set ip next-hop verify-availability 213.184.198.1 101 track 2

And this works, even though 213.184.198.1 is the default gateway, the traffic is re-routed because of this route-map.

The problem starts when 158.38.16.129 dissapeares. The other gateway, 213.184.198.1 should then take over, but it does not. Even though the trackers clearly says that the gateway is down:

InternettGW#sh track

Track 1

Response Time Reporter 1 reachability

Reachability is Down

5 changes, last change 00:29:57

Delay down 5 secs

Latest operation return code: Timeout

Tracked by:

ROUTE-MAP 0

STATIC-IP-ROUTING 0

Track 2

Response Time Reporter 2 reachability

Reachability is Up

3 changes, last change 00:37:47

Delay down 5 secs

Latest operation return code: OK

Latest RTT (millisecs) 1

Tracked by:

ROUTE-MAP 0

STATIC-IP-ROUTING 0

InternettGW#sh route-map

route-map test, permit, sequence 10

Match clauses:

ip address (access-lists): 10

Set clauses:

ip next-hop verify-availability 158.38.16.129 100 track 1 [down]

ip next-hop verify-availability 213.184.198.1 101 track 2 [up]

Policy routing matches: 30 packets, 3500 bytes

Any ideas why?

I am posting the complete configuration if someone is interested.

Thank you.

----------------------------------------------------------------------------

!

version 12.4

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname InternettGW

!

boot-start-marker

boot system flash:c1841-advipservicesk9-mz.124-4.XC7.bin

boot-end-marker

!

enable secret 5 $1$EBrC$vpM9NVPlgxLd2SpRe0ymi0

enable password whatteevver

!

no aaa new-model

!

resource policy

!

ip cef

!

ip name-server 213.184.198.11

ip sla 1

icmp-echo 158.38.0.168 source-interface FastEthernet0/0

frequency 5

ip sla schedule 1 life forever start-time now

ip sla 2

icmp-echo 213.184.200.1 source-interface FastEthernet0/1

frequency 5

ip sla schedule 2 life forever start-time now

!

archive

log config

hidekeys

!

track 1 rtr 1 reachability

delay down 5

!

track 2 rtr 2 reachability

delay down 5

!

interface FastEthernet0/0

description * ISP-B *

ip address 158.38.16.140 255.255.255.128

ip policy route-map test

duplex auto

speed auto

!

interface FastEthernet0/1

description * ISP-A *

ip address 213.184.198.137 255.255.255.0

ip policy route-map test

duplex auto

speed auto

!

interface FastEthernet0/0/0

switchport mode trunk

!

interface FastEthernet0/0/1

shutdown

!

interface FastEthernet0/0/2

shutdown

!

interface FastEthernet0/0/3

shutdown

!

interface Vlan1

no ip address

shutdown

no mop enabled

!

interface Vlan53

ip address 10.50.53.10 255.255.255.0

!

ip route 0.0.0.0 0.0.0.0 158.38.16.129 101 track 1

ip route 0.0.0.0 0.0.0.0 213.184.198.1 100 track 2

ip route 10.10.10.205 255.255.255.255 10.50.53.1

ip route 10.50.50.240 255.255.255.255 10.50.53.1

ip route 158.38.0.168 255.255.255.255 158.38.16.129

ip route 213.184.200.1 255.255.255.255 213.184.198.1

!

ip http server

no ip http secure-server

!

access-list 10 permit 158.38.16.142 // Thisis the ip address I'm testing with.

!

route-map test permit 10

match ip address 10

set ip next-hop verify-availability 158.38.16.129 100 track 1

set ip next-hop verify-availability 213.184.198.1 101 track 2

!

control-plane

!

line con 0

line aux 0

line vty 0 4

password whatevvere

login

!

scheduler allocate 20000 1000

!

webvpn context Default_context

ssl authenticate verify all

!

no inservice

!

end

Laurent Aubert · ‎05-20-2009

Hi,

From what I see, your source is on the same subnet as your router and the ISP gateway.

Could you check if your router is sending ICMP redirect messages with a debug icmp command. If it's the case, your host may cache this information for a while...

you can disable ICMP redirect messages generation with the no ip redirect interface command.

More info: http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094702.shtml

HTH

Laurent.

dal · ‎05-20-2009

Thank you for your reply.

Yes, the source will always be on the same subnet as my router and ISP Gateway. Is it better if it comes from a third subnet?

when turning on debug, and try a traceroute, I get these messages:

*May 20 16:58:54.318: ICMP: time exceeded (time to live) sent to 158.38.16.142 (dest was 195.139.129.130)

Nothing else is seen, except the same info over and again from the 2 trackers:

*May 20 17:19:13.123: ICMP: echo reply rcvd, src 213.184.200.1, dst 213.184.198.137

*May 20 17:19:13.127: ICMP: echo reply rcvd, src 158.38.0.168, dst 158.38.16.140

(The last one is not seen of course, when I shut down that gateway)

It seems that something is looping somewhere, but why?

Thank you.

bryantmarsh · ‎05-20-2009

I ran into the same problem. The next-hop in the route-map never worked for me either.

I eventually had to do the following:

ip sla monitor 1

type echo protocol ipIcmpEcho XXX.XXX.XXX.XXX source-interface GigabitEthernet0/1

frequency 15

ip sla monitor schedule 1 life forever start-time now

ip sla monitor 2

type echo protocol ipIcmpEcho 192.168.28.3 source-interface Serial1/0

frequency 15

ip sla monitor schedule 2 life forever start-time now

track 101 rtr 1 reachability

delay down 60 up 10

!

track 102 rtr 2 reachability

interface GigabitEthernet0/0

ip address 10.0.0.254 255.255.255.0

no ip redirects

no ip unreachables

ip flow ingress

ip policy route-map SETGATEWAY

duplex full

speed 1000

media-type rj45

ip route 0.0.0.0 0.0.0.0 192.168.29.3 track 101

ip route 0.0.0.0 0.0.0.0 192.168.30.1 254 track 102

access-list 2001 permit ip 192.168.0.0 0.0.255.255 any log-input

access-list 2001 permit ip 172.16.0.0 0.0.255.255 any log-input

route-map SETGATEWAY permit 10

match ip address 2001

set ip default next-hop 192.168.29.3

dal · ‎05-21-2009

bryantmarsh, thank you for your answer.

I have tried your config now, but it still don't work :(

Laurent Aubert · ‎05-20-2009

Hi,

What you see is expected, Traceroute increment the TTL starting with one up to the destination is reach to discover each host on the path.

You don't have to do anything except waiting to see if ICMP redirect are generated by the router.

The cache timeout due to ICMP redirect could be long (10mn by default on windows server). So if it's the root cause, you should see an ICMP redirect from the debug icmp each time the host cache timeout.

You should check your host to see if there is any default route pointing directly to 158.38.16.129 or disable ICMP redirect on the router and see if it resolves your issue.

From a design perspective, I agree that this host should be on another link. In this case apply the route-map on this new interface. It's safer to have your public host behind a router so you can better protect it.

Actually, you don't need to specify the 2nd ip next-hop in your route-map. If 158.38.16.129 is unavailable, the routing table will be used and we know the installed default route already points to the other ISP.

HTH

Laurent.

dal · ‎05-22-2009

Hello again.

I cannot see any ICMP Redirects in the debug, no matter how long it stays on.

I really don't think this has something do to with the host itself.

There are noe default routes pointing to 158.38.16.129 on the host either.

I have also tried to disable ICMP redirects, but no difference :(

I'm running out of options here :(

About the design;

The intention was to put this router in front of our existing firewall, so we could use the existing rules from it.

And the traffic coming through the Firewall is already NAT'ed.

The plan was to use 2 NAT ip addresses from the firewall to the router to seperate the traffic coming from the inside:

Example:

Traffic from 10.20.x.x will be seen as 213.184.198.150 on the outside.

Traffic from 10.30.x.x will be seen as 158.38.16.150 on the outside.

Since the firewall is not capable of PBR, I was hoping to do it with my Cisco Router, as explained above.

We are really close now, but no cigar :)

One thing, though:

Even when both ISP's are up, and the traffic is flowing as intended, I still get this messages from the debug ip icmp:

*May 22 08:09:13.622: ICMP: time exceeded (time to live) sent to 158.38.16.142 (dest was 195.139.129.130)

Isn't that strange?

Thanks.

dal · ‎05-22-2009

Hello again.

A little update on this:

It seems that you were right.

After some more testing, it looks that the failover works perfectly fine, but ONLY on ip addresses it does not know about from before.

IP Adresses it DOES know about, starts working again after a time (30 mins?)

So obviously there is a cache somewhere. But where? Not on the remote servers or anywhere on the remote side I hope, then we have a big problem :)

And how to bypass it?

Thank you.

Laurent Aubert · ‎05-22-2009

Hi,

You need to check the documentation of the OS of your server to see how it handles ICMP redirect messages.

HTH

Laurent.

dal · ‎05-25-2009

Hello again,

It is working now.. sometimes. And sometimes it don't

I don't really get it.

But you are saying that I need to disable ICMP redirect on whatever host I choose to test this with?

And I don't have to disable ICMP redirects on the router itself where this configuration is taken from?

So if I choose to test this with another Cisco router (or L3 Switch), this should be no problem?

Thanks again,

- Ã˜ystein

Laurent Aubert · ‎05-25-2009

Hi,

Only routers generate ICMP redirect messages and Host must know how to process them (part of the ICMP stack).

In your case, disable ICMP redirect on the router interface pointing to your host. Wait enough time to be sure your host cache information timeout (or clear it if you know how)and repeat your test. Your host should be able to reach any Internet sites.

HTH

Laurent.

dal · ‎05-26-2009

Hi.

Traffic from the interface where I have to use route-map to route it through the right interface works very well now.

But traffic from the interface that has the default route originally will not fail over when the default route changes.

I use another Cisco 1841 to test this with now, and here are the two configurations:

The router I'm testing from:

-------

!

ip name-server 213.184.198.11

ip name-server 213.184.200.1

ip name-server 158.38.0.168

!

interface FastEthernet0/0

ip address 158.38.16.150 255.255.255.128

no ip redirects

no ip route-cache

duplex auto

speed auto

no mop enabled

!

interface FastEthernet0/1

ip address 213.184.198.88 255.255.255.0

no ip redirects

no ip route-cache

duplex auto

speed auto

!

ip default-gateway 213.184.198.137

----------------------------

This is the Internet Gateway as it stands right now:

-----------------------------

hostname InternettGW

ip name-server 213.184.198.11

ip name-server 158.38.0.168

ip sla 1

icmp-echo 158.38.0.168 source-interface FastEthernet0/0

frequency 5

ip sla schedule 1 life forever start-time now

ip sla 2

icmp-echo 213.184.200.1 source-interface FastEthernet0/1

frequency 5

ip sla schedule 2 life forever start-time now

!

track 1 rtr 1 reachability

delay down 5 up 5

!

track 2 rtr 2 reachability

delay down 5 up 5

!

interface FastEthernet0/0

ip address 158.38.16.143 255.255.255.128

no ip redirects

duplex auto

speed auto

!

interface FastEthernet0/1

ip address 213.184.198.137 255.255.255.0

no ip redirects

ip policy route-map test

duplex auto

speed auto

!

interface FastEthernet0/0/0

switchport mode trunk

!

interface Vlan53

description * Administration Interface Only! *

ip address 10.50.53.10 255.255.255.0

!

ip route 0.0.0.0 0.0.0.0 158.38.16.129 101 track 1

ip route 0.0.0.0 0.0.0.0 213.184.198.1 100 track 2

ip route 10.0.0.0 255.0.0.0 10.50.53.1

ip route 158.38.0.168 255.255.255.255 158.38.16.129

ip route 213.184.200.1 255.255.255.255 213.184.198.1

!

access-list 10 permit 158.38.16.142

access-list 10 permit 158.38.16.150

!

route-map test permit 10

match ip address 10

set ip next-hop verify-availability 158.38.16.129 10 track 1

!

----------------------------------

Any tips on how to complete this final stage?

Thanks again,

- Ã˜ystein

Laurent Aubert · ‎05-26-2009

The route-map shouldn't be on F0/0 instead of 0/1 ?

Laurent.

dal · ‎05-26-2009

No I think not, since the default gateway on the test router is 213.184.198.137.

Therefore, it is F0/1 that will recieve all packets.