PBR: set ip next-hop verify-availability with track not working

[ob]

I am trying to setup PBR to route traffic from a certain subnet, going to certain IP ranges, over two different next-hops.

source subnet (vlan151, VDI clients): 10.1.1.0/24

destination 1: 172.16.0.0/24 (and others but left out for simplification, the internal network)

destination 2: all other subnets (internet in this case)

The source subnet (vlan151, VDI clients) is in location A.

Destination 1 is in location A and the traffic towards it should stay there and be routed locally.

Destination 2 is internet and should be routed over location B in a different country, connected over MPLS. When location B is unavailable, the traffic should route locally over location A.

I want to use route-maps to achieve this and created the following.

1. A GRE tunnel from the Core switch (C6880-X-LE, c6880x-adventerprisek9-mz.SPA.152-1.SY6.bin) in location A to the core switch (WS-C4500X-32, cat4500e-universalk9.SPA.03.08.06.E.152-4.E6.bin) in location B so I can use the IP on the B side as a next hop.

Location A

!
interface Tunnel1
ip vrf forwarding TEST
ip address 192.168.3.1 255.255.255.248
no ip redirects
tunnel source 192.168.1.1
tunnel destination 192.168.2.1
tunnel vrf TEST
end

Location B

!
interface Tunnel1
ip address 192.168.3.2 255.255.255.248
tunnel source 192.168.2.1
tunnel destination 192.168.1.1
end

The tunnel works fine. I can ping back and forth without issues.

2. Two ACLs to match both types of traffic on the core switch in Location A.

ip access-list extended ACL-Internal
remark to Internal
permit ip 10.1.1.0 0.0.0.255 172.16.0.0 0.0.0.255

ip access-list extended ACL-Internet

remark to Internet
permit ip 10.1.1.0 0.0.0.255 0.0.0.255 any

3. Two Route-maps

route-map RM-PBR permit 20
match ip address ACL-Internal
!
route-map RM-PBR permit 30
match ip address ACL-Internet
set ip next-hop verify-availability 192.168.3.2 100 track 151

I know the first route-map is empty but it seems to work the way I want. I want that when this internal route-map is applied, regular routing is used. I don't know how to do this more elegantly.

4. IP SLA/track config

ip sla 151
icmp-echo 192.168.3.2 source-interface Vlan151
vrf TEST
threshold 300
timeout 500
frequency 10
ip sla schedule 151 life forever start-time now

!
track 151 ip sla 151 reachability
delay down 15 up 15
!

This IP SLA/track works like I expect it to work. I also see it as [up] when I do a "show route-map"

5. Applied the routemap to the vlan interface.

!
interface Vlan151
ip vrf forwarding TEST
ip address 10.1.1.1 255.255.255.0
ip helper-address x.x.x.x
ip helper-address y.y.y.y
no ip proxy-arp
ip policy route-map RM-PBR
end

----------------------------------------------

If I now focus on just the "RM-PBR permit 30" (Internet) route-map. I just don't understand the behaviour it shows me. It does not work. When I browse the internet from vlan 151, I go out over the local (location A) firewall. When I traceroute to 8.8.8.8, I see it going over location A.

But when I check the debug, it shows me:

Apr 9 11:50:57.343: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8, len 92, policy match
Apr 9 11:50:57.343: SW1: IP: route map RM-PBR, item 30, permit
Apr 9 11:50:57.343: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8 (Tunnel1), len 92, policy routed
Apr 9 11:50:57.343: SW1: IP: Vlan151 to Tunnel1 192.168.3.2
Apr 9 11:50:57.351: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8, len 92, policy match
Apr 9 11:50:57.351: SW1: IP: route map RM-PBR, item 30, permit
Apr 9 11:50:57.351: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8 (Tunnel1), len 92, policy routed

Apr 9 11:50:57.351: SW1: IP: Vlan151 to Tunnel1 192.168.3.2
Apr 9 11:50:57.355: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8, len 92, policy match
Apr 9 11:50:57.355: SW1: IP: route map RM-PBR, item 30, permit
Apr 9 11:50:57.355: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8 (Tunnel1), len 92, policy routed
Apr 9 11:50:57.355: SW1: IP: Vlan151 to Tunnel1 192.168.3.2

So it does seem to match and I also think it tells me it will route over location B.. But it definitely does not. When I check whatsmyip, it shows me the Location A internet IP. What is also strange is that I also tried to browse to some other websites but I don't see that in the in the debug at all.. 

But then, when I change the route-map config of "RM-PBR permit 30" into (extra line at the bottom):

route-map RM-PBR permit 30
match ip address ACL-Internet
set ip next-hop verify-availability 192.168.3.2 100 track 151
set ip next-hop 192.168.3.2

...the whole setup seems to work fine. The whatsmyip also shows me the location B internet IP. All internal traffic is still nicely routed on location A.

debug:

Internal route-map, no change but why don't I see the route-map it tries to use?:
Apr 9 12:07:13.314: SW1: IP: s=10.1.1.9 (Vlan151), d=172.16.234.222, len 62, FIB policy match
Apr 9 12:07:13.314: SW1: IP: s=10.1.1.9 (Vlan151), d=172.16.234.222, len 62, PBR Counted
Apr 9 12:07:13.314: SW1: IP: s=10.1.1.9 (Vlan151), d=172.16.234.222, len 62, FIB policy rejected - normal forwarding

Internet route-map, this is what I expect but I see no difference with the previous debug:
Apr 9 12:07:14.594: SW1: IP: s=10.1.1.9(Vlan151), d=8.8.8.8, len 92, policy match
Apr 9 12:07:14.594: SW1: IP: route map RM-PBR, item 30, permit
Apr 9 12:07:14.594: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8 (Tunnel1), len 92, policy routed
Apr 9 12:07:14.594: SW1: IP: Vlan151 to Tunnel1 192.168.3.2
Apr 9 12:07:14.598: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8, len 92, policy match
Apr 9 12:07:14.598: SW1: IP: route map RM-PBR, item 30, permit
Apr 9 12:07:14.598: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8 (Tunnel1), len 92, policy routed
Apr 9 12:07:14.598: SW1: IP: Vlan151 to Tunnel1 192.168.3.2
Apr 9 12:07:14.598: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8, len 92, policy match
Apr 9 12:07:14.598: SW1: IP: route map RM-PBR, item 30, permit
Apr 9 12:07:14.598: SW1: IP: s=10.1.1.9 (Vlan151), d=8.8.8.8 (Tunnel1), len 92, policy routed
Apr 9 12:07:14.598: SW1: IP: Vlan151 to Tunnel1 192.168.3.2

Internet route-map, but to some other website (it works and goes over location B):
Apr 9 12:07:11.586: SW1: IP: s=10.1.1.9 (Vlan151), d=176.34.99.51, len 1500, FIB policy match
Apr 9 12:07:11.586: SW1: IP: s=10.1.1.9 (Vlan151), d=176.34.99.51, len 1500, PBR Counted
Apr 9 12:07:11.586: SW1: IP: s=10.1.1.9 (Vlan151), d=176.34.99.51, g=192.168.3.2, len 1500, FIB policy routed
Apr 9 12:07:11.586: SW1: IP: s=10.1.1.9 (Vlan151), d=176.34.99.51, len 1500, policy match
Apr 9 12:07:11.586: SW1: IP: route map RM-PBR, item 30, permit
Apr 9 12:07:11.586: SW1: IP: s=10.1.1.9 (Vlan151), d=176.34.99.51 (Tunnel1), len 1500, policy routed
Apr 9 12:07:11.586: SW1: IP: Vlan151 to Tunnel1 192.168.3.2

Why don't I see the 3rd rule (... FIB policy routed) in the traffic to 8.8.8.8 (which was a traceroute)

So, this works but now I don't have a fallback when the tunnel is down and my internet traffic gets blackholed.

Does anyone see what I am doing wrong? Why is the "set ip next-hop verify-availability 192.168.3.2 100 track 151" not doing it's thing? It is driving me crazy! The fact that this setup works makes me think that at least the ACLs and the IP SLA/track are all configured correctly. 

[Giuseppe Larosa]

Hello ob,

as far as I remember you need both commands in the route-map clause:

>>

route-map RM-PBR permit 30
match ip address ACL-Internet
set ip next-hop verify-availability 192.168.3.2 100 track 151
>>set ip next-hop 192.168.3.2

the set ip next-hop verify-avalaibility just adds the check on the next hop but the true set action is performed by the second command

Have you checked what happens when the next-hop is not available with both commands in the route-map clause?

Edit:

I see in some example that this new version of the set ip next-hop verify-availabilty using IP SLA should be able to work alone see

https://www.cisco.com/c/en/us/support/docs/ip/ip-routed-protocols/48003-pbrtracking.html?dtid=osscdc000283

So now I understand your concerns.

Hope to help

Giuseppe

[ob]

Hi Giuseppe,

Thanks for the reply.

What I get from all the documentation I've read is that you only have to go that way if rely on CDP (instead of track) for the verify availabilty. So you would get:

route-map RM-PBR permit 30
match ip address ACL-Internet
set ip next-hop verify-availability
set ip next-hop 192.168.3.2

When you use a track, I seems it does not work that way and just the one line should be enough. I cannot use the above setup because 192.168.3.2 is no CDP neighbor.

But I did test your config and the routing does work like that but the verify is not. So when I shut the tunnel, all traffic that follows this route-map just black holes so there is no fallback like this.

[Giuseppe Larosa]

Hello Ob,

you are right this new version of the command should be able to work alone.

I'm sorry for the misunderstanding . I rated your correction accordingly

Best regards

Giuseppe

[Giuseppe Larosa]

Hello Ob,

you are using a VRF: the SVI, the GRE Tunnel and the IP SLA are all in the context of VRF TEST.

I just wonder if this can be the reason for your issues. I would suggest if possible for you to make new tests using an SVI, a GRE tunnel and IP SLAs that are not in a VRF but in the global routing table just to see if in this case the behaviour changes.

Consider this a suggestion for further troubleshooting

Hope to help

Giuseppe

[ob]

I think I will give this a try in a new test setup but I don't really see how the VRF's could have an impact on this because it does work without the verify-availability. Also on the location B side, there is no VRF (only the default) on the switch and as far as I know VRF's are only locally significant so as long as I do everything in VRF TEST on the A side, it should not matter what VRF the B side has (A & B always are different vrf's, even if I would use VRF TEST on both sides).

[Giuseppe Larosa]

Hello Ob,

my suggestion to remove the VRF is to build a simplified setup that allows you to test the functionality without the added complexity of VRF just to see if anything changes.

Besides this, in the command reference I have seen another command

set ip vrf <vrf-name> next-hop <IP-address>

but this command looks like to have no option for verify-availability.

Probably the use case of this command is for a different scenario then yours, however it shows that some interactions with VRF in PBR is present.

Hope to help

Giuseppe

[Richard Burts]

Am I understanding the discussion that modifying the route map and removing the verify-availability makes the route map and PBR work correctly? This certainly suggests that the issue is not related to vrf but suggests that this platform and this version of code may not support the verify-availability function. What is this platform and version of code? And are we dealing with real network equipment or is this on some simulator?

HTH

Rick

[ob]

You are correct. The route-map and PBR are working totaly fine and as expected when not using the "verify-availabity" function. That setup is just not what we want because it has no fallback when the next-hop is unreachable.

We are working with real hardware here.
Site A: C6880-X-LE, c6880x-adventerprisek9-mz.SPA.152-1.SY6.bin
Site B: WS-C4500X-32, cat4500e-universalk9.SPA.03.08.06.E.152-4.E6.bin

[Richard Burts · Accepted solution]

Thank you for verifying that this is on real equipment and that the PBR is configured on 6880. I have looked and can not find a definitive statement about whether the 6880 supports verify-availability in PBR. But I think it is significant that when I look in the config guide that it does not mention verify-availability as an option. If you have a support contract with Cisco it would be good to open a case with Cisco TAC and have them give you an answer on this. I suspect it is a non supported feature on this platform and version of code.

HTH

Rick

[ob]

Sadly, it seems like this is the case ...I have not gotten it officially confirmed by TAC but other network engineers did come to the same conclusion. Pretty strange that a switch like a 6880 can't handle a command like this...

I resorted to EEM to resolve my problem, which works perfectly. I just have EEM remove the PBR from the vlan interface when track 151 reports down and have it re-applied when track 151 reports up.

[Richard Burts]

Thanks for the update confirming that my suggestion about it being a non supported feature seems correct and that you have developed a work around that does provide the functionality that you need. So +5 for that. Thank you for marking this question as solved. This will help other participants in the community to identify discussions which have helpful information. And this discussion certainly does have helpful information. The important lesson here is that we need to be careful on platforms that are newer that they may not have all of the features implemented that are available on more mature platforms.

HTH

Rick

[Richard Burts]

There are a couple of things I would like to comment about:

- the access list that you post is not correct syntax

ip access-list extended ACL-Internet

remark to Internet
permit ip 10.1.1.0 0.0.0.255 0.0.0.255 any

you should remove the extra 0.0.0.255

- you do not need two route map instances. There is a simple solution that will allow you to use a single route map

ip access-list extended ACL-Internet

deny ip 10.1.1.0 0.0.0.255 172.16.0.0 0.0.0.255

remark to Internet
permit ip 10.1.1.0 0.0.0.255 any

with this a single entry in the route map will not policy route your local traffic and will policy route traffic from your LAN to the remote LAN.

- I am not sure why the verify-availability is not working. Can you verify that the tracking IP SLA is working correctly?

HTH

Rick

[ob]

Hi Richard,

The extra 0.0.0.255 in the access-list was a typo in my post, made when changing my real IP addresses. The config is correct in real life :).

I did what you said an made one ACL and one route-map. It all gives me the same results as using 2 acl's and "2" route-maps..

So I now have tried:

ip access-list extended ACL-pbr
deny ip 10.1.1.0 0.0.0.255 172.16.0.0 0.0.0.255

permit ip 10.1.1.0 0.0.0.255 any

!

route-map RM-PBR permit 30
match ip address ACL-pbr
set ip next-hop verify-availability 192.168.3.2 100 track 151

Above config is not working. When I do a "show track" it shows UP. When I do a "show route-map", it also says [UP]. And when I shut the tunnel, it shows [Down] with both commands. But it still just throws me in the regular routing table and ignores the next-hop.

What I find very strange is, that with above config I see traffic hitting on the ACL when I do a tracert from my test machine to 8.8.8.8. (or some other internet IP) BUT when I use a browser and go to some websites ... no hits on the ACL whatsoever.. How is that possible??

What makes it even more strange is that when I use the following route-map config (no other changes):

route-map RM-PBR permit 30
match ip address ACL-pbr
set ip next-hop 192.168.3.2

... I do see webbrowsing hitting on the ACL. (with this config, the PBR works like I want except there is no verify and traffic is blackholed when i shut the tunnel :( )

How can the route-map config have an effect on the traffic hitting the ACL yes or no?

I also tried to use CDP for the verification by enabling CDP on both ends of the tunnel. CDP seems to work as the two switches become CDP neighbors when I check "show cdp neighbors".

!

route-map RM-PBR permit 30
match ip address ACL-pbr

set ip next-hop 192.168.3.2
set ip next-hop verify-availability

But this has the exact same effect als just using "set ip next-hop 192.168.3.2". So the PBR works but the verify does nothing.

[paul driver]

Hello

i agree with @Richard Burts you should really only require 1 route-map but I would suggest ultilise 2 stanzas denying vlan 151 traffic towards 172.xxxx

Example -

route-map RM-PBR deny 10
match ip address ACL-Internal

route-map RM-PBR permit 99
set ip next-hop verify-availability 192.168.3.2 100 track 151