Problem with Static routes when using VRF-Lite

kbystrak1 · ‎05-20-2021

I have an issue with spokes on a DMVPN. The spokes are setup using VRF-Lite to route traffic for the DMVPN out the local ISP while also routing traffic from the local LAN back across the DMVPN to a core router. The problem i am having is that i can't get any static routes to work for the traffic coming in on the local LAN connection. I want to route traffic for a hosted PBX phone system out the local internet connection at the site and NOT back across the DMVPN like everything else. I have included the relevant parts of the config below. I have removed some parts to protect the customers information. Have also replaced the customers real IP with a generic 75.75.75.75 in the config.

no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
!
boot-start-marker
boot-end-marker
!
!
vrf definition ISP1
rd 1:1
!
address-family ipv4
exit-address-family
!
logging buffered 4096
!
aaa new-model
!
aaa authentication login default local
!
aaa session-id common
!
!
no ip bootp server
no ip domain lookup
ip inspect name DEFAULT100 tcp
ip inspect name DEFAULT100 udp
ip ips notify SDEE
ip ips name sdm_ips_rule
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
password encryption aes
cts logging verbose
!
redundancy
!
track 1 ip sla 1 reachability
!
track 11 ip sla 11 reachability
!
track 111 list boolean or
object 1
object 11
!
ip tcp synwait-time 10
ip ssh authentication-retries 2
ip ssh rsa keypair-name sshkeys
ip ssh logging events
ip ssh version 2
!

!
crypto keyring ISP1 vrf ISP1
pre-shared-key address 0.0.0.0 0.0.0.0 key --REMOVED--
!
crypto isakmp policy 1
encr 3des
authentication pre-share
group 2
!
!
crypto ipsec transform-set ESP-3DES-SHA1 esp-3des esp-sha-hmac
mode transport
!
crypto ipsec profile Profile1
set transform-set ESP-3DES-SHA1
!

!
interface Tunnel11
description DMVPN Tunnel
bandwidth 6000
ip address 172.16.11.25 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication --REMOVED--
ip nhrp map multicast --REMOVED--
ip nhrp map 172.16.11.10 --REMOVED--
ip nhrp network-id --REMOVED--
ip nhrp holdtime 360
ip nhrp nhs 172.16.11.10
ip nhrp registration no-unique
ip nhrp shortcut
ip nhrp redirect
ip load-sharing per-packet
ip tcp adjust-mss 1360
delay 1000
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel key --REMOVED--
tunnel vrf ISP1
tunnel protection ipsec profile Profile1
!
!
interface GigabitEthernet0/0
description Connected to LAN
ip address 10.25.20.1 255.255.0.0
ip access-group 100 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip flow ingress
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
no mop enabled
!
interface GigabitEthernet0/1
description Comcast Internet (Static)
vrf forwarding ISP1
ip address 75.75.75.75 255.255.255.252
ip access-group 102 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip inspect DEFAULT100 out
ip ips sdm_ips_rule in
ip ips sdm_ips_rule out
ip virtual-reassembly in
duplex auto
speed auto
no cdp enable
no mop enabled
!
!
router eigrp 1
network 10.0.0.0
network 172.16.0.0
!
ip forward-protocol nd
!
!
ip nat inside source route-map COMCAST interface GigabitEthernet0/1 overload
ip route 0.0.0.0 0.0.0.0 10.10.20.1 5 name Route_via_DMVPN
ip route 0.0.0.0 0.0.0.0 75.75.75.76 15 name JUST_IN_CASE
ip route 208.93.135.178 255.255.255.255 75.75.75.76 name Hosted_SIP_PBX track 111
ip route 8.8.8.8 255.255.255.255 10.10.20.2 name Test
ip route vrf ISP1 0.0.0.0 0.0.0.0 75.75.75.76
ip route vrf ISP1 4.2.2.1 255.255.255.255 96.70.190.142 permanent name IP_SLA_1_TRACK_1_COMCAST
ip route vrf ISP1 208.67.220.220 255.255.255.255 96.70.190.142 permanent name IP_SLA_11_TRACK_11_COMCAST
!
!
ip sla auto discovery
ip sla 1
icmp-echo 4.2.2.1
threshold 500
timeout 500
frequency 5
ip sla schedule 1 life forever start-time now
ip sla 11
icmp-echo 208.67.220.220
threshold 500
timeout 500
frequency 5
ip sla schedule 11 life forever start-time now
!
!
route-map COMCAST permit 10
match ip address 104
match interface GigabitEthernet0/1
!
!
access-list 100 deny ip host 255.255.255.255 any
access-list 100 deny ip 127.0.0.0 0.255.255.255 any
access-list 100 permit ip any any
access-list 102 remark COMCAST ACL for GI0/1
access-list 102 permit udp any any eq non500-isakmp
access-list 102 permit udp any any eq isakmp
access-list 102 permit esp any any
access-list 102 permit ahp any any
access-list 102 permit gre any any
access-list 102 permit icmp any any echo-reply
access-list 102 permit icmp any any time-exceeded
access-list 102 permit icmp any any unreachable
access-list 102 deny ip 10.0.0.0 0.255.255.255 any
access-list 102 deny ip 172.16.0.0 0.15.255.255 any
access-list 102 deny ip 192.168.0.0 0.0.255.255 any
access-list 102 deny ip 127.0.0.0 0.255.255.255 any
access-list 102 deny ip host 255.255.255.255 any
access-list 102 deny ip host 0.0.0.0 any
access-list 102 deny ip any any log
access-list 104 remark NAT Access List for route maps
access-list 104 permit ip 10.25.0.0 0.0.255.255 any

First, we can confirm that the track 111 is up

2921#show track
Track 1
  IP SLA 1 reachability
  Reachability is Up
    6624 changes, last change 01:21:41
  Latest operation return code: OK
  Latest RTT (millisecs) 28
  Tracked by:
    Track List 111
Track 11
  IP SLA 11 reachability
  Reachability is Up
    6540 changes, last change 01:21:41
  Latest operation return code: OK
  Latest RTT (millisecs) 36
  Tracked by:
    Track List 111
Track 111
  List boolean or
  Boolean OR is Up
    5214 changes, last change 01:21:41
    object 1 Up
    object 11 Up
  Tracked by:
    Static IP Routing 0

Then we can check the global routing table:

2921#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override

Gateway of last resort is 10.10.20.1 to network 0.0.0.0

S* 0.0.0.0/0 [5/0] via 10.10.20.1
8.0.0.0/32 is subnetted, 1 subnets
S 8.8.8.8 [1/0] via 10.10.20.2
10.0.0.0/8 is variably subnetted, 28 subnets, 3 masks
--REMOVED--
172.16.0.0/16 is variably subnetted, 9 subnets, 2 masks
--REMOVED--
2921#

Notice that the route to 208.93.135.178 is not anywhere in the routing table but the route to 8.8.8.8 is

Its not in the VRF routing table either:

2921#show ip route vrf ISP1

Routing Table: ISP1
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override

Gateway of last resort is 75.75.75.76 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 75.75.75.76
4.0.0.0/32 is subnetted, 1 subnets
S 4.2.2.1 [1/0] via 75.75.75.76
--REMOVED--
208.67.220.0/32 is subnetted, 1 subnets
S 208.67.220.220 [1/0] via 75.75.75.76
2921#

From a PC connected you can see the route taken is over the DMVPN tunnel:

C:\Users\User>tracert -d 208.93.135.178

Tracing route to 208.93.135.178 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 10.25.20.1
2 8 ms 8 ms 8 ms 172.16.11.10
3 9 ms 9 ms 8 ms 10.10.20.2
4 11 ms 11 ms 11 ms --REMOVED--
5 24 ms 20 ms 18 ms --REMOVED--
6 18 ms 19 ms 19 ms --REMOVED--
7 19 ms 17 ms 18 ms --REMOVED--
8 25 ms 21 ms 23 ms 96.110.92.17
9 21 ms 24 ms 22 ms 68.85.133.137
10 24 ms 23 ms 23 ms 96.110.40.17
11 23 ms 28 ms 23 ms 96.110.32.186
12 27 ms 26 ms 26 ms 192.205.37.41

etc... it eventually gets there but over the DMVPN which is not what we want.

So we can see the router is using its Gateway of last resort of 10.10.20.1 from the Global routing table to route traffic for the 208.93.135.178 IP despite the fact there is a static route for it in the configuration.

So how do I get the static route to show up in the global routing table and work?

Giuseppe Larosa · ‎05-22-2021

Hello @kbystrak1 ,

your specific static route is not installed in Global routing table because the specified next-hop is NOT in GRT but it is out an interface in vrf ISP1

Let's review the static route configuration section:

ip route 0.0.0.0 0.0.0.0 10.10.20.1 5 name Route_via_DMVPN
ip route 0.0.0.0 0.0.0.0 75.75.75.76 15 name JUST_IN_CASE
ip route 208.93.135.178 255.255.255.255 75.75.75.76 name Hosted_SIP_PBX track 111
ip route 8.8.8.8 255.255.255.255 10.10.20.2 name Test
ip route vrf ISP1 0.0.0.0 0.0.0.0 75.75.75.76
ip route vrf ISP1 4.2.2.1 255.255.255.255 96.70.190.142 permanent name IP_SLA_1_TRACK_1_COMCAST
ip route vrf ISP1 208.67.220.220 255.255.255.255 96.70.190.142 permanent name IP_SLA_11_TRACK_11_COMCAST

If I have understood your post the static route of interest is:

ip route 208.93.135.178 255.255.255.255 75.75.75.76 name Hosted_SIP_PBX track 111

what if you change it in:

ip route vrf ISP1 208.93.135.178 255.255.255.255 75.75.75.76 name Hosted_SIP_PBX track 111

In this way the static route should be installed in vrf ISP1.

However, you would need it in GRT because your LAN netwotk and the the DMVPN are in GRT.

You need also an appropriate NAT for packets to be sent to the SIP PBX.

Possible solutions are :

create a new subnet on the LAN side on vrf ISP1 for the SIP phones . This is usually considered good design as it separates VOIP endpoints from PCs.

However, if you are using soft phones on PCs this is not possible

Find a way to leakage traffic from GRT to vrf ISP1.

Example :

a link between two unused ports on the router with one side in GRT and the other side in vrf ISP1 can make the trick.

The two should use a common IP subnet and it should have ip nat outside on the GRT side.

At that point the new static route in GRT would use that next-hop in the shared subnet.

Hope to help

Giuseppe

kbystrak1 · ‎05-25-2021

Unfortunately, there are a lot of sites using this config on this network and creating new VLANS for all of them or using spare interfaces is not going to be possible. There is also about 20% of the users using softphones and 80% using IP sets. So the separate VLAN wouldn't work for that reason also.

I did try adding the static route to to the VRF like you suggested. It didn't may any difference.

I did find this article on leaking routes between the GRT and and VRF. It mentions at the beginning that it is easy to do using static statements where the next hop is known. But it doesn't say how. It just says it is easy. https://www.cisco.com/c/en/us/support/docs/ip/ip-routing/200158-Configure-Route-Leaking-between-Global-a.html

I also tried putting this into the config:

ip route 208.93.135.178 255.255.255.255 GigabitEthernet0/1 75.75.75.76

and removing the original statement.

Doing so generated some helpful progress:

2921#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is 10.10.20.1 to network 0.0.0.0

S*    0.0.0.0/0 [5/0] via 10.10.20.1
      8.0.0.0/32 is subnetted, 1 subnets
S        8.8.8.8 [1/0] via 10.10.20.2
      10.0.0.0/8 is variably subnetted, 28 subnets, 3 masks
         --REMOVED--
      208.0.0.0/32 is subnetted, 1 subnets
S        208.93.135.178 [1/0] via 75.75.75.76, GigabitEthernet0/1
      172.16.0.0/16 is variably subnetted, 9 subnets, 2 masks
         --REMOVED--

The route is now in the GRT. A traceroute from the client PC shows this:

C:\Users\user>tracert -d 208.93.135.178

Tracing route to 208.93.135.178 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  10.25.20.1
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12     *        *        *     Request timed out.
 13     *        *        *     Request timed out.
 14     *        *        *     Request timed out.
 15     *        *        *     Request timed out.
 16     *        *        *     Request timed out.
 17     *        *        *     Request timed out.
 18     *        *        *     Request timed out.
 19     *        *        *     Request timed out.
 20     *        *        *     Request timed out.
 21     *        *        *     Request timed out.
 22     *        *        *     Request timed out.
 23     *        *        *     Request timed out.
 24     *        *        *     Request timed out.
 25     *        *        *     Request timed out.
 26     *        *        *     Request timed out.
 27     *        *        *     Request timed out.
 28     *        *        *     Request timed out.
 29     *        *        *     Request timed out.
 30     *        *        *     Request timed out.

Trace complete.

This makes me wonder if the issue with the route is correct but there is now a NAT issue.

I started a ping from the PC and then checked the NAT

2921#show ip nat trans
Pro Inside global      Inside local       Outside local      Outside global
icmp 75.75.75.75:1   10.25.60.15:1      208.93.135.178:1      208.93.135.178:1

It looks like NAT is trying. But something else is wrong somewhere because the ping fails as does other traffic to the IP.

james.brunner · ‎05-26-2021

I have this exact setup for 100+ offices using local internet breakout and DMVPN services.

You have to use some route-maps and force traffic in the routes - crazy as it sounds...

So, you have your "ip route vrf ISPA 0.0.0.0 0.0.0.0 N1.N2.N3.N4" so the vrf has a default route, and you have a global default route - but we tweak this to "ip route 0.0.0.0 0.0.0.0 <ISP interface> <next_hop_ip> track <blah>" to ensure it leaves via the ISP interface.

Then, you'd normally have an ACL for inside networks to use the NAT like "access-list standard NAT_ACL" and "permit 10.0.0.0 0.0.0.255"

Now create an route-map like "route-map NAT_RM_ISPA permit 10" and "match ip address NAT_ACL" and "match interface <ISP interface>" in your case G0/1.

Then on your ip nat statement, change it to something like "ip nat inside source route-map NAT_RM_ISPA interface g0/1 overload"

In summary (yeah that isn't a neat reply!) and kinda tweaked to match your config:

ip route vrf ISPA 0.0.0.0 0.0.0.0 n.n.n.n
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/1 n.n.n.n track <blah>
ip access-list standard NAT_ACL
 permit 10.0.0.0 0.0.0.255
route-map NAT_RM_ISPA permit 10
 match ip address NAT_ACL
 match interface GigabitEthernet0/1
ip nat inside source route-map NAT_RM_ISPA interface GigabitEthernet0/1 overload

Oh and you can even run zone-based firewalling on the internet access too.

[UPDATE: Now I'm no longer on my phone I can see that your config matches the above except for the addition of the exit port name in the global default route. Sorry if I came across badly when your config was nearly identical! lol]

Hope this helps!

JB.

Giuseppe Larosa · ‎05-26-2021

Hello @james.brunner ,

good remarks on using route-map with match interface that checks the exit interface with NAT.

The only change I think our Original Poster needs is to use an extended ACL so that its NAT is triggered only when talking with the SIP PABX public IP

ip access-list extended NAT-to-SIPBOX

permit ip 10.10.20 0.0.0.255 host 208.93.135.178

route-map NAT_RM_ISPA permit 10
 match ip address NAT-to-SIPBOX
 match interface GigabitEthernet0/1
ip nat inside source route-map NAT_RM_ISPA interface GigabitEthernet0/1 overload

Hope to help

Giuseppe

kbystrak1 · ‎06-01-2021

I decided to give this a try also.

I reduced my ACL for the route-map down to just the IP of the PBX. The result was the same. The router shows the NAT translation taking place but on the PC it is not actually working.

kbystrak1 · ‎06-01-2021

JB,

Thanks for the suggestion. As you said, our configs are nearly identical. The only difference being your global default route has the interface name in it. I don't want all my traffic going out the local internet, just traffic to one IP. So instead of a global route, i am trying to set a specific one.

I did however, decide to give your idea a shot just to see what the effects would be. I added a global default route as your suggested using the interface name and the next hop IP. I left the tracking off for testing purposes.

It produced the same result i am seeing with my single static route. Only this time when I show the IP NAT Translations table it is full of pages of translations that the router thinks it is doing for all the traffic from the PCs and phones onsite. However, i am unable to access the internet from any PC onsite. It is similar to my single static route in that the router appears to be doing the correct thing but is not. I wonder, you didn't mention, do you have IP NAT ENABLE or IP NAT INSIDE/OUTSIDE on your interfaces? I am using IP NAT INSIDE and IP NAT OUTSIDE on my interfaces.

kbystrak1 · ‎06-01-2021

Another thing odd that is worth noting. I saved this config and rebooted the router. Now, with the exact same config as before, the static route to the 208.93.135.178 IP is no longer in the routing table. The route is there if you do a show run and it is present in the config, but the router is ignoring it and not putting it in the routing table after the reboot.