Load balancing two ISP's navigation traffic

[luchonat1]

Hello! I'm preparing a setup where I have a 1921 ISR router and I need to use two different ISP's to provide internet access (dynamically nated) to the internal LAN.

If I got it right, just by having two default routes, the router will be doing load balancing, sending packets over both links. So I have a couple of questions:

How CEF will work in this scenario?

How PfR will work in this scenario?

And I also have an issue that I cannot solve:

I have the default routes of both ISP's added dynamically by tracking objects. So for testing purposes having only one ISP working I start to ping from a host in the LAN to a host in the WAN. When the second ISP comes up the ping starts to fail! I think that the problem is that packets with the IP address associated with ISP1 are being sent through ISP2.

Something similar happen when pinging from internet to one of the router's interface, and also when connecting to an EasyVPN server configured in the same router.

So how can I "force" that packets with source address of each ISP be sent through the correct interface?

[Georg Pauwen]

Hello,

can you post the config you have so far ? In general, route maps are used to determine which traffic goes where.

Your 'ping' problem might be related to NAT, which WAN hosts are you pinging ? Try pinging something like 8.8.8.8 or 8.8.4.4, these are the Google name servers and reachable through all ISPs...

[luchonat1]

Sure! thanks in advance!

version 15.4 
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname snlc-r01-r00-r6a
!
boot-start-marker
boot-end-marker
!
!
logging buffered 51200 warnings
!
aaa new-model
!
!
aaa authentication login ciscocp_vpn_xauth_ml_1 local
aaa authorization network ciscocp_vpn_group_ml_1 local
!
!
!
!
!
aaa session-id common
clock timezone GMT -3 0
!
!
!
!
!
!
!
!
!
!
!
!
!
!

!
ip dhcp excluded-address 10.10.10.1
ip dhcp excluded-address 172.16.100.1 172.16.100.220
ip dhcp excluded-address 192.168.24.1 192.168.24.19
ip dhcp excluded-address 192.168.16.1 192.168.16.249
!
ip dhcp pool ccp-pool
import all
network 10.10.10.0 255.255.255.128
default-router 10.10.10.1
lease 0 2
!
ip dhcp pool Private
import all
network 172.16.100.0 255.255.255.0
domain-name senalco.lan
dns-server 172.16.100.1
default-router 172.16.100.1
!
ip dhcp pool Telephony
import all
network 192.168.24.0 255.255.255.0
domain-name tel.senalco.lan
dns-server 192.168.24.1
option 150 ip 192.168.24.2
default-router 192.168.24.1
option 66 ascii 192.168.24.2
!
ip dhcp pool Administrative
import all
network 192.168.16.0 255.255.255.0
domain-name adm.senalco.lan
dns-server 192.168.16.1
default-router 192.168.16.1
!
ip dhcp pool PublicWiFi
import all
network 192.168.28.0 255.255.255.0
domain-name pub.senalco.lan
dns-server 8.8.4.4
default-router 192.168.28.1
!
!
!
ip flow-cache timeout active 1
ip domain name senalco.lan
ip name-server 8.8.8.8
ip name-server 8.8.4.4
ip cef
no ipv6 cef
!
parameter-map type inspect global
log dropped-packets enable
max-incomplete low 18000
max-incomplete high 20000
multilink bundle-name authenticated
!
!
!
key chain PfR_DM
key 1
key-string PfR_DM
cts logging verbose
!
!
pfr master
policy-rules PfR_DM_MAP
!
border 10.255.1.1 key-chain PfR_DM
interface GigabitEthernet0/1 external
interface GigabitEthernet0/0 external
interface Vlan20 internal
interface Vlan21 internal
interface Vlan1 internal
interface Vlan13 internal
interface Vlan12 internal
interface Vlan11 internal
interface Vlan10 internal
!
learn
periodic-interval 1
!
!
pfr border
local Loopback100
master 10.255.1.1 key-chain PfR_DM
license udi pid CISCO1921/K9 sn FJC2131L12U
license boot module c1900 technology-package securityk9
license boot module c1900 technology-package datak9
!
!
username admin privilege 15 secret 5 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
username lnatale privilege 15 secret 5 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
!
redundancy
!
!
!
!
lldp run
track timer interface 5
!
track 1 ip sla 1
delay down 1 up 1
!
track 2 ip sla 2
delay down 1 up 1
!
no ip ftp passive
!
class-map type inspect match-all ccp-cls--1
match access-group name inside-nav
class-map type inspect match-all ccp-cls--3
match access-group name management-nav
class-map type inspect match-all ccp-cls--2
match access-group name wifi-nav
class-map type inspect match-all ccp-cls--4
match access-group name management-to-inside
!
policy-map type inspect ccp-policy-ccp-cls--4
class type inspect ccp-cls--4
inspect
class class-default
drop log
policy-map type inspect ccp-policy-ccp-cls--1
class type inspect ccp-cls--1
inspect
class class-default
drop log
policy-map type inspect ccp-policy-ccp-cls--2
class type inspect ccp-cls--2
inspect
class class-default
drop log
policy-map type inspect ccp-policy-ccp-cls--3
class type inspect ccp-cls--3
inspect
class class-default
drop log
!
zone security outside
zone security inside
zone security public-wifi
zone security management
zone-pair security sdm-zp-inside-outside source inside destination outside
service-policy type inspect ccp-policy-ccp-cls--1
zone-pair security sdm-zp-public-wifi-outside source public-wifi destination outside
service-policy type inspect ccp-policy-ccp-cls--2
zone-pair security sdm-zp-management-outside source management destination outside
service-policy type inspect ccp-policy-ccp-cls--3
zone-pair security sdm-zp-management-inside source management destination inside
service-policy type inspect ccp-policy-ccp-cls--4
!
!
crypto isakmp policy 1
encr 3des
authentication pre-share
group 2
!
crypto isakmp client configuration group management
key XXXXXXXXXXXXXXXXXXXXXXX
pool SDM_POOL_1
acl roamers
netmask 255.255.255.0
crypto isakmp profile ciscocp-ike-profile-1
match identity group management
client authentication list ciscocp_vpn_xauth_ml_1
isakmp authorization list ciscocp_vpn_group_ml_1
client configuration address respond
virtual-template 1
!
!
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
mode tunnel
!
crypto ipsec profile CiscoCP_Profile1
set transform-set ESP-3DES-SHA
set isakmp-profile ciscocp-ike-profile-1
!
!
!
!
!
!
!
interface Loopback0
ip address 192.168.252.1 255.255.255.0
!
interface Loopback100
ip address 10.255.1.1 255.255.255.255
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description FiberCorp$ETH-WAN$
ip dhcp client route track 1
ip address dhcp client-id GigabitEthernet0/0
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip virtual-reassembly in
zone-member security outside
duplex auto
speed auto
!
interface GigabitEthernet0/1
description Telecentro$ETH-WAN$
ip dhcp client route track 2
ip address dhcp client-id GigabitEthernet0/1
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip virtual-reassembly in
zone-member security outside
duplex auto
speed auto
!
interface GigabitEthernet0/0/0
description snlc-s02-roo-r6a/24
switchport mode trunk
no ip address
!
interface GigabitEthernet0/0/1
description snlc-s06-roo-r6a/25
switchport mode trunk
no ip address
!
interface GigabitEthernet0/0/2
no ip address
!
interface GigabitEthernet0/0/3
no ip address
!
interface Virtual-Template1 type tunnel
ip unnumbered Loopback0
zone-member security management
tunnel mode ipsec ipv4
tunnel protection ipsec profile CiscoCP_Profile1
!
interface Vlan1
description Administrative
ip address 192.168.16.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
zone-member security management
!
interface Vlan10
description Servers
ip address 192.168.20.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
zone-member security inside
!
interface Vlan11
description Telephony
ip address 192.168.24.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
zone-member security inside
!
interface Vlan12
description Public WiFi
ip address 192.168.28.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
zone-member security public-wifi
!
interface Vlan13
description Cameras
ip address 192.168.32.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
zone-member security inside
!
interface Vlan20
description Private LAN & WiFi
ip address 172.16.100.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
zone-member security inside
!
interface Vlan21
description Development
ip address 172.16.128.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
zone-member security inside
!
ip local pool SDM_POOL_1 192.168.252.10 192.168.252.254
ip forward-protocol nd
!
ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip dns server
ip nat inside source route-map FiberCorp-nat interface GigabitEthernet0/0 overload
ip nat inside source route-map Telecentro-nat interface GigabitEthernet0/1 overload
!
ip access-list standard secure_vty
permit 10.10.10.0 0.0.0.127
permit 192.168.16.0 0.0.0.255
permit 192.168.252.0 0.0.0.255
remark Secure VTY Access
deny any
!
ip access-list extended inside-nav
remark CCP_ACL Category=128
permit ip 172.16.100.0 0.0.0.255 any
permit ip 172.16.128.0 0.0.0.255 any
permit ip 192.168.20.0 0.0.0.255 any
permit ip 192.168.24.0 0.0.0.255 any
permit ip 192.168.32.0 0.0.0.255 any
ip access-list extended management-nav
remark CCP_ACL Category=128
permit ip 192.168.16.0 0.0.0.255 any
ip access-list extended management-to-inside
remark CCP_ACL Category=128
permit ip 192.168.16.0 0.0.0.255 172.16.100.0 0.0.0.255
permit ip 192.168.16.0 0.0.0.255 172.16.128.0 0.0.0.255
permit ip 192.168.16.0 0.0.0.255 192.168.20.0 0.0.0.255
permit ip 192.168.16.0 0.0.0.255 192.168.24.0 0.0.0.255
permit ip 192.168.16.0 0.0.0.255 192.168.32.0 0.0.0.255
permit ip 192.168.252.0 0.0.0.255 192.168.24.0 0.0.0.255
ip access-list extended navigation-nat
remark CCP_ACL Category=18
permit ip 192.168.16.0 0.0.0.255 any
permit ip 192.168.24.0 0.0.0.255 any
permit ip 192.168.28.0 0.0.0.255 any
permit ip 172.16.100.0 0.0.0.255 any
deny ip any any
ip access-list extended roamers
remark CCP_ACL Category=4
permit ip 192.168.16.0 0.0.0.255 any
permit ip 192.168.24.0 0.0.0.255 any
ip access-list extended wifi-nav
remark CCP_ACL Category=128
permit ip 192.168.28.0 0.0.0.255 any
!
ip sla 1
icmp-echo A.B.C.D source-interface GigabitEthernet0/0 !ISP1 def gw
threshold 40
timeout 1000
frequency 3
ip sla schedule 1 life forever start-time now
ip sla 2
icmp-echo A.B.C.D source-interface GigabitEthernet0/1 !ISP2 def gw
threshold 40
timeout 1000
frequency 3
ip sla schedule 2 life forever start-time now
!
route-map FiberCorp-nat permit 10
match ip address navigation-nat
match interface GigabitEthernet0/0
!
route-map Telecentro-nat permit 10
match ip address navigation-nat
match interface GigabitEthernet0/1
!
!
access-list 23 permit 10.10.10.0 0.0.0.127
access-list 23 permit 192.168.16.0 0.0.0.255
access-list 23 permit 192.168.252.0 0.0.0.255
!
!
!
ipv6 access-list secure_6vty
deny ipv6 any any
!
pfr-map PfR_DM_MAP 200
match pfr learn throughput
set delay relative 30
set mode route control
set mode monitor both
set resolve delay priority 2 variance 20
!
control-plane
!
!
!
line con 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
access-class secure_vty in
privilege level 15
ipv6 access-class secure6_vty in
transport input ssh
line vty 5 15
access-class secure_vty in
privilege level 15
ipv6 access-class secure6_vty in
transport input ssh
!
scheduler allocate 20000 1000
ntp master
ntp update-calendar
ntp server 1.south-america.pool.ntp.org
ntp server 0.ar.pool.ntp.org prefer
!
end

[Georg Pauwen]

Hello,

I am not sure if they are simply omitted, but I cannot see any default routes in the config you have attached. You need:

ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/1

[luchonat1]

The routes are inserted by de DHCP clients with the statements:

ip dhcp client route track 1
ip address dhcp client-id GigabitEthernet0/0

and

ip dhcp client route track 2
ip address dhcp client-id GigabitEthernet0/1

Nevertheless, using static IP configuration (non-DHCP) and inserting routes with

ip route 0.0.0.0 0.0.0.0 A1.B1.C1.D1 track 1 ! (ISP1 dg)
ip route 0.0.0.0 0.0.0.0 A2.B2.C2.D2 track 2 ! (ISP2 dg)

has been tested and symptoms are the same.

[Georg Pauwen]

Hello,

try to clear all NAT translations manually when the second ISP comes up:

clear ip nat translation forced

Which destination host are you pinging ? 8.8.8.8 ?

[luchonat1]

Hello Georg:

I've come to think that that's exactly the problem for the first case (pinging from inside to outside). I'm pinging different public IP's (one of them are Google's public DNS 8.8.8.8 and 8.8.4.4 and others are public IP's of servers I have in the Internet.

So, i don't recall having clear the nat translation table, but i think that I did that and things started to function normally! I will test it again tomorrow when I get on site!! Just to know, is there a way to force a nat table cleanup after track has detected that a link is up?

But also, this would not solve the second problem: pinging from outside to one of the router interfaces!

[Georg Pauwen]

Hello,

you can automate clearing the NAT translation by implementing the EEM script below:

event manager applet CLEAR_NAT_DOWN
event track 1 state down
action 1.0 cli command "enable"
action 1.1 cli command "clear ip nat translation forced"

event manager applet CLEAR_NAT_UP
event track 1 state up
action 1.0 cli command "enable"
action 1.1 cli command "clear ip nat translation forced"

I'll have a look at the outside to inside issue...

[luchonat1]

Ok, so I've done some tests over the VPN bringing interfaces up and down:

Clearing the NAT table seems to be solving the problem of pinging from inside to outside. But pinging from an outside host to router interfaces public IP continues to fail when the two ISP's are UP. Just to be clear:

Pinging ISP1 connected interface, then bring up ISP2 connected interface -> ping starts to fail.

Strange though, i did not get disconnected from the VPN.

[luchonat1]

Also, I have another aspect of the Issue:

When pinging from the router's CLI to google nameservers and forcing it to use one of the interfaces, the only one that succeeds is the one that is stated as gateway of last resort.

output of sh  ip route is:

Gateway of last resort is A1.B1.C1.D1 to network 0.0.0.0

S* 0.0.0.0/0 [254/0] via A1.B1.C1.D1
                    [254/0] via A2.B2.C2.D2

etc...

So, in that case, ping will only succeed through ISP 1 (A1.B1.C1.D1).

[luchonat1]

Hello Georg, did you have any chance to check on the outside to router's pubic facing interface issue??

Thanks!!

[Georg Pauwen]

Hello,

with CEF enabled and two equal cost static routes, the router does per-destination load balancing.

Does the ping succeed with a destination address that you did not previously try, that is, for which there is no entry in the NAT table yet ?

ping 8.8.4.4 source A2.B2.C2.D2

?

[luchonat1]

Hi Georg!

I can't confirm it right now! I'll test it tomorrow! Right now, I'm not able to connect through VPN...

[luchonat1]

Hello Georg:

I confirm that ping does NOT succeed in that condition.

Also I've checked the output of 

sh ip cef A.B.C.D for that IP before and after issuing the ping, and what I get is 

0.0.0.0/0
nexthop A1.B1.C1.D1 GigabitEthernet0/0
nexthop A2.B2.C2.D2 GigabitEthernet0/1

BUT:

I've done other test:

Issuing:

(config)#no ip cef

(config)#ip cef

#ping A.B.C.D source A1.B1.C1.D1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to A.B.C.D, timeout is 2 seconds:

Packet sent with a source address of A1.B1.C1.D1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 236/240/244 ms
#ping A.B.C.D source A2.B2.C2.D2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to A.B.C.D, timeout is 2 seconds:
Packet sent with a source address of A2.B2.C2.D2
.....
Success rate is 0 percent (0/5)

(config)#no ip cef

(config)#ip cef

#ping A.B.C.D source A2.B2.C2.D2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to A.B.C.D, timeout is 2 seconds:

Packet sent with a source address of A2.B2.C2.D2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 236/240/244 ms
#ping A.B.C.D source A1.B1.C1.D1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to A.B.C.D, timeout is 2 seconds:
Packet sent with a source address of A1.B1.C1.D1
.....
Success rate is 0 percent (0/5)

And I'm using an A.B.C.D of one of my internet servers and not google nameservers just in case some device inside or the router itself is trying to reach that IP address to solve DNS names.

Also, as I am using the router as the local DNS server this issue is bringing me navigation problems!

For me, the problem seems to be that when the router itself needs to send a packet, the outgoing interface is not matched with the source IP of the packet (that should be one of the public facing IP's of the router!) How can this be forced??

Thanks a lot for your time!!

[luchonat1 · Accepted solution]

Ok so finally, I solved this issue.

First of all, I deactivated PFR. It's not completely clear to me if it is usefull in this scenario.

But the problem with router's local traffic persisted, so I created ACL's to select traffic which source IP's are the router's interfaces IP's, and then a route map to modify their next-hop; and finally the key to all this is to apply it to router's local traffic (traffic generated by the router itself) that is not subject to cef and nat!

For reference, the commands are:

ip access-list extended ISP_1_Local
permit ip host ISP_1_ROUTER_IP any
ip access-list extendedISP_1_Local
permit ip host ISP_2_ROUTER_IP any

route-map router-local permit 10
match ip address ISP_1_Local
set ip next-hop ISP_1_DG_IP
!
route-map router-local permit 20
match ip address ISP_2_Local
set ip next-hop ISP_2_DG_IP

ip local policy route-map router-local

And that did the trick!