How to configure multiple outgoing interfaces + NAT + PfR

[pkochegarov1]

Hello,

I have the following config running on Cisco2851.
Five interfaces (four ADSL and one LAN 10Mb/s) connected to Internet using pppoe.
Local policy is used to make working route tracking.
The PfR also configured to load balance traffic coming from LAN to Internet.
PAT is also configured with "oer" keyword at the end of string to not relocate working translations.
But the router is not performing good. :-(
After investigation I found that the selection of the exit interface and setting source ip for
NAT is not synchronized. The provider's router just drops the incoming packet due to uRPF check.
Also, the selection of the exit interface is not PFR aware (mode select-exit best) during
NAT session setup, and router selects one of the possible exit interfaces randomly.

I have two questions:
1. How to make synchronization of NAT and Routing to build matching pair of Out_IP=Out_Interface and make my setup working?
2. How to select the less loaded interface during setup of NAT phase and Routing phase and really involve PfR?

Actually, these two questions is just my one requirement: during setup of NAT session, I need
to find less loaded interface (PfR should check current rx/tx load), select it, and keep it untouched.

Thanks,

Sergey

Config:

!
version 15.1
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname bif
!
boot-start-marker
boot system flash:c2800nm-adventerprisek9-mz.151-4.M8.bin
boot-end-marker
!
!
enable secret 5 $1$3ggj$huERPVt0luOX6qo6
!
no aaa new-model
!
!
crypto pki token default removal timeout 0
!
!
dot11 syslog
no ip source-route
!
!
ip cef
!
!
!
no ip domain lookup
ip domain name zzz.mgm
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
!
key chain PFR
 key 0
  key-string 7 107E2F2B
!
!
!
!
!
!
voice-card 0
!
!
pfr master
 logging
 !
 border 192.168.254.254 key-chain PFR
  interface Dialer5 external
  interface Dialer4 external
  interface Dialer3 external
  interface Dialer2 external
  interface Dialer1 external
  interface GigabitEthernet0/0 internal
 mode select-exit best
!
pfr border
 logging
 local Loopback0
 master 192.168.254.254 key-chain PFR
!
!
license udi pid CISCO2851 sn FCZ0929
username se privilege 15 secret 5 $1$DUbm$RuZKP8X.19uBtm21
username ru privilege 15 secret 5 $1$1V.h$iotp/bjhUg4ho93d
!
redundancy
!
!
ip ssh version 2
!
track 1 ip sla 1 reachability
 delay down 30 up 15
!
track 2 ip sla 2 reachability
 delay down 30 up 15
!
track 3 ip sla 3 reachability
 delay down 30 up 15
!
track 4 ip sla 4 reachability
 delay down 30 up 15
!
track 5 ip sla 5 reachability
 delay down 30 up 15
!
!
!
!
!
!
!
!
interface Loopback0
 ip address 192.168.254.254 255.255.255.255
!
interface GigabitEthernet0/0
 description ### LAN ###
 ip address 192.168.68.1 255.255.255.0
 no ip redirects
 no ip proxy-arp
 ip flow ingress
 ip nat inside
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface GigabitEthernet0/1
 description ### WDSL link to Dialer 5 ###
 no ip address
 duplex auto
 speed auto
 pppoe enable group global
 pppoe-client dial-pool-number 5
!
interface ATM0/0/0
 description ### DSL link 1 to Dialer 1 ###
 no ip address
 no atm ilmi-keepalive

 shutdown
 pvc 1/32
  pppoe-client dial-pool-number 1
 !
!
interface ATM0/1/0
 description ### DSL link 2 to Dialer 2 ###
 no ip address
 no atm ilmi-keepalive
 pvc 1/32
  pppoe-client dial-pool-number 2
 !
!
interface ATM0/2/0
 description ### DSL link 3 to Dialer 3 ###
 no ip address
 no atm ilmi-keepalive
 pvc 1/32
  pppoe-client dial-pool-number 3
 !
!
interface ATM0/3/0
 description ### DSL link 4 to Dialer 4 ###
 no ip address
 no atm ilmi-keepalive
 pvc 1/32
  pppoe-client dial-pool-number 4
 !
!
interface GigabitEthernet1/0
 description ### Virtual interface to NME-16ES-1G-P ###
 ip address 192.168.254.253 255.255.255.254
!
interface Dialer1
 description ### Dialer for line 1 ###
 bandwidth 224
 bandwidth receive 1728
 ip address negotiated
 ip mtu 1492
 ip nat outside
 ip virtual-reassembly in
 encapsulation ppp
 ip tcp adjust-mss 1452
 load-interval 30
 dialer pool 1
 ppp authentication chap callin
 ppp chap hostname
 ppp chap password
 no cdp enable
!
interface Dialer2
 description ### Dialer for line 2 ###
 bandwidth 224
 bandwidth receive 1728
 ip address negotiated
 ip mtu 1492
 ip flow ingress
 ip nat outside
 ip virtual-reassembly in
 encapsulation ppp
 ip tcp adjust-mss 1452
 dialer pool 2
 ppp authentication chap callin
 ppp chap hostname
 ppp chap password
 no cdp enable
!
interface Dialer3
 description ### Dialer for line 3 ###
 bandwidth 224
 bandwidth receive 1728
 ip address negotiated
 ip mtu 1492
 ip flow ingress
 ip nat outside
 ip virtual-reassembly in
 encapsulation ppp
 ip tcp adjust-mss 1452
 dialer pool 3
 ppp authentication chap callin
 ppp chap hostname
 ppp chap password
 no cdp enable
!
interface Dialer4
 description ### Dialer for line 4 ###
 bandwidth 224
 bandwidth receive 1728
 ip address negotiated
 ip mtu 1492
 ip flow ingress
 ip nat outside
 ip virtual-reassembly in
 encapsulation ppp
 ip tcp adjust-mss 1452
 dialer pool 4
 ppp authentication chap callin
 ppp chap hostname
 ppp chap password
 no cdp enable
!
interface Dialer5
 description ### Dialer for WDSL line ###
 bandwidth 10000
 bandwidth receive 10001
 ip address negotiated
 ip mtu 1492
 ip flow ingress
 ip nat outside
 ip virtual-reassembly in
 encapsulation ppp
 ip tcp adjust-mss 1452
 load-interval 30
 dialer pool 5
 ppp authentication chap callin
 ppp chap hostname
 ppp chap password
 no cdp enable
!
ip local policy route-map LOCAL-PBR
no ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
ip nat inside source route-map NAT1 interface Dialer1 overload oer
ip nat inside source route-map NAT2 interface Dialer2 overload oer
ip nat inside source route-map NAT3 interface Dialer3 overload oer
ip nat inside source route-map NAT4 interface Dialer4 overload oer
ip nat inside source route-map NAT5 interface Dialer5 overload oer
ip nat inside source static tcp 192.168.68.160 22 $$$Dialer5-IP$$$ 2222 extendable
ip nat inside source static tcp 192.168.68.160 22 $$$Dialer2-IP$$$ 2222 extendable
ip nat inside source static tcp 192.168.68.160 22 $$$Dialer3-IP$$$ 2222 extendable
ip nat inside source static tcp 192.168.68.160 22 $$$Dialer4-IP$$$ 2222 extendable
ip nat inside source static tcp 192.168.68.230 21 $$$Dialer1-IP$$$ 21 extendable
ip nat inside source static tcp 192.168.68.160 25 $$$Dialer1-IP$$$ 25 extendable
ip nat inside source static tcp 192.168.68.22 143 $$$Dialer1-IP$$$ 143 extendable
ip nat inside source static tcp 192.168.68.22 443 $$$Dialer1-IP$$$ 443 extendable
ip nat inside source static tcp 192.168.68.160 22 $$$Dialer1-IP$$$ 2222 extendable
ip route 0.0.0.0 0.0.0.0 Dialer1 track 1
ip route 0.0.0.0 0.0.0.0 Dialer2 track 2
ip route 0.0.0.0 0.0.0.0 Dialer3 track 3
ip route 0.0.0.0 0.0.0.0 Dialer4 track 4
ip route 0.0.0.0 0.0.0.0 Dialer5 track 5
!
ip sla 1
 icmp-echo 8.8.8.8 source-ip $$$Dialer1-IP$$$
 timeout 1000
 frequency 5
ip sla schedule 1 life forever start-time now
ip sla 2
 icmp-echo 8.8.8.8 source-ip $$$Dialer2-IP$$$
 timeout 1000
 frequency 5
ip sla schedule 2 life forever start-time now
ip sla 3
 icmp-echo 8.8.8.8 source-ip $$$Dialer3-IP$$$
 timeout 1000
 frequency 5
ip sla schedule 3 life forever start-time now
ip sla 4
 icmp-echo 8.8.8.8 source-ip $$$Dialer4-IP$$$
 timeout 1000
 frequency 5
ip sla schedule 4 life forever start-time now
ip sla 5
 icmp-echo 8.8.8.8 source-ip $$$Dialer5-IP$$$
 timeout 1000
 frequency 5
ip sla schedule 5 life forever start-time now
access-list 100 permit ip any any
access-list 101 permit ip host $$$Dialer1-IP$$$ any
access-list 102 permit ip host $$$Dialer2-IP$$$ any
access-list 103 permit ip host $$$Dialer3-IP$$$ any
access-list 104 permit ip host $$$Dialer4-IP$$$ any
access-list 105 permit ip host $$$Dialer5-IP$$$ any
access-list 199 permit ip 192.168.68.0 0.0.0.255 any
!
!
!
!
route-map LOCAL-PBR permit 10
 match ip address 101
 set interface Dialer1
!
route-map LOCAL-PBR permit 20
 match ip address 102
 set interface Dialer2
!
route-map LOCAL-PBR permit 30
 match ip address 103
 set interface Dialer3
!
route-map LOCAL-PBR permit 40
 match ip address 104
 set interface Dialer4
!
route-map LOCAL-PBR permit 50
 match ip address 105
 set interface Dialer5
!
route-map LOCAL-PBR permit 100
 match ip address 100
 set global
!
route-map NAT3 permit 10
 match ip address 199
 match interface Dialer3
!
route-map NAT2 permit 10
 match ip address 199
 match interface Dialer2
!
route-map NAT1 permit 10
 match ip address 199
 match interface Dialer1
!
route-map NAT5 permit 10
 match ip address 199
 match interface Dialer5
!
route-map NAT4 permit 10
 match ip address 199
 match interface Dialer4
!
!
!
control-plane
!
!
!
!
mgcp profile default
!
!
!
!
!
!
line con 0
line aux 0
line 66
 no activation-character
 no exec
 transport preferred none
 transport input all
 transport output lat pad telnet rlogin lapb-ta mop udptn v120 ssh
line vty 0 4
 session-timeout 15
 login local
 transport input all
line vty 5 15
 session-timeout 15
 login local
 transport input all
!
scheduler allocate 20000 1000
end

Show ip route:

sh ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
  Known via "static", distance 1, metric 0 (connected), candidate default path
  Routing Descriptor Blocks:
    directly connected, via Dialer5
      Route metric is 0, traffic share count is 1
  * directly connected, via Dialer3
      Route metric is 0, traffic share count is 1
    directly connected, via Dialer4
      Route metric is 0, traffic share count is 1
    directly connected, via Dialer2
      Route metric is 0, traffic share count is 1

Log:

*Apr 16 07:04:18.103: IP: s=192.168.68.2 (GigabitEthernet0/0), d=8.8.4.4, len 66, input feature
*Apr 16 07:04:18.103:     UDP src=61183, dst=53, Stateful Inspection(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.103: IP: s=192.168.68.2 (GigabitEthernet0/0), d=8.8.4.4, len 66, input feature
*Apr 16 07:04:18.103:     UDP src=61183, dst=53, Ingress-NetFlow(21), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.103: IP: s=192.168.68.2 (GigabitEthernet0/0), d=8.8.4.4, len 66, input feature
*Apr 16 07:04:18.103:     UDP src=61183, dst=53, Virtual Fragment Reassembly(25), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.103: IP: s=192.168.68.2 (GigabitEthernet0/0), d=8.8.4.4, len 66, input feature
*Apr 16 07:04:18.103:     UDP src=61183, dst=53, Access List(31), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.103: IP: s=192.168.68.2 (GigabitEthernet0/0), d=8.8.4.4, len 66, input feature
*Apr 16 07:04:18.103:     UDP src=61183, dst=53, Virtual Fragment Reassembly After IPSec Decryption(39), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.103: IP: s=192.168.68.2 (GigabitEthernet0/0), d=8.8.4.4, len 66, input feature
*Apr 16 07:04:18.103:     UDP src=61183, dst=53, MCI Check(80), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.103: IP: s=192.168.68.2 (GigabitEthernet0/0), d=8.8.4.4, len 66, input feature
*Apr 16 07:04:18.103:     UDP src=61183, dst=53, TCP Adjust MSS(82), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.103: FIBipv4-packet-proc: route packet from GigabitEthernet0/0 src 192.168.68.2 dst 8.8.4.4
*Apr 16 07:04:18.103: FIBfwd-proc: Default:0.0.0.0/0 process level forwarding
*Apr 16 07:04:18.103: FIBfwd-proc: depth 0 first_idx 3 paths 4 long 0(0)
*Apr 16 07:04:18.103: FIBfwd-proc: try path 3 (of 4) v4-ap-Dialer5 first short ext 0(-1)
*Apr 16 07:04:18.103: FIBfwd-proc: v4-ap-Dialer5 valid
*Apr 16 07:04:18.103: FIBfwd-proc: Dialer5 no nh type 3  - deag
*Apr 16 07:04:18.103: FIBfwd-proc: ip_pak_table 0 ip_nh_table 65535 if Dialer5 nh none deag 1 chg_if 0 via fib 0 path type attached prefix
*Apr 16 07:04:18.103: FIBfwd-proc: packet routed to Dialer5 p2p(0)
*Apr 16 07:04:18.103: FIBipv4-packet-proc: packet routing succeeded
*Apr 16 07:04:18.103: FIBfwd-proc: ip_pak_table 0 ip_nh_table 65535 if Dialer5 nh none uhp 1 deag 0 ttlexp 0
*Apr 16 07:04:18.103: FIBfwd-proc: sending link IP ip_pak_table 0 ip_nh_table 65535 if Dialer5 nh none uhp 1 deag 0 chgif 0 ttlexp 0 rec 0
*Apr 16 07:04:18.103: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Dialer5), len 66, output feature
*Apr 16 07:04:18.103:     UDP src=61183, dst=53, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.103: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Dialer5), len 66, output feature
*Apr 16 07:04:18.103:     UDP src=61183, dst=53, Stateful Inspection(27), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.103: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Dialer5), len 66, output feature
*Apr 16 07:04:18.103:     UDP src=61183, dst=53, CCE Post NAT Classification(38), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.107: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Dialer5), len 66, output feature
*Apr 16 07:04:18.107:     UDP src=61183, dst=53, Firewall (firewall component)(39), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.107: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Dialer5), len 66, output feature
*Apr 16 07:04:18.107:     UDP src=61183, dst=53, TCP Adjust MSS(50), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.107: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Dialer5), len 66, output feature
*Apr 16 07:04:18.107:     UDP src=61183, dst=53, NAT ALG proxy(55), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.107: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Dialer5), len 66, output feature
*Apr 16 07:04:18.107:     UDP src=61183, dst=53, Post-Ingress-NetFlow(68), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.107: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Dialer5), len 66, output feature
*Apr 16 07:04:18.107:     UDP src=61183, dst=53, Dialer idle reset(84), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.107: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Dialer5), len 66, output feature
*Apr 16 07:04:18.107:     UDP src=61183, dst=53, Dialer idle reset(85), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Apr 16 07:04:18.107: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Dialer5), g=8.8.4.4, len 66, forward
*Apr 16 07:04:18.107:     UDP src=61183, dst=53
*Apr 16 07:04:18.107: IP: s=$$$Dialer4-IP$$$ (GigabitEthernet0/0), d=8.8.4.4 (Virtual-Access3), len 66, sending full packet
*Apr 16 07:04:18.107:     UDP src=61183, dst=53

[pkochegarov1]

Update: this problem is mostly related to udp traffic as shown in the debug with route-cache off. I have too much DNS timeouts.

[pkochegarov1]

Update2: when I configure outgoing access lists on Dialer interfaces running in normal route-cache mode with two simple statements I see matches in the "deny ip any any" line after performing DNS requests from LAN host.

Extended IP access list 122
    10 permit ip host $$$Dialer2-IP$$$ any  (15 matches)
    20 deny ip any any (2 matches)

So, I suppose there is bug in UDP/DNS NAT requests.

[hailin huang]

hi,is this question is ok?
if you forget do this config like below:

pfr master
learn
delay
throughput
periodic-interval 3
monitor-period 1
!
pfr master
delay threshold 200
jitter threshold 50 
mode route control 
mode monitor passive
mode select-exit best 

i will do like this,four ADSL connect a switch ,this switch connect a router 2911(with data license)

at 2911 do four  pppoe

i want to load balance at this four adsl.