Dual ISP splitting traffic

Robert Murdock · ‎02-05-2015

Hello Cisco geniuses!

I have a network where I am required to split traffic between two ISPs. ISP1 (Verizon) and ISP2 (Comcast). ISP1 uses T1 connections and ISP2 uses a standard cable modem. Currently all of my traffic goes out of the T1 connection but I would like for HTTP(80) and HTTPS(443) to go out of my ISP2 connection. On ISP1 I need to have clients come in on ports 500 for S2S VPN, 8421 IBM Access, 3024 IBM Access.

I have actually programmed my 2811 and had my PBR working but all of a sudden it stopped. Unfortunately my syslog and NCM did not record any changes to the routers and the routers were written and configs saved and backed up. Im not sure if my ISP2 modem failed because my PBR no longer works. If you could find it in your hearts to review my config and see if anything is missing.

interface MFR1
mtu 4470
no ip address
no ip redirects
no ip proxy-arp
encapsulation frame-relay IETF
no ip mroute-cache
load-interval 30
no arp frame-relay
frame-relay multilink bid to u300785
frame-relay lmi-type ansi
!
interface MFR1.500 point-to-point
ip address 152.179.XX.XX 255.255.255.252
ip access-group 110 in
no ip redirects
no ip proxy-arp
no cdp enable
no arp frame-relay
frame-relay interface-dlci 500 IETF
!
interface FastEthernet0/0
description to ASA5520
ip address 65.216.XX.XX 255.255.255.248
ip nat inside
ip virtual-reassembly
ip route-cache flow
ip policy route-map COMCAST_TRAFFIC
duplex auto
speed auto
!
interface FastEthernet0/1
description Interface to ISP2
ip address 23.31.XX.XX 255.255.255.248
no ip redirects
no ip proxy-arp
ip nat outside
ip virtual-reassembly
ip route-cache flow
duplex auto
speed auto
no cdp enable

ip route 0.0.0.0 0.0.0.0 152.179.XX.XX (default route to ISP1)

!

access-list 120 permit tcp any any eq www
access-list 120 permit tcp any any eq 443
access-list 120 permit udp any any eq domain
access-list 120 deny udp any any eq ntp
access-list 120 deny udp any any eq 8933
access-list 120 deny udp any any eq 8943
access-list 120 deny udp any any eq 19560
access-list 120 deny udp any any eq 65535
access-list 120 deny tcp any any eq 1025
access-list 120 deny udp any any eq 1025
access-list 120 deny tcp any any eq 8933
access-list 120 deny tcp any any eq 8943
access-list 120 deny tcp any any eq 19560
access-list 120 deny tcp any any eq 65535
access-list 120 deny tcp any any eq 50
access-list 120 deny tcp any eq 51 any
access-list 120 deny tcp any any eq 51
access-list 120 deny tcp any eq 500 any
access-list 120 deny tcp any eq 4500 any
access-list 120 deny tcp any eq 50 any
access-list 120 deny udp any eq 50 any
access-list 120 deny udp any eq 51 any
access-list 120 deny tcp any eq 4820 any
access-list 120 deny tcp any eq 4823 any
access-list 120 deny tcp any eq 4822 any
access-list 120 deny tcp any eq smtp any
access-list 120 deny tcp any eq 5223 any
access-list 120 deny ahp any any
access-list 120 deny ip any any
access-list 120 deny tcp any eq 8421 any
access-list 120 deny tcp any eq 3024 any
access-list 120 deny udp any eq 3024 any
access-list 120 deny udp any eq 8421 any

!

access-list 110 permit ip any any
access-list 110 deny 53 any any
access-list 110 deny 55 any any
access-list 110 deny 77 any any
access-list 110 deny pim any any
access-list 110 deny ip host 0.0.0.0 any
access-list 110 deny ip 127.0.0.0 0.255.255.255 any
access-list 110 deny ip 192.0.2.0 0.0.0.255 any
access-list 110 deny ip 224.0.0.0 31.255.255.255 any
access-list 110 deny ip 10.0.0.0 0.255.255.255 any
access-list 110 deny ip 172.16.0.0 0.15.255.255 any
access-list 110 deny ip 192.168.0.0 0.0.255.255 any
access-list 110 deny ip 207.159.122.144 0.0.0.7 any

!

route-map COMCAST_TRAFFIC permit 10
match ip address 120
set ip next-hop 23.31.XX.XX

Amit Goyal · ‎02-06-2015

Hi Robert,

>Please elaborate when you say it stopped working. Is this a PBR which not working, NAT is not taking place, or some other issue with return path?

>in ACL 120 i see that you have three permit statement and rest of them are deny statements.

you can simply the ACL 120 as per below.

access-list 120 permit tcp any any eq www
access-list 120 permit tcp any any eq 443
access-list 120 permit udp any any eq domain

access-list 120 deny ip any any

>You can apply below ACL on interface FastEthernet0/1 to check if packets are going or not.

ip access-list extended TEST

permit tcp any any eq www

permit tcp any any eq 443

permit udp any any eq domain

permit ip any any

int fa0/1

ip access-group TEST out

>Send the desired traffic via Fa0/1 and check if you see hits in ACL with below commands.

show ip access-list TEST

>After making the above test please provide results along with NAT config in your next reply to this thread.

HTH

-Amit

Robert Murdock · ‎02-07-2015

I believe the problem is with return traffic. When I go out to see how others see my ip address the ISP 2 address shows which is correct. When going to speedtest.Com it's not showing the correct speed. I think the issue is with return traffic. The PBR is matching packets and NAT is working just the return traffic seems to be coming in the ISP 1 Interface. Previously I had it working and it showed the correct IP with correct speed. I believe one of my employees removed an "in" statement on one of the interfaces but my sys log was offline and my Orion NCM was offline as well. Maybe a deny for www on the ISP 1 in interface?

Amit Goyal · ‎02-07-2015

Hi Robert,

>Are you running any routing protocol with ISP1 and ISP2?

>If yes, are you advertising the public ip into that routing protocol?

>If yes then there is a chance that public ip of ISP2 is being advertised via ISP1 which is being preferred for return traffic.

>If above is not the case then i do not see any reason of return traffic to come via ISP1. It can still be checked with ISP1.

>You can add below statement with least sequence number in ACL 110

ip access-list ext 110

<seq> permit tcp any any eq www

check the output of "show ip access-list 110"

HTH

-Amit

Robert Murdock · ‎02-07-2015

Thank you for your prompt replies. I do not have any routing between the two as they are two different providers. I will place the www on my 110 acl and see if I get any hits.

Robert Murdock · ‎02-09-2015

I would like to add to further clarify my situation. I really do not need to NAT anything on ISP2 as all of my critical servers are accessed by NAT ISP1 mostly to 65.216.XX.XX:3024 which goes internally to 10.0.0.X:3024.

Amit Goyal · ‎02-09-2015

>i see you have public ip on LAN facing interface so you dont need NAT on the router.

>I believe all NAT rules are applied on Firewall.

>SO you just need to check if the traffic for critical application is hitting to your router to correct ip address (65.216.XX.XX) or not. If yes then it will be forwarded to right interface by router.

>If the traffic is not hitting then it needs to be check with ISP1. (could be some routing issue at their end)

>You can check the traffic hits with the help of ACL on ISP1 interface. Same test has been explained in earlier communication.

Robert Murdock · ‎02-10-2015

Oh wow. So if I were to change my default route to a lower metric and remove all NAT statements I should be fine? You are correct to say that my NAT is handled by my firewall and my public IP space is on the F0/0 interface which has a public IP from ISP 1 as well. Which then translates into my vlan 10.0.0.0. I will try taking all NAT statements off. But should I still have a pbr that sends my 8421 and vpn traffic to my mfr1.500 interface?

Amit Goyal · ‎02-10-2015

>Yeah no NAT statement required on Router if it is done on ASA.

>Since you have default route pointing to ISP2, you must have PBR to redirect traffic (Special traffic) to ISP1.

>But return traffic may hit to ISP2 based on the routing in internet cloud. It will cause Asymmetric routing. if there is no NAT applied on router, Asymmetric routing should not cause any issue.

Robert Murdock · ‎02-10-2015

That's exactly what I'm seeing. The return traffic isn't coming in from ISP1. I'll try this again when I take the nat statements off. Thanks I have learned so much from you

Amit Goyal · ‎02-10-2015

Hi Robert,

If the communication on this thread helped you in anyway, then please rate it so that it help others.

HTH

-Amit

Robert Murdock · ‎02-10-2015

it has helped but unfortunately i still am unable to split my traffic without sacrificing something. I removed all NAT statements and left my default route to ISP2 and my internet connections dropped as well as my clients accessing servers via ISP1. When placing Nat back on

F0/0

ip nat inside

F0/1

ip nat outside

ip nat inside source route-map cable-nat interface f0/1 overload

this restores internet but clients cannot connect in.

when i change my default route to ISP1 and PBR that sends www and 443 traffic to my ISP2 modem, clients can connect in but my internet speeds reflect that of ISP1 yet the IP address shows that of ISP2 when going to speedtest.com. I am really at a loss here for a solution.

Robert Murdock · ‎02-10-2015

in a fit of desperation my configs to my dual isp router (2811) and my ASA are attached (5520)

ISP1---(VPN/3024)------2811------(www/443)----ISP2

I

ASA5520 (Default route to F0/0 of 2811)

I

2600 LAN (default route to ASA5520)

Robert Murdock · ‎02-07-2015

I added a permit of www to my acl 110. It does not show any hits. I am really confused now. I have considered changing how my traffic routes now. If i change the default route to my ISP2 address, and change my PBR. I need to have my external clients connect via ISP1 over ports 8421, 500, 4500, and 3024. So if i create another ACL that permits any any eq those ports and apply it to my MFR1.500 interface will that bring them in correctly? I did a test of changing the default route to 0.0.0.0 0.0.0.0 23.31.XX.XX which is my ISP2 modem and I saw the ISP2 IP address as well as the speed of 84Mb/s. But after that change my clients could not connect to my 65.216.XX.XX addresses over the ports specified. On my ISP2 address space I do not have any critical services coming in, it is only used for internet outgoing from my corporate office. I believe changing the default route to my ISP2 is the way I should go. Can you advise on how my PBR should look to bring in my external clients over ports 500,4500,8421 and 3024?

Amit Goyal · ‎02-07-2015

Hi Robert,

>if ACL doesn't show any hits that means return traffic of www is not coming via ISP1.

As per latest requirement you can do below.

>Have a default route to ISP2. (i believe you have already configured it)

>For traffic via ISP1 have a PBR as per below.

ip access-list extended TEST

permit statements for ports 8421, 500, 4500, and 3024

route-map PBR

match ip address TEST

set ip next-hop <ISP1>

>Please make sure to tweak your NAT statements accordingly.

HTH

-Amit