ISP Failver 2 ISP's and 2 Firewalls

[ALIAOF_]

I'm working on setting up ISP failover and looking for some direction as the setup is a bit different than what I am used to:

There are two firewalls, two ISP's and two core switches setup in GLBP configuration.

One Firewall1 is connected to ISP1 and being used for internet access and NAT's for our servers

Firewall2 is connected to ISP2 and being used only for site to site VPNs

We have multiple VLAN's

ISP1 --> FIREWALL1 & FIREWALL2 (Primary)--> CORESW1

ISP2 --> FIREWALL1 & FIREWALL2 (Secondary) --> CORESW2

Now these firewalls are setup in Active/Standby cofiguration so the Primary ones are going to the Core1 and Secondary ones are going to the Core2.

On the FIREWALL1 there is a PAT setup like this:

object network outside_pat

subnet 0.0.0.0 0.0.0.0

nat (inside,outside) dynamic 1.1.1.1

There is none on the secondary so I was thinking about doing one there and setup tracking on my L3 switches.  But since the second firewall has all the site to site VPN's I'm thinking I will need to use the interface IP address so something like this:

object network fw2_outside_pat

subnet 0.0.0.0 0.0.0.0

nat (inside,outside) dynamic interface

Next on the L3 switches right now I have the default route going to the FIREWALL1's Inside IP lets say of 192.168.1.1.  And the Inside IP of FIREWALL2 is 192.168.1.10.  So I will setup tracking and a second default route:

track 999 ip route 0.0.0.0 0.0.0.0 reachability

ip route 0.0.0.0 0.0.0.0 192.168.1.10 track 999

                             This is FIREWALL2 IP

So basically if the default route is no longer working via 192.168.1.1 second route to 192.168.1.10 should become active. 

Am I missing anything here?

[Simon Brooks]

Hi

First off you cannot have differing configs on true active-standby asa's. My guess is these are independent of each other however you have them on the same subnet (assuming 24bit). Un-necessary broadcast traffic going to each firewall that doesnt need it. Create 2 point to point routed links between cores and asa's if unable to do so then dont worry as current setup is fine. Have ip sla track the reachability of the inside ip on asa1 and track that on a default route with metric 1 on cores. Have a backup default route with a higher metric without a track pointing to secondary. You are correct with pat statements on asa's. Is this what your after?


Sent from Cisco Technical Support Android App

[ALIAOF_]

Thanks for the reply but I never said anything about different configs on the active/standby ASA's.  I meant that I have 4 ASA's total and 2 pairs.  First pair is for ISP 1 and second pair is for ISP 2.  Second pair is only doing VPN currently. 

From each pair Primary ASA's are connected to the First Core and Secondary ASA's are connected to the Second Core.

I'm not sure what you mean by the two ASA's on the same subnet yes they actually were already setup like that and that network unfortunately was setup by the previous network guy and it is /22 (don't ask) I have been trying to change that to /24 for months.  Any ways that is another discussion.  So basically I have seen multiple ASA's on the same subnet many times so I'm not sure what do you mean can you elaborate a bit please?

I can not create point to point routed links between the cores and ASA's though.

If I track the inside IP addresses of the ASA's that won't help me because if there is an issue with the ASAs Inside IP address will still be reachable.

[Bilal Nawaz · Accepted solution]

Hey Mohammad, I saw your post in the other thread and I think it was this one you wanted my attention to. So I have lab'd up again I think I understand your request about default routing to the first pair of ASA's and then the secondary failover route to the second pair of ASA's. I have also assumed that your whole 'routed network is in 192.168.1.x' range.

Hence I have this scenario which hopefully illustrates similar to yours. I set up "ASA's" with simple static routes and NAT/PAT to the internet.

Here in this scenario, there is an IP out on the internet - 100.0.0.1 which I will try and get to in all my tests. I will use the providers interface ip (WAN1 fa0/0) for tracking my default route. Then I will have a backup default route, by making it less prefered. I do this with increasing metric. Therefore it doesnt make it to the routing table until the primary route fails.

Doesnt matter about GLBP too much here because its for downstream devices right? not upstream.

The reason why I wont track 100.0.0.1 is because i will cause flaps with routes and tracking. Because I will be able to get to it via ASA 1 but when the routes fail over, i will still have access to 100.0.0.1 and I will have a default route in the routing table that doesn't work, then it fails over again, then it works via ASA2 then it puts the primary default route back, and we have this flapping problem....

So instead, I will track my providers IP which should be good enough.

Here's show run on CORE01 - some output has been erased.

CORE01#

CORE01#show run

hostname CORE01

!

ip sla monitor 1

type echo protocol ipIcmpEcho 1.1.1.1 source-interface Vlan192

frequency 5

ip sla monitor schedule 1 life forever start-time now

!

!

!

track 1 rtr 1 reachability

!

interface FastEthernet1/0

!

interface FastEthernet1/1

switchport access vlan 192

duplex full

speed 100

spanning-tree portfast

!

interface FastEthernet1/10

switchport mode trunk

!        

!

interface Vlan192

ip address 192.168.1.254 255.255.255.0

!

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 192.168.1.1 name D.R._WAN1_VIA_ASA_CLX1_PMRY track 1

ip route 0.0.0.0 0.0.0.0 192.168.1.10 100 name D.R._WAN2_VIA_ASA_CLX2_BCKUP

ip route 1.1.1.1 255.255.255.255 192.168.1.1 name TO_WAN1_VIA_ASA_CLX1

!

!

CORE01#

=========================================

Here is config on CORE02 (pretty much the same tbh!)

=========================================

CORE02#

CORE02#show run

Building configuration...

hostname CORE02

!

ip sla monitor 1

type echo protocol ipIcmpEcho 1.1.1.1 source-interface Vlan192

frequency 5

ip sla monitor schedule 1 life forever start-time now

!

!        

track 1 rtr 1 reachability

!

!

interface FastEthernet1/0

!

interface FastEthernet1/1

switchport access vlan 192

duplex full

speed 100

spanning-tree portfast

!

interface FastEthernet1/10

switchport mode trunk

!

interface Vlan192

ip address 192.168.1.253 255.255.255.0

!

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 192.168.1.1 name D.R._WAN1_VIA_ASA_CLX1_PMRY track 1

ip route 0.0.0.0 0.0.0.0 192.168.1.10 100 name D.R._WAN2_VIA_ASA_CLX2_BCKUP

ip route 1.1.1.1 255.255.255.255 192.168.1.1 name TO_WAN1_VIA_ASA_CLX1

!

CORE02#

Just to show that the tracking is 'up' and the default route is expected to go via ASA_CLX1 which is the 192.168.1.1 route.

Just to do a traceroute to 100.0.0.1 to see that both should go via ASA_CLX1 (WAN 1)

So both are going via the correct path (WAN 1 which is 1.1.1.X network). Now I will fail WAN 1 by shutting down the interface on WAN 1. Reachability to 1.1.1.1 should stop and therefore tracking should fail. Then we should get our secondary route come in to play. Lets see what happens on both Core's when connectivity to WAN 1 is lost.

As you see, our secondary default route with metric 100 has come in to play. The track went down and took the primary route out of the routing table. Since there is a secondary route, its going via ASA_CLX2. Lets see what happens with traceroute to 100.0.0.1.

Here we can see that both are now taking the path via the second ASA and the WAN 2 link via 2.2.2.X network.

As soon as WAN 1 will recover, the tracking on both will pick this up, and then both will start to route via ASA1 and WAN 1 again.

Hope this helps

Please rate useful posts & remember to mark any solved questions as answered. Thank y

[ALIAOF_]

Bilal yes this is the post I was talking about thank you for your assistance and putting all that effort into it.  So I get it all and makes sense and I was thinking along similar lines, couple of thoughts:

The reason why I wont track 100.0.0.1 is because i will cause flaps with  routes and tracking. Because I will be able to get to it via ASA 1 but  when the routes fail over, i will still have access to 100.0.0.1 and I  will have a default route in the routing table that doesn't work, then  it fails over again, then it works via ASA2 then it puts the primary  default route back, and we have this flapping problem....

See I'm not getting this part exactly.  Because for example my ASA's outside interface is 1.1.1.2 and default gateway is ISP's router which is 1.1.1.1.  But since ASA is directly connected to it that will always be reachable so technically failover will never happens unless something happens to the link between the ASA's outside interface and ISP's router does that makes sense?

By the way we have multiple VLAN's that route to the VIP of the Cores.

Now second question is about the NAT on the secondary ASA, I might have to post that part under firewalls but just want to see if y'all might can tell me if that makes sense.  So on the VPN ASA since the interface IP's is being used for the VPN.  I will have to setup the dynamic NAT like this using the interface vs another IP address:

object network fw2_outside_pat

subnet 0.0.0.0 0.0.0.0

nat (inside,outside) dynamic interface

[Bilal Nawaz · Accepted solution]

Okay, so let me explain why its such a problem - your question 1

If we track the 100.0.0.1 address, on both Core's we will be able to get to WAN1. When WAN 1 fails we should switch over right? Okay, so what happens next?

Our secondary default route kicks in to play, now we are going via WAN 2 right?... BUT... we have reachability to 100.0.0.1..

Then what happens, oh the SLA says, I can get to 100.0.0.1 now, let me put that primary default route back. Then it puts the default route back. But the primary WAN 1 is still down! Okay so now we gotta switch over again.

So this continues and can be a major pain!! LOL.

Anyway, I've tweaked this design and brought in some magic! It's much more appropriate and works like a charm. I will enable OSPF in my domain in the backbone area. I will plan to advertise the default route into OSPF via ASACXL1 (WAN1), only if I have my default route to the internet (which is being tracked). If my default route is gone, stop advertising my default route, and use the default route to the second side.

This is called conditional advertisement. Advertising the default route ONLY IF, I have a route in my routing table.

In this case of failover ASACXL2 will start to advertise its default route into OSPF, it has a static route out to the internet for WAN2 as well.

So just to put some context around this:

Nothing is different on CORE01 and 02. All that is different is that OSPF has been enabled on VLAN 192. I've done some real configuration on the ASA's to show you how it all works.

Here's basic config on the Core's:

==========================================

CORE01#

==========================================

CORE01#show run

Building configuration...

Current configuration : 1507 bytes

!

version 12.4

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname CORE01

!

interface FastEthernet1/1

switchport access vlan 192

spanning-tree portfast

!

interface FastEthernet1/10

switchport mode trunk

!

interface Vlan192

ip address 192.168.1.254 255.255.255.0

ip ospf 1 area 0

!

router ospf 1

log-adjacency-changes

==========================================

CORE02#

==========================================

CORE02#show run

Building configuration...

Current configuration : 1507 bytes

!

version 12.4

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname CORE02

!

interface FastEthernet1/1

switchport access vlan 192

spanning-tree portfast

!

interface FastEthernet1/10

switchport mode trunk

!

interface Vlan192

ip address 192.168.1.253 255.255.255.0

ip ospf 1 area 0

!

router ospf 1

log-adjacency-changes

==========================================

==========================================

So just simple OSPF here. Most of the hard work will be on the ASA's.

Things to consider here

1. Advertising out a default route from both ASA's
2. ASACXL1 should be prefered than ASACXL2 (will be done via metrics)
3. Both default route advertisements will be conditional, only if 0.0.0.0 0.0.0.0 next hop and exit interface match
4. route-map will be set, for the condition to match the default route in the routing table
5. the default route will be tracked to 100.0.0.1 which is an address out on the internet (both ASA's will be doing this)
6. If the track is up, the default route will exist and be advertised. If it does not exists and the track is down, the default route will not be advertised.
7. When its not advertised via ASACXL1 it will be advertised by ASACXL2 which will advertise default route with a higher metric - when the primary recovers, since it will have a lower metric all traffic will go back to normal. (i.e. via ASA1)

So there's quite a lot here - just to show you the config on ASA_CXL1:

==========================================

==========================================

ASACLX1(config)# show run

: Saved

:

ASA Version 8.0(2)

!

hostname ASACLX1

enable password 8Ry2YjIyt7RRXU24 encrypted

names

!

interface Ethernet0/0

nameif ##INSIDE_CORE01##

security-level 100

ip address 192.168.1.1 255.255.255.0

!

interface Ethernet0/1

nameif ##OUTSIDE_WAN1##

security-level 0

ip address 1.1.1.2 255.255.255.252

!

passwd 2KFQnbNIdI.2KYOU encrypted

!

access-list WAN1 standard permit any

access-list OUTSIDE_TO_IN extended permit ip any any

pager lines 24

mtu ##INSIDE_CORE01## 1500

mtu ##OUTSIDE_WAN1## 1500

no failover  

icmp unreachable rate-limit 1 burst-size 1

no asdm history enable

arp timeout 14400

global (##OUTSIDE_WAN1##) 1 interface

nat (##INSIDE_CORE01##) 1 192.168.1.0 255.255.255.0

access-group OUTSIDE_TO_IN in interface ##INSIDE_CORE01##

access-group OUTSIDE_TO_IN in interface ##OUTSIDE_WAN1##

!

########################################################

! THIS IS FOR MY CONDITIONAL ADVERTISEMENT OF DEFAULT

########################################################

route-map conditional permit 10

match ip address WAN1

match interface ##OUTSIDE_WAN1##

match metric 1

!

!

########################################################

! DEFAULT ROUTE GENERATION IF ROUTEMAP IS TRUE

########################################################

router ospf 1

network 192.168.1.1 255.255.255.255 area 0

log-adj-changes

default-information originate route-map conditional

!

########################################################

! PREFERED ROUTES BEING TRACKED

########################################################

!

route ##OUTSIDE_WAN1## 0.0.0.0 0.0.0.0 1.1.1.1 1 track 1

!

########################################################

! SLA AND TRACKING SET UP TO PING INTERNET ADDRESS OF

! 100.0.0.1 VIA THE OUTSIDE INTERFACE OF ASA WITH TWEAK'D

! TIMERS

########################################################

!

sla monitor 1

type echo protocol ipIcmpEcho 100.0.0.1 interface ##OUTSIDE_WAN1##

num-packets 2

frequency 5

sla monitor schedule 1 life forever start-time now

!

track 1 rtr 1 reachability

telnet timeout 5

ssh timeout 5

console timeout 0

no threat-detection basic-threat

no threat-detection statistics access-list

!

!

prompt hostname context

Cryptochecksum:00000000000000000000000000000000

: end

ASACLX1(config)#

==========================================

==========================================

ASACXL2#

ASACXL2# SHOW RUN

: Saved

:

ASA Version 8.0(2)

!

hostname ASACXL2

enable password 8Ry2YjIyt7RRXU24 encrypted

names

!

interface Ethernet0/0

nameif ##INSIDE_CORE02##

security-level 100

ip address 192.168.1.10 255.255.255.0

!

interface Ethernet0/1

nameif ##OUTSIDE_WAN2##

security-level 0

ip address 2.2.2.2 255.255.255.252

!

passwd 2KFQnbNIdI.2KYOU encrypted

ftp mode passive

access-list OUTSIDE_TO_IN extended permit ip any any

access-list WAN2 standard permit any

pager lines 24

mtu ##INSIDE_CORE02## 1500

mtu ##OUTSIDE_WAN2## 1500

!

no asdm history enable

arp timeout 14400

global (##OUTSIDE_WAN2##) 1 interface

nat (##INSIDE_CORE02##) 1 192.168.1.0 255.255.255.0

access-group OUTSIDE_TO_IN in interface ##INSIDE_CORE02##

access-group OUTSIDE_TO_IN in interface ##OUTSIDE_WAN2##

!

########################################################

! THIS IS FOR MY CONDITIONAL ADVERTISEMENT OF DEFAULT

! NOTE THIS HAS METRIC OF 100 - ROUTE MUST MATCH WITH

! THIS METRIC

########################################################

route-map conditional permit 10

match ip address WAN2

match interface ##OUTSIDE_WAN2##

match metric 100

!

!

########################################################

! DEFAULT ROUTE GENERATION IF ROUTEMAP IS TRUE

! SET THE METRIC TO 100 SO ASA 1 IS PREFERED

########################################################

!

router ospf 1

network 192.168.1.10 255.255.255.255 area 0

log-adj-changes

default-information originate metric 100 route-map conditional

!

########################################################

! PREFERED ROUTES BEING TRACKED - AD OF 100 WHICH IS

! BETTER THAN OSPF SO IT WILL USE THIS STATIC ROUTE

! IF THIS FAILS IT WILL USE THE OSPF ROUTE THAT IS ADVERTISED

! BY ASA1

########################################################

!

route ##OUTSIDE_WAN2## 0.0.0.0 0.0.0.0 2.2.2.1 100 track 1

!

sla monitor 1

type echo protocol ipIcmpEcho 100.0.0.1 interface ##OUTSIDE_WAN2##

num-packets 2

frequency 5

sla monitor schedule 1 life forever start-time now

!

track 1 rtr 1 reachability

!

!

prompt hostname context

Cryptochecksum:00000000000000000000000000000000

: end        

ASACXL2#

########################################################

########################################################

########################################################

So now we've gone through the config lets see how things are with all connections working and in UP state.

On ASACLX1:

As you can see, ASACXL1 has OSPF adjacencies with both CORE's and ASACXL2.

Note in the OSPF database the LSA type 5's have been generated with default routes..

Lets look at CORE01 now....

See here, the default route is being advertised with the default metric - all neighbourships have been built on vlan 192.

So both COREs have a default route via 192.168.1.1 - which is good. On ASACXL2 (WAN2) we should have our own default route out to the internet, and not the OSPF one, remember we are serving connections already and default route should be normal.

I have my default static route which is being tracked with AD of 100 - better than OSPF 110.

But NOTE, in my OSPF database I have both default routes. So if ASACXL2 default route goes away because track changes to down, OSPF will introduce the default route in the routing table... We will test this when we try failover.

Traceroutes

VIA WAN 1

==========================

ASACLX1# traceroute 100.0.0.1

Type escape sequence to abort.

Tracing the route to 100.0.0.1

1  1.1.1.1 20 msec *  20 msec

ASACLX1#

==========================

CORE01#traceroute 100.0.0.1

Type escape sequence to abort.

Tracing the route to 100.0.0.1

  1 1.1.1.1 60 msec *  28 msec

CORE01#

==========================

CORE02#traceroute 100.0.0.1

Type escape sequence to abort.

Tracing the route to 100.0.0.1

  1 1.1.1.1 32 msec *  52 msec

CORE02#

==========================

VIA WAN 2

==========================

ASACXL2# traceroute 100.0.0.1

Type escape sequence to abort.

Tracing the route to 100.0.0.1

1  2.2.2.1 10 msec *  20 msec

ASACXL2#

=========================

Great, so we know everything is working. Now for the 'magic' and failover.

I will fail WAN 1 by shutting down the 100.0.0.1 interface.

What should we see here?

1. ASACXL1 will lose its static route, but we should see the other default route via WAN2 advertised by OSPF.
2. CORE01 and 02 should lose its OSPF routes via WAN1 and should gain from ASACXL2
3. All three, ASACXL1, CORE01 and 02 should have routes via secondary.

ASACLX1 has lost its static route because the track status is down. And the secondary route via the second ASA is now active. Note that it has a metric of 100 which we set in our config. The COREs should have the same.

So both are now going via ASACXL2 WAN 2 which is our secondary connection. The metric is important here, we had metric 1 associated with the primary route, for it to make it to the routing table but we had to change the metric of the second default route being advertised to make it less prefered but only used as backup.

So all is working with failover. I will bring everything back online and fail over WAN 2. We should see ASACXL 2 go via WAN 1 instead now.

As you can see, the track state changed to down..... I lost my default static route which was being tracked, but I had OSPF default route via ASACXL1 WAN1. So I have failover both ways not matter what and it is very solid.

ASACXL2# traceroute 100.0.0.1

Type escape sequence to abort.

Tracing the route to 100.0.0.1

1  1.1.1.1 30 msec *  10 msec

ASACXL2#

Sorry about the very long post, may require a lot of change in your network in enabling OSPF, but i've tested this and it seems like a good solution.

With regards to your second question, im not a ASA expert and probably best to post a Q in the firewall section.

Hope this helps.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

[ALIAOF_]

Bilal that is pretty awesome work thank you very much appreciated it.  And thank you for clearing up on tracking the outside IP address indeed it does make sense now.

I like enabling the OSPF solution but unfortunately I can not enable any routing protocol right now it will require a whoooole lot more changes just because of the way network was initially setup.  So for now I'll just go with setting up tracking on the cores and using a higher metric. 

I'll post back my results

[ALIAOF_]

Ah never mind I forgot to add the track keyword and remove the old route lol.  It works awesome lost 3 packets and switch back was seamless.

My NAT on the second firewall works the way I thought it would.  Bilal thank you again. 

[Bilal Nawaz]

Haha :) , I got a bit worried when I saw the notification for your post. But I'm glad you managed to fix it all up!

Sent from Cisco Technical Support iPhone App

[ALIAOF_]

Yeah that was the easy part boy getting the second firewall going was a pain lol because the dynamic NAT broke the internet access for the Cisco VPN users just the way it was setup by the previously.  Any ways got that working after making additional NAT changes on that firewall.  So all is working like a charm now .  Have a great weekend to ya.

I gotta get my hands on some used network gear.

[lawless@chime.org]

I have 2 IPS's, 2 WAN locations - connected by fiber L3, and 2 pairs of ASA's - one pair for each site (currently not configured. How do I setup both sites for each site to uses the local ISP and fails over to the other ISP? What hardware would I also need? Do I need 2 /24 public IP addresses?

Thanks

[Simon Brooks]

Ok I see two options then. Use icmp ip sla on core 1 to a public ip such as 4.2.2.2. Have a static route for that ip to point to asa1. Have a default route with metric 1 with the track for the ip sla on it. And have an additional default route with metric 10 pointing to secondary asa group. Do this on both cores ( assuming you are running hsrp between both and have different priorities fir different vlans).

Track 1 fails so secondary route to other firewall gets put into table.

Any use?


Sent from Cisco Technical Support Android App

[Simon Brooks]

Oops didnt mention second option

Track the line protocol to asa instead if a public ip.


Sent from Cisco Technical Support Android App