DUAL ISPs ON CISCO PIX

[veltech]

Hi All,

We have a strange problem with a PIX 515E running 8.0 (4).

BACKGROUND

The Pix is configured with two ISP connections, primary and secondary. We use a simple SLA monitor to check for reachability of the ISP 1 gateway, if this goes down ISP 2 comes up and internet is restored. In addition there are a few site to site VPNs terminating on the Pix and some ad hoc dynamic VPN connections.

PROBLEM

When the primary ISP 1 failed the other day all was well and the Pix transferred to ISP 2 almost in a heartbeat as it is supposed to do. The site to site VPN also changed over without an issue. However, when the primary ISP came back up the secondary route did not clear down in the routing table as evidenced by the "sh route" output. Furthermore the site to site VPN remained connected to the secondary connection. We cleared this fault by doing a simple "shut" and "no shut" on the secondary interface.

Has anyone any idea why this is happening ?

Thanks.

[amitaaga]

Looks like SLA monitoring is not working as expected for you. Make sure that it is configured as per the following doc:

http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_configuration_example09186a00806e880b.shtml#show

If it is configured correctly but is still not working as expected then check to see if SLA monitoring is operational when the primary route comes back up after failing using "show sla monitor operational-state".

[Ajay Saini]

If the secondary ISP came up, I believe that the sla monitoring is working fine. When the issue happens next, and the primary does not come back up. Try to ping the ip address which is being tracked sourced from the primary ISP.

if the ping works fine, then the primary should come back up as well. Also, get syslogs to see if you see something unusual.

[veltech]

Hi All,

Sorry for delayed response. We have run the sh ip sla command and here is the output.

firewall# show sla monitor operational-state 123

Entry number: 123

Modification time: 20:44:07.659 BST Sun Sep 8 2013

Number of Octets Used by this Entry: 1300

Number of operations attempted: 0

Number of operations skipped: 0

Current seconds left in Life: 0

Operational state of entry: Inactive

Last time this entry was reset: Never

This suggests that we have an error with the SLA monitor, but as we said in the original post it fails over perfectly with a failure of the primary connection, but is not flipping over when the primary connection comes live. We have been through the configuration guide and verified all steps which are fairly simple.

Yes, after the primary connection comes back up we can ping the IP address that is being monitored using the primary interface as source. Also, interestingly if we set a ping going to both IP gateways, ie primary and secondary we even see the primary response time of the echo change when the primary comes back up. We can tell because the primary is a dedicated fibre whereas the secondary is good old ADSL. It is however still showing the ISP2 within the routing table. This tends to point towards the routing table not flushing the secondary connection after a fail over and perhaps nothing to do with the SLA..

Anyone got anymore suggestions?

Regards,

[jumora]

Can you please share the configuration.

Known issues:

SLA Monitoring on ASA

Problem:

SLA monitoring does not work after the ASA is upgrade to version       8.0.

Solution:

The problem is possibly be due to the IP       Reverse-Path command configured in the OUTSIDE interface. Remove the command in ASA and try to check the SLA       Monitoring.

Bug ID CSCtc16148

http://tools.cisco.com/squish/6B83a

[veltech]

Hi All,

We need to make a correction to our earlier post.. We said

"Yes, after the primary connection comes back up we can ping the IP address that is being monitored using the primary interface as source"

In fact we can not ping the IP gateway from the primary interface after the connection has come back up. The routing table still shows the route as the secondary connection thereby requiring an admin shutdown of the secondary interface which then puts the primary interface back into the routing table.

Here is the configuration relating to the IP SLA monitor..

sla monitor 123

type echo protocol ipIcmpEcho XXX.XXX.XX.XXX interface Outside

num-packets 3

frequency 10

sla monitor schedule 123 life forever start-time now

track 1 rtr 123 reachability

route Outside 0.0.0.0 0.0.0.0 X.X.X.X 1 track 1

route ISP2 0.0.0.0 0.0.0.0 X.X.X.X 254

Any ideas anyone ???

Regards,

[jumora]

Ok, I am not sure if you checked my post can you please get me the configuration so we can check.

[jumora]

Plus, you can not recover the main link if the IP address that you set to monitor is not available so I am not sure how you are recovering

[jumora]

Please run the next command:

show run ip verify reverse-path

If the command is in place just remove it

enable

config t

no ip verify reverse-path interface Outside

[jumora]

Did the information given help out; please remember to reply "Answered" if the information given to you resolved your problem.

[veltech]

Hi Jumora,

Unfortunately the problem is not resolved.

ip verify reverse-path is not running on the outside interface. Here is a further screenshot of the results of the SLA output:

firewall# sh sla monitor operational-state 123

Entry number: 123

Modification time: 23:41:42.374 BST Tue Oct 22 2013

Number of Octets Used by this Entry: 1480

Number of operations attempted: 13064

Number of operations skipped: 0

Current seconds left in Life: Forever

Operational state of entry: Active

Last time this entry was reset: Never

Connection loss occurred: FALSE

Timeout occurred: FALSE

Over thresholds occurred: FALSE

Latest RTT (milliseconds): 1

Latest operation start time: 11:58:52.372 BST Thu Oct 24 2013

Latest operation return code: OK

RTT Values:

RTTAvg: 1       RTTMin: 1       RTTMax: 1

NumOfRTT: 3     RTTSum: 3       RTTSum2: 3

This all seems fine and as we have said the IP SLA fails to the secondary ISP without an issue, it just will not reinstate the primary ISP connection after it becomes available again. The only way we can get this to flip back to the primary connection is to do a "shut" and "no shut" on the secondary interface. The "sh route" output verifies that this is happening which seems to point towards the secondary route not getting cleared from the routing table correctly.

Any more ideas?

Regards,

[jumora]

Can you send me the show run ICMP?

Also, can you try changing the IP address that you are using to monitor for 4.2.2.2 just as a test.

The strange thing is that you state that the main link recovers but this is not possible if the address that you are using to monitor is never available.

If you cannot change the monitoring address on the SLA configuration, please check with a packet-tracer how the ASA would consider to route to that destine address from a LOCAL address (PC).

packet-tracer input inside icmp 8 0 detail

If you have any VPN configuration please confirm that you do not have a match for ip any any or icmp any any as this could affect SLA monitoring.

[veltech]

Hi Jumora,

Here are the answers to your questions.

firewall# sh run icmp

icmp unreachable rate-limit 1 burst-size 1

icmp deny any echo Outside

icmp permit any echo-reply Outside

icmp deny any echo ISP2

icmp permit any echo-reply ISP2

Yes, the primary link will recover when we shut down ISP2, which is the secondary connection. The same results were obtained using 4.2.2.2

We do have VPNs on this device and have checked the configuration which all seems fine, no ip any any or icmp any any..

It may be that this problem is not with the SLA monitor as we seem to get all the right output from the show monitor. I think now the problem maybe with the static routes which are not clearing the secondary route without shutting down the secondary interface. The question is why this is happening????

Any more thoughts??

[jumora]

Answer, remove the next line:

icmp deny any echo Outside

[veltech]

Hi Jumora,

Just logged in and removed "icmp deny any echo Outside" and it didn't work. This is a real head scratching problem as we configure a lot of these failovers and have not had this problem before. I am going to do some more debug tomorrow and see if I can see anything on the routing side that may be causing the routing table to retain the secondary ISP route, in particular I am going to look at the DHCP config on the Outside interface, although it picks up a static IP. Although rarely required I am thinking of a full system restart to see if that flushes any cached errors in the routing table.

Any more ideas ??

Regards,