10-19-2013 10:13 AM - edited 03-11-2019 07:53 PM
Hi All,
We have a strange problem with a PIX 515E running 8.0 (4).
BACKGROUND
The Pix is configured with two ISP connections, primary and secondary. We use a simple SLA monitor to check for reachability of the ISP 1 gateway, if this goes down ISP 2 comes up and internet is restored. In addition there are a few site to site VPNs terminating on the Pix and some ad hoc dynamic VPN connections.
PROBLEM
When the primary ISP 1 failed the other day all was well and the Pix transferred to ISP 2 almost in a heartbeat as it is supposed to do. The site to site VPN also changed over without an issue. However, when the primary ISP came back up the secondary route did not clear down in the routing table as evidenced by the "sh route" output. Furthermore the site to site VPN remained connected to the secondary connection. We cleared this fault by doing a simple "shut" and "no shut" on the secondary interface.
Has anyone any idea why this is happening ?
Thanks.
10-19-2013 11:04 AM
Looks like SLA monitoring is not working as expected for you. Make sure that it is configured as per the following doc:
If it is configured correctly but is still not working as expected then check to see if SLA monitoring is operational when the primary route comes back up after failing using "show sla monitor operational-state".
10-20-2013 12:10 PM
If the secondary ISP came up, I believe that the sla monitoring is working fine. When the issue happens next, and the primary does not come back up. Try to ping the ip address which is being tracked sourced from the primary ISP.
if the ping works fine, then the primary should come back up as well. Also, get syslogs to see if you see something unusual.
10-22-2013 01:06 PM
Hi All,
Sorry for delayed response. We have run the sh ip sla command and here is the output.
firewall# show sla monitor operational-state 123
Entry number: 123
Modification time: 20:44:07.659 BST Sun Sep 8 2013
Number of Octets Used by this Entry: 1300
Number of operations attempted: 0
Number of operations skipped: 0
Current seconds left in Life: 0
Operational state of entry: Inactive
Last time this entry was reset: Never
This suggests that we have an error with the SLA monitor, but as we said in the original post it fails over perfectly with a failure of the primary connection, but is not flipping over when the primary connection comes live. We have been through the configuration guide and verified all steps which are fairly simple.
Yes, after the primary connection comes back up we can ping the IP address that is being monitored using the primary interface as source. Also, interestingly if we set a ping going to both IP gateways, ie primary and secondary we even see the primary response time of the echo change when the primary comes back up. We can tell because the primary is a dedicated fibre whereas the secondary is good old ADSL. It is however still showing the ISP2 within the routing table. This tends to point towards the routing table not flushing the secondary connection after a fail over and perhaps nothing to do with the SLA..
Anyone got anymore suggestions?
Regards,
10-22-2013 02:08 PM
Can you please share the configuration.
Known issues:
Problem:
SLA monitoring does not work after the ASA is upgrade to version 8.0.
Solution:
The problem is possibly be due to the IP Reverse-Path command configured in the OUTSIDE interface. Remove the command in ASA and try to check the SLA Monitoring.
Bug ID CSCtc16148
http://tools.cisco.com/squish/6B83a
10-22-2013 04:17 PM
Hi All,
We need to make a correction to our earlier post.. We said
"Yes, after the primary connection comes back up we can ping the IP address that is being monitored using the primary interface as source"
In fact we can not ping the IP gateway from the primary interface after the connection has come back up. The routing table still shows the route as the secondary connection thereby requiring an admin shutdown of the secondary interface which then puts the primary interface back into the routing table.
Here is the configuration relating to the IP SLA monitor..
sla monitor 123
type echo protocol ipIcmpEcho XXX.XXX.XX.XXX interface Outside
num-packets 3
frequency 10
sla monitor schedule 123 life forever start-time now
track 1 rtr 123 reachability
route Outside 0.0.0.0 0.0.0.0 X.X.X.X 1 track 1
route ISP2 0.0.0.0 0.0.0.0 X.X.X.X 254
Any ideas anyone ???
Regards,
10-22-2013 04:24 PM
Ok, I am not sure if you checked my post can you please get me the configuration so we can check.
10-22-2013 04:48 PM
Plus, you can not recover the main link if the IP address that you set to monitor is not available so I am not sure how you are recovering
10-22-2013 04:52 PM
Please run the next command:
show run ip verify reverse-path
If the command is in place just remove it
enable
config t
no ip verify reverse-path interface Outside
10-23-2013 01:41 PM
Did the information given help out; please remember to reply "Answered" if the information given to you resolved your problem.
10-24-2013 04:13 AM
Hi Jumora,
Unfortunately the problem is not resolved.
ip verify reverse-path is not running on the outside interface. Here is a further screenshot of the results of the SLA output:
firewall# sh sla monitor operational-state 123
Entry number: 123
Modification time: 23:41:42.374 BST Tue Oct 22 2013
Number of Octets Used by this Entry: 1480
Number of operations attempted: 13064
Number of operations skipped: 0
Current seconds left in Life: Forever
Operational state of entry: Active
Last time this entry was reset: Never
Connection loss occurred: FALSE
Timeout occurred: FALSE
Over thresholds occurred: FALSE
Latest RTT (milliseconds): 1
Latest operation start time: 11:58:52.372 BST Thu Oct 24 2013
Latest operation return code: OK
RTT Values:
RTTAvg: 1 RTTMin: 1 RTTMax: 1
NumOfRTT: 3 RTTSum: 3 RTTSum2: 3
This all seems fine and as we have said the IP SLA fails to the secondary ISP without an issue, it just will not reinstate the primary ISP connection after it becomes available again. The only way we can get this to flip back to the primary connection is to do a "shut" and "no shut" on the secondary interface. The "sh route" output verifies that this is happening which seems to point towards the secondary route not getting cleared from the routing table correctly.
Any more ideas?
Regards,
10-24-2013 10:29 AM
Can you send me the show run ICMP?
Also, can you try changing the IP address that you are using to monitor for 4.2.2.2 just as a test.
The strange thing is that you state that the main link recovers but this is not possible if the address that you are using to monitor is never available.
If you cannot change the monitoring address on the SLA configuration, please check with a packet-tracer how the ASA would consider to route to that destine address from a LOCAL address (PC).
packet-tracer input inside icmp
If you have any VPN configuration please confirm that you do not have a match for ip any any or icmp any any as this could affect SLA monitoring.
10-25-2013 02:39 PM
Hi Jumora,
Here are the answers to your questions.
firewall# sh run icmp
icmp unreachable rate-limit 1 burst-size 1
icmp deny any echo Outside
icmp permit any echo-reply Outside
icmp deny any echo ISP2
icmp permit any echo-reply ISP2
Yes, the primary link will recover when we shut down ISP2, which is the secondary connection. The same results were obtained using 4.2.2.2
We do have VPNs on this device and have checked the configuration which all seems fine, no ip any any or icmp any any..
It may be that this problem is not with the SLA monitor as we seem to get all the right output from the show monitor. I think now the problem maybe with the static routes which are not clearing the secondary route without shutting down the secondary interface. The question is why this is happening????
Any more thoughts??
10-25-2013 03:32 PM
Answer, remove the next line:
icmp deny any echo Outside
10-25-2013 04:05 PM
Hi Jumora,
Just logged in and removed "icmp deny any echo Outside" and it didn't work. This is a real head scratching problem as we configure a lot of these failovers and have not had this problem before. I am going to do some more debug tomorrow and see if I can see anything on the routing side that may be causing the routing table to retain the secondary ISP route, in particular I am going to look at the DHCP config on the Outside interface, although it picks up a static IP. Although rarely required I am thinking of a full system restart to see if that flushes any cached errors in the routing table.
Any more ideas ??
Regards,
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide