ASA 5510 - OSPF over IPsec tunnel causing tunnel to flap

[tfabian-smith]

Hey all. I'm attempting to get OSPF running over an IPsec tunnel between two ASA 5510s. However, I seem to be running into a catch-22 kind of scenario. I'm guessing there is something really simple I'm missing here and I hope someone can provide some insight as to what that could be.

Please note, I am currently setting this up in a test network. There are two "sites" in the setup and two routers acting as a fake ISP between the two ASAs.

This is the current physical topology:

HP 5406 --> ASA-1 inside

ASA-1 outside --> Ubuntu VM router 1 eth0

Ubuntu VM router 1 eth1 --> Ubuntu VM router 2 eth1

Ubuntu VM router 2 eth0 --> ASA-2 outside

ASA-2 inside --> Enterasys N3

The two Ubuntu VMs acting as routers are the fake ISP. SITE-1 consists of the HP 5406 and ASA-1. SITE-2 consists of the Enterasys N3 and ASA-2.

Networks:

10.1.0.0/20 are SITE-1 internal networks

192.168.10.0/29 is the SITE-1 to ASA transit network

  5406: 192.168.10.1

  ASA-1 inside: 192.168.10.2

172.16.100.0/24 is SITE-1's management network

133.38.38.0/29 is the ASA-1 outside to Ubuntu VM router 1 transit network

  ASA-1 outside: 133.38.38.2

  Ubuntu VM router 1: 133.38.38.1

200.200.200.0/29 is the transit network between the two Ubuntu VM routers

  200.200.200.1 is Ubuntu VM router 1

  200.200.200.2 is Ubunti VM router 2

133.39.39.0/29 is the ASA-2 outside to Ubunti VM router 2 transit network

  ASA-2 outside is 133.39.39.2

  Ubuntu VM router is 133.39.39.1

172.16.200.0/24 is SITE-2's management network

192.168.20.0/29 is the transit network between ASA-2 and the N3

  192.168.20.1 is the N3

  192.168.20.2 is the ASA-2 inside interface

10.2.0.0/20 are SITE-2 internal networks

Both SITE-1 and SITE-2 are in OSPF area 0.0.0.0. The HP 5406 has router-id 0.0.0.1. ASA-1 is router-id 0.0.0.2. The N3 is router-id 0.0.0.3 and ASA-2 is router-id 0.0.0.4.

I have been able to form the IPsec tunnel between the two ASAs and set static OSPF neighbors between the two ASA outside interfaces. As long as I am advertising the transit network associated with the outside interface with OSPF, am redistributing static routes and have OSPF static neighbors set on each ASA pointing at the other ASAs outside interface I am able to form a neighbor relationship between the two ASAs and obtain routes. However, this setup appears to cause the IPsec link to flap, which in turn causes the OSPF neighbor relationship to time out and drop routes. This happens about every 3 to 5 minutes.

I've found a few forum posts/articles stating that you should statically route between the two tunnel endpoints and not advertise the tunnel transit networks between the ASAs in OSPF (or other dynamic routing protocols). But when I do it this way, the ASAs never see each other as OSPF neighbors, even when set as static neighbors. I can see LSA1 entries for the far ASA on each ASA when it's configured this way, but relationship never forms. However, when I stop advertising the networks with the tunnel endpoints between the ASAs and I set the route between the endpoints static on each one then the IPsec tunnel is solid and doesn't flap.

These are the forum posts/blog entries I'mr referring to specifically:

https://supportforums.cisco.com/t5/vpn/tunnel-flapping/td-p/2920076 

https://supportforums.cisco.com/t5/security-documents/eigrp-and-ospf-neighbors-flap-over-the-gre-tunnel/ta-p/3131522

I've attached sanitized configs for each test ASA. I'm also happy to provide whatever other info might be useful in troubleshooting this issue.

I appreciate any assistance! Thanks!

[tfabian-smith]

I had forgotten - and should note - that apparently when you remove a network from being advertised via OSPF on the ASAs it will also remove any related static neighbor settings. It took me a minute to figure this out as the static neighbor still appears in ASDM but not in the actual config. Also, you cannot set static neighbors without advertising the network you're setting the static neighbors for. (I'm running ASA 9.1(6) and ASDM 7.9(2)152 on both ASAs.)

So, that would explain why the neighbor relationship isn't forming when configured with static routes between the tunnel endpoints and I'm not advertising the network the tunnel endpoints are in via OSPF.

This is a quote from one of the forum posts I linked to in the original post:

"If the tunnel interface learns that the best path to the tunnel destination is through the tunnel itself, the interface shuts down temporarily."

Again, it seems like a catch-22. I can't set static neighbors without advertising the tunnel endpoint network, but I can't form neighbor relationships between the ASAs without doing so.

[tfabian-smith · Accepted solution]

Okay, I got it figured out. 

The solution was to add a static route on both ASAs pointing at the outside interface of the other ASA.

So, from ASA-1

route outside 133.39.39.2 255.255.255.255 133.38.38.1 1

and from ASA-2

route outside 133.38.38.2 255.255.255.255 133.39.39.1 1

Originally, I had set up static routes this way when doing the original configuration and setting up basic connectivity:

ASA-1:

route outside 133.39.39.0 255.255.255.255 133.38.38.1 1

ASA-2

route outside 133.38.38.0 255.255.255.248 133.39.39.1 1

When the routes using the network range were in place, OSPF would not form neighbor relationships between the ASAs. When I removed the static route the neighbor relationship would form but cause the IPsec tunnel to flap. Re-adding the static routes by pointing directly at each outside interface appears to have solved the problem. The IPsec tunnel has been stable and OSPF is working correctly between the ASAs.