I'm not a novice in ASA, but this problem got me baffled. I'll start without too much of a technical detail: I have two ASAs with an IPSEC L2L VPN between them, like this:
LAN1 - inside - ASA1 - outside - (public internet) - outside - ASA2 - inside - LAN2
The crypto map on each ASA has a set reverse-route statement, so that ASA1 has a static route to LAN2 pointing to its inside interface, and ASA2 has a static route to LAN1 pointing to its inside interface. Each ASA also has ip verify reverse-path on each interface. And of course, there's a NAT exempt rule on each ASA for the networks participating in the VPN.
So far, so good. Hosts from LAN1 can reach hosts from LAN2 and vice versa (including the L3 switch in LAN2 acting as a DHCP relay on behalf of the local clients while the DHCP server is the Windows domain controller located in LAN1). I can successfully SSH to the ASA2 inside interface from LAN1 (there's no management access from outside). All that works like charm.
Then I want to send syslog and netflow from ASA2 to an NMS in LAN1. Since there's a static route to LAN1 pointing to the inside interface, that's the destination interface for logging host and flow-export destination commands. Still, the NMS doesn't receive anything from ASA2. I try to ping the NMS from ASA2. Ping fails, and I watch it failing while being connected via SSH from the NMS!
Looks like the time for packet-tracer. I try to reproduce my ping, and here's the phase where it fails:
Phase: 4 Type: ROUTE-LOOKUP Subtype: input Result: ALLOW Config: Additional Information: in 10.45.254.1 255.255.255.255 identity Result: input-interface: inside input-status: up input-line-status: up output-interface: outside output-status: up output-line-status: up Action: drop Drop-reason: (rpf-violated) Reverse-path verify failed
As I've mentioned, I'll give only as much detail as necessary, and that's what necessary: 10.45.254.1 is the address of the ASA2 inside interface. Then, since I know that any other address from inside has no problems going into the tunnel, I run packet-tracer again, but with the source address changed to 10.45.254.2, and see the difference.
<the first 3 phases identical> Phase: 4 Type: ROUTE-LOOKUP Subtype: input Result: ALLOW Config: Additional Information: in 10.45.254.0 255.255.255.252 inside Phase: 5 Type: CONN-SETTINGS Subtype: Result: ALLOW <other phases until 17> Result: input-interface: inside input-status: up input-line-status: up output-interface: outside output-status: up output-line-status: up Action: allow
Now, shrinking that to a single line, the self-originated traffic obviously doesn't pass the RPF check because of this:
in 10.45.254.1 255.255.255.255 identity
The incoming traffic has this:
in 10.45.254.0 255.255.255.252 inside
The firewall's own interface's IP address is not deemed reachable from that interface, but from some kind of loopback interface. And that would make sense if there were an incoming packet with that address as a source! But when the same check is applied not to the incoming, but to the self-originated traffic, that starts to look really, really weird! I haven't seen anything like that in other scenarios (if an RPF check like that were applied to self-originated traffic all the time, I guess a firewall would not be able to originate any traffic at all!)
- what makes the ASA apply an RPF check to its locally originated traffic in this particular case?
- how can that be avoided/turned off/bypassed (without affecting RPF checks for actual incoming traffic)?
The ASAs are two 5505s, version 9.1(7)29.
A quick update. Waking up in the morning with a fresher brain I realised that the problem with packet-tracer was just it: problem with packet-tracer. Since I had to specify the incoming interface explicitly (and there's no way to specify identity), it treated the source address as incoming, and not as self-originated. I just used the wrong tool. You can't use packet-tracer for debugging self-originated traffic.
So, the problem to solve is how to put self-originated traffic into the tunnel. Theoretically, it seems to satisfy the crypto map & NAT exemption, unless I'm missing something, but for some reason it doesn't get there...
So. The issue is solved. Funny thing is that the ASA config was all right.
1) Testing outbound connectivity by ping turned out to be not a good idea because the AV network screen on the NMS was blocking incoming ICMP requests. Having changed that setting, the ASA2 started to successfully ping the NMS.
2) What I failed to mention was that there had recently been connectivity issues between ASA1 and ASA2. After the connectivity was restored, the tunnel came up triggered by ping probes and SNMP polls from the NMS, and then I started to check for syslog messages and netflow traffic on the NMS - and found none, despite the fact the tunnel was functioning. Both logging host and flow-export destination were already in the config at the time of the outage. My guess is that ASA could have started to route them via the default route when the reverse-route to LAN1 became unreachable, and "didn't notice" when it got installed in the routing table again, or something like that - in other words, that when the tunnel came back up, all the "interesting" traffic started to flow via the tunnel, except for the traffic destined to logging host and flow-export destination. I wasn't blindly trusting the NMS software but trying to capture the corresponding traffic via Wireshark, and it was dead silent until the fix.
3) The fix was ridiculously simple. I removed the flow-export event-type all destination statement from policy-map global-policy class class-default and the global flow-export destination statement, and then immediately reintroduced them (in the reverse order). Immediately the netflow traffic showed up in Wireshark and in the NMS.
4) Fixing syslog turned out a bit trickier - simply removing and re-entering logging host inside didn't seem to help (in fact, I tried to do that before my successful try on netflow). What did help was: removing logging host inside; entering logging host outside with the same IP; removing logging host outside; entering logging host inside, and voila - I got a snowfall of syslog messages on my NMS.
The bottom line: after the "fix" I ended up with exactly the same config that was there when I discovered the issue. I didn't change anything elsewhere, either, except for allowing incoming pings on the receiving side solely for testing purposes. I'd like to know how to make ASA recover from tunnel outages by itself, though...