Problems running ASA5510 as 172.x.y.z (final resolution: subnet masks)

[Loxmyth]

On my 5510, NAT rules appear to be needed for successful communication between interfaces at different security levels.
On Ricky's 5540, the security levels alone are sufficient to control flow between these; no NAT rules are required.

I'd say that was the result of nat-control being set differently on the two boxes, except that we're both running firmware versions after nat-control was deprecated.

So... Is there a command that has equivalent effect, that we might have set differently on the two boxes without noticing it in the .cfg dumps? Or might that control still be there but invisible and inaccessible, so our two machines are unavoidably going to behave differently?

(We've been beating our heads against this for the past week, trying to find the point of divergence between our setups. This looks suspiciously likely.)

[tvotna]

@Loxmyth, "nat-control" was deprecated and removed in version 8.3 when NAT code was completely re-written. There is no hidden equivalent or something like this. So, if NAT or PAT is not configured, proper ACLs are configured and routing is configured properly, traffic will pass between interfaces with different security levels in both directions subject to ACLs.

If NAT or PAT is configured (e.g. for subset of interfaces or traffic), things become more complicated. E.g. there is a feature "NAT RPF check" which can prevent communications depending on the NAT type being used (twice vs. manual) and software version. In such cases "packet-tracer" can help with troubleshooting.

[Loxmyth]

Thanks. What I can't figure out is why I'm having such trouble getting a basic "just keep lower security from opening connections to higher security, without explicit NAT" configuration going (and why it's working for Ricky). I'm starting to suspect that my error might not be in the .conf file but in something else.

Unfortunately I haven't found a good example of that minimal configuration in any of the docs or websites I've checked. I'd actually have expected it to be the default behavior of the box, but if so I haven't been able to find my way back to it.

Could I ask a HUGE favor, and ask you to take a look at my configuration and see if you can spot what I've botched?

(Quoting Frances Sternhagen as Dr. Marian Lazarus in Outland: "Such a smart piece of equipment, and a wreck like me trying to run it...")

[tvotna]

Are you asking about outside to inside access or inside to outside or dmz to public, etc?

nat (any,public) source dynamic any interface
nat (any,outside) source dynamic any interface dns
nat (any,DMZ) source dynamic any interface
nat (any,inside) source dynamic any interface

For example, if you remove "nat (any,DMZ) source dynamic any interface" you should still be able to access DMZ 172.17.2.x from inside 172.17.1.y (higher security level to lower security level). To access inside from DMZ you need to permit traffic with ACL.

[Loxmyth]

That's what I would have expected. But that isn't what I'm seeing.
With these NAT rules in place, I can ping or ssh from machines on inside to public (172.17.1.102 to 172.17.3.20).
If I use ASDM to delete the rule for public

ASDM sends the commands
    no nat 1
    clear xlate global 172.17.3.0 netmask 255.255.255.0
and I can no longer ping or ssh to anything on the ..3.0 subnet from machines on either inside (..1.102) or management (..17.100).

It isn't 100% clear whether the problem is that packets aren't getting out, packets aren't coming back, or somehow the target machines are themselves not accepting traffic from anything outside their own 255.255.255.0 masked range. If the latter, it might be that something is wrong with what dhcp is pushing out to them...

I haven't yet put wireshark on this.
ASDM traceroute from inside (or 172.17.1.1)  to 172.17.3.20 without the NAT rule times out.
ASDM tcp ping from inside  to 172.17.3.20 without the NAT rule succeeds
ASDM tcp ping from inside/172.17.1.102  to 172.17.3.20 without the NAT rule times out
ASDM icmp ping from inside to 172.17.3.20 without the NAT rule times out
ASDM traceroute from inside to 172.17.3.20 without the NAT rule times out
ASDM use-icmp traceroute from inside to 172.17.3.20 without the NAT rule times out

If I restore the NAT rule, communication down to public is permitted again.

It really seems to want the explicit NAT. I don't understand why, since everyone agrees that higher-to-lower-security connection is supposed to Just Work.

[Marius Gunnerud]

Is this a lab setup?  Also, you mention the following:

Have you run a packet-tracer when you have removed the NAT rule?  What are the results?

You could also run a capture on the public interface and inside interface simultaneously to see if you see traffic in both direction.

[Loxmyth]

Lab is a decent description. I can run experiments without seriously disrupting anyone's work.

Packet tracing: Let's see. With NAT disabled on public, using a host that I know is up:

ASDM traceroute to .3.20 without specifying source works, of course.
Setting source inside times out.
Setting source .1.1 (inside's address) times out.
Adding use-icmp still times out.

Windows tracert from .1.102 to .3.20: All * * * Request timed out.
Linux traceroute from .1.110: All * * *

Haven't used the capture feature before; I'll try it.

[Loxmyth]

ADSM packet tracer: I'm not 100% sure I'm grokking what this is telling me, but:

Tracing from Interface inside, type ip, from .1.102 to .3.20, with no NAT on public, everything is ok until
    NAT rpf-check, action DROP
    Config: nat(any,inside) source dynamic any interface
There shouldn't be any _outgoing_ NAT from inside to outside... should there?
    Input Interface: inside
    Output Interface: public
    Info: (acl-drop) Flow is denied by configured rule

If I set Interface: public,  otherwise same parameters (which I think simulates what would happen if the packet was just being transferred from inside to public?), all actions are OK and the packet is allowed.

[Marius Gunnerud]

Yeah, that NAT statement there, nat(any,inside) source dynamic any interface, is your issue.  Remove this and you should be good to go.  Also, I highly recommend not using any as the source interface, use specific interface names. Use as specific NAT rules as possible and you will  not run into these types of issues in the future.

[Loxmyth]

But without NAT on inside, I can't ping inside from management...?

Remember, the real problem here is that if I disable all the NAT rules, tcp is failing to make any connections... despite the intended behavior in that case being that things on higher security interfaces can talk to things on lower security interfaces. I can't even http out to the net unless there's a NAT rule for outside. (And wasn't NAT supposed to be protocol-aware so any tcp exchange that went through NAT got its responses converted back?)

I absolutely agree that for my simple case I shouldn't need or be using NAT at all. But it's the only way I've found to make things work vaguely as expected. Goal of this discussion was to figure out why that root problem exists on my box and fix it.

[Loxmyth]

In other words, the configuration attached hereto (same as above but delete all the NATs) does _not_ work on my 5510. And I don't know why not. I've been told that when my config was loaded onto a 5540 it did work without NATs, but I'm not sure we're at exactly the same firmware level.

I'm sure it's something obvious to someone who really understands the ASA. I grok many of the individual pieces but clearly not their interaction.

[Marius Gunnerud]

What type of devices are you pinging towards on the Public network? Are you sure that this is not a routing issue on those devices?

I highly suggest you do a packet capture on both the inside and public interfaces and then compare the two.  If you see packets entering the inside interface, leaving the public interface, but not getting anything in return there is an issue between the public interface and the device you are trying to connect to.  Keep in mind that the capture needs to be set up with the IPs you expect to see.  So if you have NAT configured then on the inside interface you would configure the real source IP and real destination IP, but on the public interface you would configure the translated source IP and the real destination IP.

If you do not have NAT configured when testing you would just use the real IPs for both source and destination on both inside and public interfaces.

But without NAT on inside, I can't ping inside from management...?

You do not need NAT for communication between RFC 1918 IPs, you do need NAT for communication to public IPs from private IPs.  So removing the NAT should not affect communication between networks connected or routed through the ASA.  However you will not be able to connected to an ASA interface that is not the ingress interface.  That is to say you cannot connected to the management interface from a device located on the inside network if that traffic passes through the ASA.  The only way this will be allowed is if the management traffic does not pass through the ASA to reach the management interface.

[Loxmyth]

The devices on Public I'm pinging most often are Raspberry Pis running Raspbian linux. It's certainly possible they're misconfigured and didn't expect to talk to anyone outside their own LAN's address range.

Just tried pinging a different device on public, laptop running antiX linux. With NAT, ping succeeds. With the NAT rule for public disabled, ping fails. Again, it's possible that this laptop Linux is unwilling to listen to anything outside its own LAN, but it's another datapoint.

I can try reversing things, put that laptop on Inside and my working machine on public, and see if that changes matters. Of the two I'd expect Windows to have more problems in networking than Linux, but...

Lemme see if I can figure out packet capture, and I'll try that without NAT.

Yes, understand about connectivity. That's why I currently have the ASA permitting access on both the internal and management interfaces; I'm hardwired to one or the other of those, and I try not to alter both at once so I've got a reliable alternative route.

We all agree that I _shouldn't_ need NAT in this setup. It's certainly possible that the ASA is not at fault. The traceroute results specifically report "(acl-drop) Flow is denied by configured rule", but that may not imply a rule in the ASA's acl.

Lemme see if I can figure out packet capture, and I'll try that without NAT.

[Loxmyth]

I was going to say that the problem with the theory that it's the hosts that are misbehaving is that, in that case, I'm not sure why I need NAT to communicate with the WAN-facing outside. ... Oh. Maybe.

The ASA is plugged into a FIOS ONT (optical fiber). It's not impossible that this is blocking traffic from address ranges other than the one it assigned the via DHCP. That would require the translation from my 172.17 space, just as basic routers (I believe) run NAT to connect their 192.168 addresses to the ISP.

So that one NAT rule, and only that one, would make sense. Yes?

---

BTW, packet capture is starting to -- finally, maybe -- convince me that the problem is in the leaf machines rather than in the ASA.

Question: The interface configuration, when I set the interface address, insists that the subnet mask be 255.255.255.0 when the interface is placed at .x.1. My understanding was that this meant if 172.17.1.x was talking to 172.17.3.y, the packet needed to cross the firewall since they're separate subnets, otherwise the firewall could completely ignore that packet. If so, how would I express a /16 address range (all of 172.17) when setting the interface address?

Should I have done that (however it's done) as a single set of addresses shared by all interfaces, and used the third octet more as a comment indicating which physical subnet the device was on, rather than assigning each interface its own /24 address? I thought ASDM was keeping me from going that direction...

[tvotna]

Hi @Loxmyth . Answering few question here.

1. This is expected behavior of ASA NAT that it drops packets due to NAT RPF check when you remove NAT rule 1 and ping from inside to public:

nat (any,outside) source dynamic any interface dns
nat (any,DMZ) source dynamic any interface
nat (any,inside) source dynamic any interface

In this case "forward flow" matches none rules and "reverse flow" matches 3rd rule and ASA doesn't like it. But actually, this concept is not easy to explain taking into consideration that ASA is a stateful firewall (why should "forward" and "reverse" flows be separated?). Also, IOS routers don't have anything like this at all. The reason behind "NAT RPF check" is how ACLs work on ASA. For example, when you allow traffic from outside to internal Web server, you specify real-IP (private IP) of the server in the outside ACL, rather than its public IP. Even if you make an error and permit whole subnet range, NAT RPF check will prevent communications from outside to inside:

nat (inside,outside) source dynamic any interfacenat (inside,outside) source static obj-10.1.1.1 interface service obj-tcp-src-443 obj-tcp-src-4443access-list outside_in permit tcp any 10.1.1.0 255.255.255.0 eq 443access-group outside_in in interface outside

So, NAT RPF check is a security mechanism (sort of).

2. Yes, typically NAT is needed to access outside to translate private space to addresses which upstream devices understand and have routing to. You can check what IP address is assigned to your outside interface with "show int ip brief". You should see default route assigned by DHCP too. Single rule should work just fine:

nat (any,outside) source dynamic any interface dns

Traffic between other interfaces, e.g. inside to public or public to inside won't be NATed and everything should work if devices have proper routing configured via ASA.

3. Packet capture can be used from CLI (also packet-tracer tool), e.g.

capture cap-in interface inside match tcp host 10.1.1.1 host 192.0.2.10 eq 23capture cap-out interface outside match tcp host 10.1.1.1 host 192.0.2.10 eq 23
show capture cap-inshow capture cap-out

Capture is bi-directional.

4. In routed firewall mode ASA won't allow you configure overlapping address spaces. It will complain "two interfaces cannot be on the same subnet".