Problem with NAT traversal of two ASA firewalls

[GRANT GATHAGAN]

I have an odd situation involving two ASA firewalls.
Firewall 1 is our main firewall, an ASA5585-X. Firewall 2 is an ASA5506

For reasons that aren't germane to this problem, one of our research groups is required to place their server and its attendant NASs behind their own firewall within our institution. That's firewall 2.

Firewall 2 is set up to limit the server's outgoing access to a server at another research site via SSH and Red Hat's update sites.
The NASs outgoing access is limited to Synology's update sites.
Additionally, I've allowed DNS and NTP access from those devices to our internal DNS and NTP servers

From outside of firewall 2, SSH access to the server is limited to the computers of several researchers, which are on our institution's private network.
Likewise, HTTPS access is allowed from those same researchers to the NAS units.

The outgoing traffic from firewall 2 to the update sites then goes through our main firewall, firewall 1, along with the majority of other outgoing traffic from our organization.

The researchers now need to add SSH access to an additional remote site.
In the interest of security, the new remote site would like to see traffic from us on a unique public IP address, as opposed to the public IP address of the outside interface of firewall 1.

Accordingly, I created a static NAT on firewall 1 for the outside interface of firewall 2.
I didn't set up any access rules, just the NAT.

Currently, the server cannot communicate with any of its allowed targets, but the NAS units can.
The logs show connections being set up on firewall 1, but they time out after 30 seconds.

Given that the NAS connections are made successfully, I think my approach is logical, but clearly I am missing something.

Any advice is welcome.

[nspasov]

This could be ACL, proxy-arp, routing, etc. but it is hard to say as there are many moving parts. Additional information and perhaps a basic diagram/sketch would be a good start. The output from packet-tracer from both firewalls simulating the end-to-end connection could possibly shine the cause of the problem. 

Thank you for rating helpful posts!

[MHM Cisco World]

1- both FW must have ACL allow traffic to Server connect behind FW-2

2- both FW need NAT 

FW-1 from it out interface  to FW-2 out interface 

FW-2 from it out interface to internal server IP

3- try use different ssh port dont use port 22

MHM

[Aref Alsouqi]

You need to allow this traffic along the whole path so it needs to be allowed on firewall 1 and firewall 2 before it gets routed to the server that is sitting behind firewall 2. Also as already mentioned you need another NAT rule on firewall 2 to untranslate this traffic and send it to the server, if not firewall 2 won't know that this traffic is meant to be sent to the server. Please share the respective sanitized configs from both firewalls for review.

[GRANT GATHAGAN]

Aloha, all

Attached is a basic diagram of the locations of the devices in question, as well as an anonymized copy of the configuration of firewall 2.
There are 4 NAS devices behind the firewall, but I've not included the additional NAS devices in the diagram to keep it small.

Understand that the configuration of firewall 2 has been working properly for at least 2 years.
The only change made was giving the public interface of firewall 2 a NAT'd public address of its own on firewall 1.
There have been no changes to the configuration of firewall 2.

I could understand the issues raised in your responses if *nothing* worked after the change, but the fact that all 4 NAS devices can check for updates successfully has me scratching my head

It's always been my understanding that, absent any rules to the contrary, data flow that is initiated from a lower security network to a higher security network and its responses are allowed by default. That was the reason I didn't create any additional access rules in firewall 1.
All data flow to the outside world that traverses firewall 1 from behind firewall 2 is initiated by the server or the NAS devices. Nothing is initiated from outside of the institution.

Regards,
Grant

[MHM Cisco World]

the config is only for second ASA and it seem you dont allow traffic from ANY to server/NAS in your OUTside ACL ?

MHM

[GRANT GATHAGAN]

Correct, the configuration is from the 2nd ASA.
There is traffic allowed from our institution's private network (private network 1) to the server and NAS, but nothing that originates from the outside world.
All traffic that traverses firewall 1 to the Internet is initiated by either the server or the NAS.

[MHM Cisco World]

Sorry continue with @Aref Alsouqi 

I am busy these days 

Thanks 

MHM

[Aref Alsouqi]

Could you please share the bits of config you added rather than the whole config? it's a bit tricky for me to review the anonymized config file as it doesn't show any IP addresses for reference nor on the topology you shared.

[GRANT GATHAGAN]

Aloha, Aref
I've attached revised documents. Hopefully, that will help.
The configuration that I shared was never changed. It's the configuration for firewall 2.

The only configuration change was done on firewall 1, which consisted of establishing a static NAT for the outside interface of firewall 2 and one of our public IP addresses:

object network firewall 2
host 1.1.1.20

object network firewall 2
nat (inside,outside) static <public IP>

Prior to that, any traffic from firewall 2 that reached the Internet was using the IP of the public interface of firewall 1; the same as all of the other devices on our network that don't have a static NAT.
In both cases, the only rule at work there is the implicit rule on the inside interface that allows all traffic from a higher security level to a lower security level.

[Aref Alsouqi]

Hi Grant. You are right on the fact that if the traffic comes from a higher to a lower security zone it will be allowed by default and the return traffic will also be allowed due to the stateful inspection. However, this is the case only if you don't have an access list applied to the higher security zone. For instance, if you have an access list applied to the inside zone that has a security level of 100 and this access list happen to allow traffic only on port 80/tcp and 443/tcp, anything else will be denied, although the traffic flow comes from a higher zone to a lower one.

I tried to take a look at the configs shared and from the NAT perspective I don't think there are any issues, you have an auto NAT on firewall 2 that translates anything coming from inside that does not have a static NAT defined, and that translation will be to firewall 2 outside interface IP. Then you mentioned that you have created a static NAT rule on firewall 1 that translates anything coming with the source IP 1.1.1.20 (firewall 2 outside interface) to the new public IP of firewall 1.

That should be it from the NAT perspective. Now from the access lists perspective you should allow this traffic on both firewalls in outbound direction (from inside to outside). Was this done? if so, could you please run two packet tracer from both firewalls simulating the traffic from the NAS or the server going to somewhere allowed to the internet and share the results for review?

Example on firewall 2:
packet-tracer input inside tcp < NAS or server IP > < a fake source port > < destination IP > < destination port >

Example on firewall 1:
packet-tracer input inside tcp 1.1.1.20 < a fake source port > < destination IP > < destination port >

[GRANT GATHAGAN]

Aloha, Aref
As the firewall 2 configuration should indicate, there are access lists as you describe.
Examples:
access-list inside_access_in extended permit tcp object R640 object cdn.redhat.com
access-list inside_access_in extended permit tcp object synology_1 object global.download.synology.com
There are no similar rules on firewall 1, since the implicit rule suffices.
Attached is a zip file with screenshots of every packet-tracer test I ran on both firewalls.
On firewall 2, I tested to see if the server could reach the update sites via https and the sole remote site via SSH.
I then tested https and SSH access to 8.8.8.8. Results were as expected
I repeated those tests with the NAS.

I then set up the NAT again on firewall 1 and tested firewall 1 for both SSH and HTTPS access to numerous sites.
All the tests were successful.
So taken individually, each firewall predicts success, but the combination fails.
Sadly, there's no way to run a packet-tracer test from end-to-end, through both firewalls.

[Aref Alsouqi]

At this point I would recommend running some packet capture on firewall 2 outside interface and firewall 1 both inside and outside interfaces while you are doing a test that should work but it's not working. This will allow us to see where the traffic gets blackholed. Also, please run the CLI command "show conn" and filter per the destination IP of the test you are doing as well as "show nat" and filter per the destination IP. Please share the output of all these troubleshooting commands for review.

[GRANT GATHAGAN]

Aloha, Aref
Thanks for the suggestion.
It'll take a few days to set this up, so I probably won't have results until next week.

Regards,
Grant

[Aref Alsouqi]

No worries Grant and you are welcome.