Static NAT issues on ASA 5510

[aubreystarr]

I have a new set up in our production environment that had Static NAT issues that turned out to be a bug in the version of IOS we were running on the new (to us) used equipment we bought. So we upgraded and that fixed the issue. The next hurrdle to overcome was the supernetted IPSEC tunnel between our office PFSense router and the new/used ASA 5510s. Resolved that only to find that VLANs on our 4507s got corrupted. Fixed that now only to find the static NATs are not working... AGAIN!!! Grrrr.

I have looked over the Static entries and the ACLs and everything looks fine. Here are some samples that are representative of most of them (we have A LOT of static NATs to production systems). I have also looked at the IPSEC tunnel settings & there's nothing should impact the external static NATs. The fixed VLANs have the 4507s doing internal routing again, so that would seem to have a positive impact, since they weren't when the vlans were corrupted. Any ideas, suggestions or things I have over looked to check, please let me know.

4507::Catalyst 4500 L3 Switch Software (cat4500-IPBASEK9-M), Version 12.2(53)SG4

5510:Compiled on Mon 02-Nov-09 21:22 by builders

System image file is "disk0:/asa805-k8.bin"

SSH to 4507 switch

Static NAT

match tcp inside host XX.XX.XX.4 eq 22 outside any

    static translation to XX.XX.XX.6/22 -<-- Public Address

    translate_hits = 0, untranslate_hits = 18

ACL:

access-list 101 line 5 extended permit tcp host XX.XX.XX.34 eq ssh host XX.XX.XX.6 eq ssh (.34 address is our office public IP)

access-list 101 line 3 extended permit tcp host XX.XX.XX.34 eq ssh host XX.XX.XX.4 eq ssh (hitcnt=0) 0xc10f694d (I even tried an ACL to the prive address, but I don't think this is necessary)

ssh XX.XX.XX.34 255.255.255.0 outside

This was working after the IOS upgrade, but now I can not even ping the public .6 address, where I could ping and ssh to it before fixing the IPSEC tunnel and vlan on the 4507.

________________________________________________________________________________________

WWW to web based client service

Satic NAT

match tcp inside host XX.XX.143.34 eq 443 outside any

    static translation to XX.XX.XX.211/443

    translate_hits = 0, untranslate_hits = 0

  match tcp inside host XX.XX.143.34 eq 80 outside any

    static translation to XX.XX.XX.211/80

    translate_hits = 0, untranslate_hits = 0

ACL

access-list 101 line 8 extended permit tcp any eq www host XX.XX.XX.211 eq www (hitcnt=0) 0x1306dc1c

access-list 101 line 9 extended permit tcp any eq https host XX.XX.XX.211 eq https (hitcnt=0) 0xd44f9116

__________________________________________________________________________________________

Thanks in advance for any help!

[Jon Marshall]

Aubrey

the acl lines don't make sense ie. for example

access-list 101 line 8 extended permit tcp any eq www host XX.XX.XX.211 eq www

what the above line says is allow any host with a source port of 80 to the .211 server on port 80. But the source port won't be 80, it will be a random port. You have the same problem with all your acls ie. you are saying the source and destination ports must be the same but they very rarely are with a TCP connection.

it should read -

access-list 101 line 8 extended permit tcp any host x.x.x.211 eq www

which says allow any host with a random source port to the .211 server on port 80. This is assuming the web server is actually .211 ?

Also it's not clear what direction etc.things are going in. So -

1) can you amend your acls as covered above

2) can you post the static config lines relevant to the above rather than the static output you have

3) can you clarify what direction this traffic is going in ie. from outside to inside or inside to outside in relation to the ASA interfaces.

4) finally can you confirm that access-list 101 is the acl attached to the outside interface of the ASA ? If not where is it applied ?

Jon

[aubreystarr]

Hi Jon,

Fancy meeting you here )

I can see te logic there, but these NATs worked before. I didn't change any thing. Anyways I tried adjusting the ACL as you suggested and still no go. Here are the configs you asked for taken from show running-config:

static (inside,outside) tcp OUT.SIDE.IP.6 ssh IN.SIDE.IP.4 ssh netmask 255.255.255.255

access-group 101 in interface outside

access-list 101 extended permit tcp host OUR.PUB.IP.34 host OUT.SIDE.IP.6 eq ssh

The above ACL adjusted as you suggest

I can't test any web services without a maint. window to take the current PFSense router off line, but the above I can test with now.

Cheers,

Aubrey

[Jon Marshall]

Aubrey

I am really confused.

The above is for connecting remotely to your office and using ssh to get to the 4500 ?

If so .34 is your public IP, but what do you mean by that ? I usually understand that to mean it is assigned to the firewall but your config suggests this is an IP assigned to a computer on the internet, perhaps your home PCs so you can remotely log in.

Also the static -

.6 -> .4   is the x.x.x.4 a private IP and the x.x.x.6 a public IP ?

your acl is saying -

allow the host with a public IP of .34 to access  x.x.x.6 using ssh. The ASA then translates this to x.x.x.4 and then presumably forwards it to the 4500 ?

So you can see that .34 must be remote to the ASA.

Can you clarify in more detail exactly how this is meant to work ?

Jon

[aubreystarr]

Yes .34 is the public (WAN) IP address of our office PFSense router/firewall. The 4500 switch and the ASA are at the colo data center, not at our office. The .4 is the IP of the switch that is attached directly to the inside interface of the ASA. The .6 is the out side routable address I want to translate to so I can remote to the switch. This was all working before though... I changed nothing.

[Jon Marshall]

Ahh, that makes sense now.

okay couple of things -

1) now you have amended the acl when you try to connect do you see any hits on the acl 101 line ?

2) do you have a default-route on the 4500 pointing to the inside interface of the ASA ?

Jon

[aubreystarr]

Hi Jon,

1) Yes, I see the same number of hits on both the ACL and the static NAT.

2) No, the default route currently points to the production PFSense router, which we want to go away and the ASA is supposed to replace. It was pointing at the ASA during previous testing, and on Sunday night when I tried to cut over. At both of those times the ACL was the old one... but that old one was working in previous testing and also receiving hits.

4507 Ip routes:

Gateway of last resort is XX.XX.132.1 to network 0.0.0.0 <--- 132.1 PFSense Router inside LAN interface

     XX.XX.XX.0/X is variably subnetted, 4 subnets, 2 masks

S       XX.XX.96.0/23 [1/0] via XX.XX.132.1 <--- Route to office IPSec tunnel

C       XX.XX.138.0/24 is directly connected, GigabitEthernet4/1 <--- 138.0 is the direct connection to ASA

C       XX.XX.136.0/24 is directly connected, Vlan30

C       XX.XX.132.0/23 is directly connected, Vlan1

S*   0.0.0.0/0 [1/0] via XX.XX.132.1 <--- 132.1 PFSense Router inside LAN interface

I can see where this is going though... and unless I miss the point, I can see that the default route is an issue. That was set correctly during my attempt to cut over on last Sunday night. To change that will require another maintenance window though, because it will disprupt our public facing services.

[Jon Marshall · Accepted solution]

Aubrey

I'm not sure how that acl line worked before we amended it. It may be that there was another line in the acl that allowed it through.  Any connection to a service such as ssh/telnet/http etc will always have a random source port and not the same port as the destination port.

It might we worth looking at the whole acl.

As for the problem at hand, yes if the 4500 does not have a default-route back to the ASA then it won't work. You could do a few temporary things -

1) because you know the address of the source for ssh you could add a host specific route on the 4500 for that address pointing to the ASA inside interface. This wouldn't solve your web problem though and is a bit tacky.

2) you could add this to your ASA config -

nat (outside) 1 0.0.0.0 0.0.0.0 outside

global (inside) 1 interface

what this will do is translate all internet source IPs to the inside interface IP of your ASA. Then the 4500 because it is on the same subnet will know where to send the traffic back to.

However that NAT statement is too general and might mess up other things on your firewall so please don't do it. We could look to refine it down just to get the things you need working but i would need to see the whole ASA config.

Jon

[aubreystarr]

Hi Jon,

I did the below just to test if the NAT was working at all with the new ACL, and it is. I will adjust my ACLs accordingly and try the cutover again during the next maintenance window. Thanks again!

jon.marshall wrote:
1) because you know the address of the source for ssh you could add a host specific route on the 4500 for that address pointing to the ASA inside interface. This wouldn't solve your web problem though and is a bit tacky. 
Jon

Cheers,

Aubrey

[Jon Marshall]

Aubrey

No problem, at least you know what the issue is now.

Jon

[aubreystarr]

Hi Jon,

The cut over did not go well last night. IPSec tunnel was working. VLANs were working. The ssh NAT to the switch was working. What didn't seem to be working was PAT, which also was working before. I say it was PAT because the ASA could see the internet, but the switches could not. I may have gotten it working if I had some more time, but I had to comply with the timimg of the maintenance window, and so with 30 minutes to go I had to choose between continuing to troubleshoot or rolling back. I couldn't take the chance that 30 minutes wasn't enough time to figure it out, so I rolled back. I removed the "tacky" route you suggested that I had used for testing the ssh NAT to the switch AND the NATs to 2 servers that weren't yet live on the existing router, and the ssh NAT to the switchstill worked after that. When I removed the the global settings you suggested however:

no global (inside) 1 interface

That stopped the ssh NAT to the switch from working again, and the switches still could not see the internet. PAT was one of the first things I set up and had working in my initial testing.

nat (outside) 1 0.0.0.0 0.0.0.0 outside

global (outside) 1 interface

The above is the setting that is in there, which looks right to me. The switches were definately pointed at the inside interface as the default gateway, and could both ping the inside interface on the ASA. I could see the office over the IPSec tunnel, and the office could see the servers on the switches, so traffic was passing through the ASA.

If you don't mind I am going to send you the configs on both the ASA and the 4507s in a private message. Thanks!

Cheers,

Aubrey

[Jon Marshall · Accepted solution]

Aubrey

removed the the global settings you suggested however:

no global (inside) 1 interface

That stopped the ssh NAT to the switch from working again, and the switches still could not see the internet. PAT was one of the first things I set up and had working in my initial testing.

nat (outside) 1 0.0.0.0 0.0.0.0 outside

global (outside) 1 interface

You were never meant to use those NAT settings. As i said in my last post it would be fine for a test but it wouldn't work when you cutover. I said we could refine it but as they are  -

nat (outside) 1 0.0.0.0 0.0.0.0 outside

global (inside) 1 interface

means nat ALL traffic coming in from the outside. So if you connect to a web server from inside when the return traffic comes back the web server is natted to the inside IP of the ASA. So all outbound internet access would break.

I got the impression from your last post that you only used the route on the 4500 as a quick test to check ssh. So if you have the above lines in your config then remove them, they should never have been added ie.

no nat (outside) 1 0.0.0.0 0.0.0.0 outside

no global (inside) 1 interface

what you should have in your config -

nat (inside) 1 0.0.0.0 0.0.0.0

global (outside) 1 interface

this will allow PAT to work for internet connections.

Does that make sense ?

Jon

[aubreystarr]

That is why I removed those global settings. I have to check, but it is possible that I removed

nat (inside) 1 0.0.0.0 0.0.0.0 and left nat (outside) 1 0.0.0.0 0.0.0.0 outside by mistake. That would certainly be the issue then .

[aubreystarr]

Jon,

are you around? I am at the data center same problem. everything works except the internet...

here are the current PAT settings on the ASA:

sh run nat

nat (inside) 0 access-list nonat

nat (inside) 1 0.0.0.0 0.0.0.0

sh run global

global (outside) 1 interface

[Jon Marshall]

Aubrey

Can you send me configs of the 4500 and the ASA ?

Is the default-route on the 4500 setup correctly now ?

Jon