Solved: NAT not working after upgrade

Network Support · ‎04-27-2011

Hey Techies.

Just upped our external ASA-5540 pair to 8.4(1), and now one of our nat's is busted.

Here's the lowdown:

Our public IP for our IronPorts ends in .167. That IP is natted to a VIP on our ACE, which load balances to the IronPorts.

The outside interface of the ASA uses .162, which has been the pat for all outbound traffic for a few years... except for the subnet that houses the IronPorts. Due to reverse lookup, that subnet uses the .167 IP address for all outbound traffic.

After the code upgrade, the nat won't work. No email sent or received. Nothing but Deny's on the ASA with flags reading either "SYN" or "RST". IE:

Apr 27 12:56:11 10.22.151.41 local5.crit %ASA-2-106001: Inbound TCP connection denied from 69.25.174.17/36917 to 207.236.211.167/25 flags SYN on interface outside

If I return the subnet pat back to the outside interface, then inbound traffic works fine, though reverse lookup fails and anyone running a reasonable spam filter won't send to us.

I'm hoping that rings a bell and some of you will have ready solutions.

Let me know if more is required.

Thanks,

mike

Jay Johnston · ‎04-27-2011

Mike,

The problem might stem from the order of the nat rules in the nat table.

We've seen issues like this in the past when you have a nat rule that is something like 'nat (inside,outside) source dynamic obj_any interface' which might be taking precedence over some other object-based inbound PAT translation. This is documented in the ASDM bug "CSCtj78215 - ASDM startup wizard should create after-auto rule for outbound PAT"..check out the release note for it at www.cisco.com/go/bug

Can you provide a 'show nat detail' output from the ASA?

- Jay

View solution in original post

Maykol Rojas · ‎04-27-2011

Hi,

If you put the pat back to the correct IP and if you go to the next hop router, can you see the arp entry for .167 pointing to the outside interface mac-address of the firewall? If not, can you please try to put an static mac-address with that IP and see if you get mail?

Let me know.

Mike.

Mike

Jay Johnston · ‎04-27-2011

Mike,

The problem might stem from the order of the nat rules in the nat table.

We've seen issues like this in the past when you have a nat rule that is something like 'nat (inside,outside) source dynamic obj_any interface' which might be taking precedence over some other object-based inbound PAT translation. This is documented in the ASDM bug "CSCtj78215 - ASDM startup wizard should create after-auto rule for outbound PAT"..check out the release note for it at www.cisco.com/go/bug

Can you provide a 'show nat detail' output from the ASA?

- Jay

Network Support · ‎04-28-2011

Here's the output:

HA-ASA-EX1# sh nat detail
Manual NAT Policies (Section 1)
1 (inside) to (dmz-cd) source static obj-10.22.150.0 obj-10.22.150.0
    translate_hits = 48, untranslate_hits = 3
    Source - Origin: 10.22.150.0/24, Translated: 10.22.150.0/24
2 (inside) to (outside) source dynamic obj-10.22.242.0 interface
    translate_hits = 17491, untranslate_hits = 207
    Source - Origin: 10.22.242.0/24, Translated: 207.236.211.162/27
3 (inside) to (outside) source dynamic obj-10.22.243.0 interface
    translate_hits = 0, untranslate_hits = 0
    Source - Origin: 10.22.243.0/24, Translated: 207.236.211.162/27
4 (inside) to (outside) source dynamic obj-10.22.150.0 interface
    translate_hits = 176472, untranslate_hits = 330
    Source - Origin: 10.22.150.0/24, Translated: 207.236.211.162/27
5 (inside) to (outside) source dynamic obj-10.22.241.0 interface
    translate_hits = 113592, untranslate_hits = 430
    Source - Origin: 10.22.241.0/24, Translated: 207.236.211.162/27
6 (inside) to (outside) source dynamic obj-10.22.244.0 interface
    translate_hits = 4292, untranslate_hits = 0
    Source - Origin: 10.22.244.0/24, Translated: 207.236.211.162/27

Auto NAT Policies (Section 2)
1 (dmz-cd) to (outside) source static Connect_Direct-01 CD-Enterprise-Ext
    translate_hits = 7, untranslate_hits = 36888
    Source - Origin: 172.16.3.175/32, Translated: 207.236.211.175/32
2 (inside) to (outside) source static obj-172.22.2.166 207.236.211.166
    translate_hits = 295, untranslate_hits = 230709
    Source - Origin: 172.22.2.166/32, Translated: 207.236.211.166/32
3 (inside) to (outside) source static obj-172.22.2.167 207.236.211.167
    translate_hits = 6, untranslate_hits = 36526
    Source - Origin: 172.22.2.167/32, Translated: 207.236.211.167/32
4 (inside) to (outside) source static obj-172.22.2.169 207.236.211.169
    translate_hits = 3286, untranslate_hits = 1679533
    Source - Origin: 172.22.2.169/32, Translated: 207.236.211.169/32
5 (inside) to (outside) source static obj-172.22.2.171 207.236.211.171
    translate_hits = 1131, untranslate_hits = 569141
    Source - Origin: 172.22.2.171/32, Translated: 207.236.211.171/32
6 (inside) to (outside) source static obj-172.22.2.173 207.236.211.173
    translate_hits = 32, untranslate_hits = 233328
    Source - Origin: 172.22.2.173/32, Translated: 207.236.211.173/32
7 (inside) to (outside) source static obj-172.22.2.174 207.236.211.174
    translate_hits = 0, untranslate_hits = 695
    Source - Origin: 172.22.2.174/32, Translated: 207.236.211.174/32
8 (inside) to (outside) source static obj-172.22.2.185 207.236.211.185
    translate_hits = 78, untranslate_hits = 30990
    Source - Origin: 172.22.2.185/32, Translated: 207.236.211.185/32
9 (inside) to (outside) source static obj-172.22.2.186 207.236.211.186
    translate_hits = 0, untranslate_hits = 25834
    Source - Origin: 172.22.2.186/32, Translated: 207.236.211.186/32
10 (inside) to (outside) source static obj-172.22.3.168 207.236.211.168
    translate_hits = 721, untranslate_hits = 472794
    Source - Origin: 172.22.3.168/32, Translated: 207.236.211.168/32
11 (inside) to (outside) source static obj-172.22.3.170 207.236.211.170
    translate_hits = 0, untranslate_hits = 261
    Source - Origin: 172.22.3.170/32, Translated: 207.236.211.170/32
12 (inside) to (outside) source static obj-172.22.3.176 207.236.211.176
    translate_hits = 5, untranslate_hits = 88513
    Source - Origin: 172.22.3.176/32, Translated: 207.236.211.176/32
13 (inside) to (outside) source static obj-172.22.3.178 207.236.211.178
    translate_hits = 52, untranslate_hits = 57354
    Source - Origin: 172.22.3.178/32, Translated: 207.236.211.178/32
14 (inside) to (outside) source static obj-172.22.3.179 207.236.211.179
    translate_hits = 9359, untranslate_hits = 5730489
    Source - Origin: 172.22.3.179/32, Translated: 207.236.211.179/32
15 (inside) to (outside) source static obj-172.22.3.180 207.236.211.180
    translate_hits = 5780, untranslate_hits = 2673943
    Source - Origin: 172.22.3.180/32, Translated: 207.236.211.180/32
16 (inside) to (outside) source static obj-172.22.3.181 207.236.211.181
    translate_hits = 74, untranslate_hits = 2582
    Source - Origin: 172.22.3.181/32, Translated: 207.236.211.181/32
17 (inside) to (outside) source static obj-172.22.3.182 207.236.211.182
    translate_hits = 0, untranslate_hits = 528510
    Source - Origin: 172.22.3.182/32, Translated: 207.236.211.182/32
18 (inside) to (outside) source static obj-172.22.3.183 207.236.211.183
    translate_hits = 253, untranslate_hits = 166592
    Source - Origin: 172.22.3.183/32, Translated: 207.236.211.183/32
19 (inside) to (outside) source static obj-172.22.3.187 207.236.211.187
    translate_hits = 3, untranslate_hits = 702
    Source - Origin: 172.22.3.187/32, Translated: 207.236.211.187/32

I highlighted the two pertinent policies... the 241 outbound pat using the interface ip address (which is currently 162, but must change to 167); and the static nat translating the 167 public IP address to the internal 172.22.2.167 VIP.

I've also attached a screenshot of the same from within ASDM.

Thanks for the help.

Network Support · ‎04-28-2011

Yep, order was the ticket. Good call. I deleted the static nat from Section 2 in the nat detail output, and recreated it in Section 1, moving it above the generic outbound pat.

New output for sh nat det:

HA-ASA-EX1# sh nat detail
Manual NAT Policies (Section 1)
1 (inside) to (dmz-cd) source static obj-10.22.150.0 obj-10.22.150.0
    translate_hits = 48, untranslate_hits = 3
    Source - Origin: 10.22.150.0/24, Translated: 10.22.150.0/24
2 (inside) to (any) source static 172.22.2.167 obj-207.236.211.167
    translate_hits = 0, untranslate_hits = 260
    Source - Origin: 172.22.2.167/32, Translated: 207.236.211.167/32
3 (inside) to (outside) source dynamic obj-10.22.242.0 interface
    translate_hits = 17733, untranslate_hits = 207
    Source - Origin: 10.22.242.0/24, Translated: 207.236.211.162/27
4 (inside) to (outside) source dynamic obj-10.22.243.0 interface
    translate_hits = 0, untranslate_hits = 0
    Source - Origin: 10.22.243.0/24, Translated: 207.236.211.162/27
5 (inside) to (outside) source dynamic obj-10.22.150.0 interface
    translate_hits = 177339, untranslate_hits = 330
    Source - Origin: 10.22.150.0/24, Translated: 207.236.211.162/27
6 (inside) to (outside) source dynamic obj-10.22.241.0 obj-207.236.211.167
    translate_hits = 868, untranslate_hits = 0
    Source - Origin: 10.22.241.0/24, Translated: 207.236.211.167/32
7 (inside) to (outside) source dynamic obj-10.22.244.0 interface

translate_hits = 0, untranslate_hits = 0
Source - Origin: 10.22.244.0/24, Translated: 207.236.211.162/27

No errors/denies in the syslogs, and header information shows correct .167 IP address, so reverse lookup is good to go.

Good lesson learned. Thanks for the direction.

Jay Johnston · ‎04-28-2011

I'm glad things are working better now, and that re-ordering things worked to mitigate the problem.

Some things I noticed:

This line:

2 (inside) to (any) source static 172.22.2.167 obj-207.236.211.167
translate_hits = 0, untranslate_hits = 260
Source - Origin: 172.22.2.167/32, Translated: 207.236.211.167/32

Note that it is translating from interface (inside) to interface (any). You might want to make this more specific, so that it is translating from (inside) to (outside).

Also, line 2 (shown above) is a one-to-one NAT translation, but then farther down, this line:

6 (inside) to (outside) source dynamic obj-10.22.241.0 obj-207.236.211.167
translate_hits = 868, untranslate_hits = 0
Source - Origin: 10.22.241.0/24, Translated: 207.236.211.167/32

is a many-to-one PAT translation to the same global IP. This might be working fine (and it might work forever with no trouble) but it struck me as odd. It might be a good idea to make the first one-to-one translation a static PAT instead of a static NAT statement, which would avoid any overlap between line 2 and line 6. You could do it like this:

nat (inside,outside) source static obj-207.236.211.167 obj-172.22.2.167 service tcp-smtp tcp-smtp

This way, just port 25 is forwarded from the .167 global IP to the inside server.

Network Support · ‎04-29-2011

Done... as you suggested. Thanks for the tip.