FWSM static NAT gets Stuck Intermittently

[ansarihuma]

Hi,

We have

wherein we have configured static NAT. It has been observed that static nat gets stuck, and the host becomes unreachable via
both ingress/egress

If i issue a clear xlate local x.x.x.x, this clears things up and
connectivity is restored.

CONFIG DETAILS AS FOLLOWS:

FWSM# show run | i LBS

nameif LBS

object-group network LBS_Firewall_dest

object-group service LBS_ACCESS_PORTS_TCP tcp---------(SOME PORT RANGE)

object-group service LBS_ACCESS_PORTS_UDP udp---------(SOME PORT RANGE)

access-list INTERNET_IN extended permit tcp any object-group LBS_ACCESS_DEST object-group LBS_ACCESS_PORTS_TCP

access-list INTERNET_IN extended permit udp any object-group LBS_ACCESS_DEST object-group LBS_ACCESS_PORTS_UDP

access-list LBS extended permit ip any any

mtu LBS 1500

monitor-interface LBS

icmp permit any LBS

static (LBS,INTERNET) A.A.A.A B.B.B.B netmask 255.255.255.255

access-group LBS in interface LBS

route LBS B.B.B.128 255.255.255.128 C.C.C.C 1

server with private IP B.B.B.B becomes unreachable intermittently, although xlate entry is happening with 1_to_1 mapping.

Please suggest.

Thanks

Fiya.

FWSM Version 4.0(8)

[brquinn]

Most likely, the xlate is actually being built to the wrong interface. This can happen if traffic incorrectly is received on a different interface than is expected. If the traffic is allowed by ACL and RPF checking is disabled, the FWSM will black-hole traffic to the incorrect interface.

Workaround is to enable RPF checking (ip verify reverse-path interface ) or lock down your ACLs to only permit traffic sourced from the correct subnets.

Thanks,

Brendan

[ansarihuma]

Hi Brendan,

Many thanks for kind reply..

RPF have not been enabled on this vlan.

Following is the connection table:

A.A.A.A---NATTED DESTINATION PUBLIC IP

B.B.B.B---NATTED DESTINATION PRIVATE IP.

P.Q.R.S-- SOURCE COMING FROM INTERNET.

**********************************CONNECTION TABLE AT NORMAL SCENARIO*****************************

FWSM# sh conn | i B.B.B.B

TCP INTERNET P.Q.R.S:1298 LBS B.B.B.B:80 idle 0:00:00 Bytes 2782 FLAGS - UOI

TCP INTERNET P.Q.R.S:1299 LBS B.B.B.B:80 idle 0:00:00 Bytes 2736 FLAGS - UOI

********CONNECTION TABLE AT OUTAGE SCENARIO**********SHOWS SWITCH MSFC WITH PUBLIC IP AS DESTINATION**********

FWSM# sh conn | i B.B.B.B

TCP INTERNET P.Q.R.S:1298 MSFC A.A.A.A:80 idle 0:00:00 Bytes 2782 FLAGS - BS

TCP INTERNET P.Q.R.S:1299 MSFC A.A.A.A:80 idle 0:00:00 Bytes 2736 FLAGS - BS

TCP INTERNET P.Q.R.S:554 LBS B.B.B.B:23170 idle 0:00:00 Bytes 48718 FLAGS - BS

Note: MSFC is the vlan mapped in FWSM to have communication between switch (MSFC) & FWSM

Albeit, we are able to ping the server IP even during outage time.

************************************SH XLATE AT BOTH SCENARIO***************************************

FWSM# sh xlate | i B.B.B.B

Global A.A.A.A Local B.B.B.B

FWSM#

Pl. suggest..

Thanks,

Huma.

[ansarihuma]

Hi,

We have enable RPF but still issue is persisting & connection table seems perfect with normal connection flags..

Please suggest ASAP.

Thanks & Regards,

Huma.

[brquinn]

Thanks for the conn output. Did you check the xlates as well? The bad xlates are what black holes the traffic.

Also, do you have rpf checks enabled on all interfaces?

Thanks,

Brendan

[brquinn]

Sorry, I meant to ask for the 'show xlate detail'. The normal 'show xlate' output does not reference the actual interface names. The 'show xlate detail' output should show that the xlate is built to an incorrect interface.

Thanks,

Brendan

[ansarihuma]

Hi Brquinn,

The sh xlate detail seems like:

***********DURING ISSUE**************

FWSM# sh xlate detail | i B.B.B.B

NAT from LBS:B.B.B.B to ASA:B.B.B.B flags Ii

NAT from LBS:B.B.B.B to INTERNET:A.A.A.A flags si

**********DURING NORMAL*************

FWSM# sh xlate detail | i B.B.B.B

NAT from LBS:B.B.B.B to INTERNET:A.A.A.A flags si

NAT from LBS:B.B.B.B to ASA:B.B.B.B flags Ii

I doubt IDENTITY NAT should not create a issue as such.

The only i can find in both scenario is the sequence of IDENTITY NAT happening, i.e. during normal scenario IDENTITY NAT is on first priority and during normal STATIC NAT is on priority.

Even rest of the other interface works absolutely fine with such IDENTITY NAT on priority.

Thanks,

Huma.

[a.henning]

Having the exact same issue...

[Ramraj Sivagnanam Sivajanam]

Hi Bro

Looks like you'll need to enable Cisco's xlate bypass feature in your FWSM. Please click on this URL for further details https://supportforums.cisco.com/thread/2164524

[a.henning]

Thanks for the tip Ramraj!

For anyone else having this intermittent NAT issue here are a couple options to improve and/or solve the issue:

Description:

NAT/PAT entries that worked for years start to display intermittent issues.

show xlate | in ^Global 209.165.200.224

NAT from OUTSIDE:209.165.200.224 to OUTSIDE:209.165.200.224

NAT from DMZ:10.1.1.224 to OUTSIDE:209.165.200.224

The first entry is created dynamically and is the problem, the second entry is correct based on the configuration

Depending on the interpretation, this might be:

•CSCso46878

An extra xlate (between the wrong interfaces) gets created when using static policy NAT and the no nat-controlcommand. This seems to occur when the policy NAT access list overlaps with a network on another interface.

Workaround: If applicable, use static NAT without an access list, and filter with an access-group.

As a temporary measure the clear xlate global 209.165.200.224 resolves the problem until the dynamic entry is created again.

Actions that improved but did not solve the issue in order of significance:

Convert and consolidate multiple PAT entries into 1-to-1 NAT

Reduce xlate timeout

Removed ACL's in NAT statements as per CSCso4687 work around suggestion

Solution in this particular case:

Upgrade code from 3.1(4) to 3.2(23). Required for xlate-bypass

Enable xlate-bypass

Changed OUTSIDE interface mask to not overlap with OUTSIDE NAT addresses

Moved devices on OUTSIDE segment to dedicated NONAT-DMZ zone with public address space.