Solved: NAT/routing issue from one subinterface to another - Page 2

WStoffel1 · ‎11-20-2012

I have a customer that sits behind an ASA on Int g0/1.143 (sec-level set to 100).

I have another customer that sits behind the same firewall on Int g0/1.156 (sec level = 100 as well).

The default gateway for each customer is the ip address on the above interfaces, 192.168.143.254 and 192.168.156.254 respectively.

The customer on 1.156 has a web server that's up and alive on the internet, natted to a public address of 72.46.x.x.

Customers on 192.168.143.0/24 cannot get to the website. When i tracert from 192.168.143.10 i can see the router that sits on the outside interface of the ASA then it fails, so it's being sent out the default gateway of the ASA.

What am i missing here?

Thanks as always....

Jouni Forss · ‎11-21-2012

Hi,

If you can make the DNS change on the local LAN DNS server to point to the private IP address of the server instead of the public this should be almost enough.

Ofcourse you need to allow the traffic on the access-list in the source interface (unless you have some wider rule already allowing that)

The NAT setup between the interfaces depends on your ASA software. In a 8.3 and newer software you wouldnt need any NAT configurations for this traffic as you wont be doing any NAT and the traffic will be using the client/server actual IP address.

In software 8.2 and below there is a thing called "nat-control" which determines if a NAT is always required or never required. The default setting, if I remember right, is that the "nat-control" is not enabled. (Its not visible in the configuration in this case). Then again if "nat-control" is enabled it will show in the configuration just above the "global" configuration lines. I havent had to deal with this in ages so I might have gotten it completely the wrong way (which setting was default that is)

If you want to capture traffic for a connection you are testing you can do it either at the test computer itself, on the server, in the switches you might have or on the ASA (which I have used alot in recent months to get familiar with using it for troubleshooting)

Now if we presume that you want to capture only HTTP connections towards the server using (and return traffic) the ASA, you could do following configurations. I also presume you are connecting using the local IP of the server (even though it should work if access rules are OK and there is not some other NAT configuration in the way)

access-list WEB-SERVER-CAPTURE permit tcp host host

capture WEB-SERVER-CAPTURE type raw-data packet-length 1522 access-list WEB-SERVER-CAPTURE interface Amherst buffer 33500000 circular-buffer

You can check the amount of data captured by the above configuration by using command "show capture". The capture wont show in the "show run" output of ASA. Only with "show capture" command. If the ASA happens to boot the capture will be removed also.

"buffer" parameter sets the amount of memory used for storing data. I'm using almost the maximum amount ASA allows per capture.

"circular-buffer" tells the ASA to overwrite the old data if the memory amount set with "buffer" is exceeded. Otherwise the capture will stop when buffer is full. Access-list and Capture name dont have to match but I usually do it that way.

You can show the capture data with "show capture "

Though I would suggest using Wireshark software for opening the capture file. For this you need to copy the file from ASA to your computer with TFTP

copy /pcap capture:WEB-SERVER-CAPTURE tftp://x.x.x.x/filename.pcap

You can also use Wireshark for capturing traffic on the computer you install it on ofcourse (I guess you might be familiar with this already, personally I havent used it for a long time even if it is a very basic tool for networking)

To remove the capture use the command "no capture ". Notice though that this command will also delete the captured data from the ASA.

If you dont want to go quite so deep you can simple watch the ASA logs through ASDM, ASA CLI or better yet Syslog server. Logging level on ASA would need to be atleast "informational" so that you can see the messages related to connection forming and closing.

- Jouni

WStoffel1 · ‎11-21-2012

I was thinking wireshark and packet captures would be a last resort, maybe i'm at that point then as nothing seems to be working. But i just got a little hope...by the way i'm running asa804-k8.bin

I've made no dns changes locally yet but when i'm on a server on the Amherst lan, 192.168.143.10, and i try to ping the web server on the Franklin lan, 192.168.156.10, the requests time out.

i added, based on your above

Static NAT to outside

static (lan1,outside) 1.2.3.4 10.10.10.10 netmask 255.255.255.255 dns

static (Franklin,outside) 74.11.x.x 192.168.156.10 netmask 255.255.255.255 dns

Static NAT between LANs

static (lan1,lan2) 1.2.3.4 10.10.10.10 netmask 255.255.255.255

static (Franklin,amherst) 74.11.x.x 192.168.156.10 netmask 255.255.255.255

I have 2 constant pings running now and i see log file entries:

For the local addressing:

Nov 21 2012 17:30:18: %ASA-3-305006: portmap translation creation failed for icmp src amherst:192.168.143.10 dst Franklin:192.168.156.10 (type 8, code 0)

Likewise the ping to the Public url yields:

Nov 21 2012 17:31:34: %ASA-3-305006: portmap translation creation failed for icmp src amherst:192.168.143.10 dst Franklin:74.11.x.x (type 8, code 0)

I'm wondering if your lan1 and lan2 are matching up with my amherst and Franklin lans, respectively. Or do i have things backwards?

Jouni Forss · ‎11-21-2012

Hi,

Seems to be some problem with NAT configurations then.

We still havent seen your configurations though so I can't say anything for sure.

What I would suggest would be to leave the server to outside Static NAT with the DNS parameter added and remove the Static NAT server to other lan to return to a normal situation.

I would first try the DNS changes before trying anything else. At the least use a test computer in the LAN where you need to connect from and use a public DNS server and see if you can connect then to the server by using DNS name.

You can also post the ASA configurations and remove the sensitive information if there is some.

Cisco explanation for your log message ID

305006

Error Message    %ASA-3-305006: {outbound static|identity|portmap|regular) translation 
creation failed for protocol src interface_name:source_address/source_port 
[(idfw_user)] dst interface_name:dest_address/dest_port [(idfw_user)]

Explanation A protocol (UDP, TCP, or ICMP) failed to create a translation through the ASA. The ASA does not allow packets through that are destined for network or broadcast addresses. The ASA provides this checking for addresses that are explicitly identified with static commands. For inbound traffic, the ASA denies translations for an IP address identified as a network or broadcast address.

The ASA does not apply PAT to all ICMP message types; it only applies PAT ICMP echo and echo-reply packets (types 8 and 0). Specifically, only ICMP echo or echo-reply packets create a PAT translation. As a result, when the other ICMP messages types are dropped, this message is generated.

The ASA uses the global IP address and mask from configured static commands to differentiate regular IP addresses from network or broadcast IP addresses. If the global IP address is a valid network address with a matching network mask, then the ASA does not create a translation for network or broadcast IP addresses with inbound packets.

For example:


static (inside,outside) 10.2.2.128 10.1.1.128 netmask 255.255.255.128


 
 
 
 The ASA responds to global address 10.2.2.128 as a network address and  to 10.2.2.255 as the broadcast address. Without an existing translation,  the ASA denies inbound packets destined for 10.2.2.128 or 10.2.2.255,  and logs this message. 

 
 When the suspected IP address is a host IP address, configure a separate static command with a host mask in front of the subnet static command (the first match rule for static commands). The following static commands cause the ASA to respond to 10.2.2.128 as a host address: 

 

static (inside,outside) 10.2.2.128 10.2.2.128 netmask 255.255.255.255 
static (inside,outside) 10.2.2.128 10.2.2.128 netmask 255.255.255.128


 
 
 
 The translation may be created by traffic started from the inside host  with the IP address in question. Because the ASA views a network or  broadcast IP address as a host IP address with an overlapped subnet  static configuration, the network address translation for both static commands must be the same. 

 
Recommended Action    None required.

- Jouni

WStoffel1 · ‎11-23-2012

Ok, posting my config is going to take some work, it's 1900 lines long and there's a lot of stuff i can't post publicly, but I'm afraid to make changes sense i may change something relevant. However i will work on that today.

But i had an interesting thing i just noticed. I got back on that server, having not changed anything since i left it Wednesday, and went to ping that website, it's coming back as the local private address. I can still get to the website externally so the web site is still up...meaning it's still using the public address elsewhere.

So something in this translation is working? The server is point to 4.2.2.2 for dns. And I have to say it's very odd to see a windows server responding back from a public DNS server with a private address.

WStoffel1 · ‎11-23-2012

Ok actually that wasn't so bad to go through. I removed all of my other customers subinterfaces and access lists, as well as the crypto and I think it's ok. Any questions on anything i changed just let me know. Also note, i left the real vlan and private subnet for Amherst and Franklin this time, so this is the actual config. I'm trying to get hosts on 192.168.133.0 to the webserver at 192.168.146.10. The webservers public address is network-object host x.x.122.45

Thanks again for all the help.

n_schloemer · ‎11-26-2012

WStoffel1,

Can you run the following packet-tracer.

packet-tracer input Amherst tcp 192.168.133.20 25354 192.168.146.10 80 detailed

and post the output.

thanks,

Nick

WStoffel1 · ‎11-26-2012

Ahh yes, I should have posted that already:

packet-tracer input Amherst tcp 192.168.133.20 25354 192.168.146.10 80 $

Phase: 1

Type: CAPTURE

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xb047cf70, priority=12, domain=capture, deny=false

hits=5947967, user_data=0xb1fda948, cs_id=0x0, l3_type=0x0

src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0000.0000.0000

Phase: 2

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0xac3d8348, priority=1, domain=permit, deny=false

hits=16069432, user_data=0x0, cs_id=0x0, l3_type=0x8

src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0000.0000.0000

Phase: 3

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 4

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 192.168.146.0 255.255.255.0 Franklin

Phase: 5

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0xad5b5650, priority=2, domain=permit, deny=false

hits=103456, user_data=0x0, cs_id=0x0, flags=0x3000, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 6

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xad4fd1e0, priority=0, domain=permit-ip-option, deny=true

hits=391153, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 7

Type: FOVER

Subtype: standby-update

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xac3d7df0, priority=21, domain=lu, deny=true

hits=127965, user_data=0x0, cs_id=0x0, flags=0x0, protocol=6

src ip=0.0.0.0, mask=0.0.0.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=80, dscp=0x0

Phase: 8

Type: NAT

Subtype: host-limits

Result: ALLOW

Config:

static (Amherst,dmz) 192.168.133.0 192.168.133.0 netmask 255.255.255.0

match ip Amherst 192.168.133.0 255.255.255.0 dmz any

static translation to 192.168.133.0

translate_hits = 465, untranslate_hits = 3637

Additional Information:

Forward Flow based lookup yields rule:

in id=0xac962570, priority=5, domain=host, deny=false

hits=475673, user_data=0xac453d20, cs_id=0x0, reverse, flags=0x0, protoc ol=0

src ip=192.168.133.0, mask=255.255.255.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Phase: 9

Type: NAT

Subtype:

Result: DROP

Config:

nat (Amherst) 1 192.168.133.0 255.255.255.0

match ip Amherst 192.168.133.0 255.255.255.0 Franklin any

dynamic translation to pool 1 (No matching global)

translate_hits = 4, untranslate_hits = 0

Additional Information:

Forward Flow based lookup yields rule:

in id=0xac798bc0, priority=1, domain=nat, deny=false

hits=3, user_data=0xb0bbfe80, cs_id=0x0, flags=0x0, protocol=0

src ip=192.168.133.0, mask=255.255.255.0, port=0

dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

Result:

input-interface: Amherst

input-status: up

input-line-status: up

output-interface: Franklin

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

FYI, this is the current NATs i have based on the above:

static (Franklin,outside) 74.11.x.x 192.168.146.10 netmask 255.255.255.255 dns

static (Franklin,Amherst) 74.11.x.x 192.168.146.10 netmask 255.255.255.255

WStoffel1 · ‎11-28-2012

Just to follow up, what I ended up doing to fix this issue was a combination of DNS server and ASA configs.

Increased the security level on the source interface relative to the destination where the web server resided.

Added static nat:

Static (amherst,Franklin) 192.168.133.0 192.168.133.0 netmask 255.255.255.255

This enabled me to ping across the interfaces, and logically all the traffic will be initiated outbound so I think it's set.

Then on the local source (Amherst) network I added a new zone for the domain of the website, ABCDE.com and added a single A record for WWW.

Problem solved....Thanks everyone for all the help!!!!!