NAT problem - traffic from internet not reaching internal source.

[skoch1skoch]

Hi all,

I have just added a 10.8.0.0 /16 network and I am having trouble figuring our why I cannot get trafiic back to devices on this network.  Traffic to/from our existing 172.16.0.0 networks works fine, but not the 10.8 network.  Ping requests are returned, but not internet traffic.  When I look at traffic I see these errors:  "

3          Jul 31 2013          09:07:59          305006          10.8.0.10          56070                              portmap translation creation failed for tcp src inside:74.125.225.128/80 dst inside:10.8.0.10/56070" and

"

3          Jul 31 2013          09:09:33          305006          10.8.0.10          56071                              portmap translation creation failed for tcp src inside:74.125.225.128/80 dst inside:10.8.0.10/56071"

So, it appears that the traffic is returned, hits the inside interface, but is not being sent back to the proper device.  Can anyone see anything in this config that may be causing this?

Thanks!

!

ASA Version 8.2(2)

!

hostname ***-ASA5510

names

name 172.16.250.15 SBS

name 172.16.0.0 inside-nets

!

interface Ethernet0/0

description Link to ***

nameif outside

security-level 0

ip address ***.***.***.*** 255.255.255.240

!

interface Ethernet0/1

nameif inside

security-level 100

ip address 172.16.20.1 255.255.255.248

!

interface Ethernet0/2

shutdown

nameif ***

security-level 0

ip address ***.***.***.*** 255.255.255.252

!

interface Ethernet0/3

nameif GuestWireless

security-level 0

ip address 172.16.30.1 255.255.255.0

!

interface Management0/0

nameif management

security-level 100

no ip address

!

banner exec Unauthorized Access is Prohibited.

banner login No Unauthorized Access.  All Access Attempts Will Be Logged.

banner motd  Authorized Access Only.

ftp mode passive

same-security-traffic permit inter-interface

same-security-traffic permit intra-interface

access-list outside_access_in extended deny ip 150.70.0.0 255.255.0.0 any

access-list outside_access_in remark ICMP type 11 for Windows Traceroute

access-list outside_access_in extended permit icmp any any time-exceeded

access-list outside_access_in remark ICMP type 3 for Cisco and Linux

access-list outside_access_in extended permit icmp any any unreachable

access-list outside_access_in extended permit tcp any host *.*.*.* eq 993

access-list outside_access_in extended permit tcp any host *.*.*.* eq imap4

access-list outside_access_in extended permit tcp any host *.*.*.* eq 4125

access-list outside_access_in extended permit tcp any host *.*.*.* eq https

access-list outside_access_in extended permit tcp any host *.*.*.* eq https

access-list remote-users_splitTunnelAcl standard permit inside-nets 255.255.0.0 

access-list inside_nat0_outbound extended permit ip inside-nets 255.255.0.0 172.16.100.0 255.255.255.0

pager lines 24

logging enable

logging timestamp

logging list Config_Changes level emergencies

logging list Config_Changes message 113019

logging list Config_Changes message 111007-111009

logging list Config_Changes message 113012

logging list vpn-log level debugging class vpnc

logging trap vpn-log

logging asdm notifications

logging facility 23

logging device-id hostname

logging host inside 172.16.250.41

logging debug-trace

logging permit-hostdown

mtu outside 1500

mtu inside 1500

mtu ISP2 1500

mtu GuestWireless 1500

mtu management 1500

ip local pool remote-user-pool 172.16.100.0-172.16.100.254 mask 255.255.255.0

icmp unreachable rate-limit 10 burst-size 5

asdm location inside-nets 255.255.0.0 inside

no asdm history enable

arp timeout 14400

global (outside) 101 interface

global (ISP2) 101 interface

nat (inside) 0 access-list inside_nat0_outbound

nat (inside) 101 0.0.0.0 0.0.0.0

nat (GuestWireless) 101 0.0.0.0 0.0.0.0

static (inside,outside) tcp interface smtp SBS smtp netmask 255.255.255.255

static (inside,outside) tcp interface 4125 SBS 4125 netmask 255.255.255.255

static (inside,outside) tcp interface https SBS https netmask 255.255.255.255

static (inside,outside) tcp interface imap4 SBS imap4 netmask 255.255.255.255

static (inside,outside) tcp interface 993 SBS 993 netmask 255.255.255.255

static (inside,outside) tcp interface 6699 Untangle 6699 netmask 255.255.255.255

static (inside,***) *.*.*.* SBS netmask 255.255.255.255 dns

static (inside,outside) [public IP] 172.16.170.10 netmask 255.255.255.255

access-group outside_access_in in interface outside

!

router ospf 1

network 10.5.0.0 255.255.0.0 area 0

network 10.8.0.0 255.255.0.0 area 0

network inside-nets 255.255.0.0 area 0

log-adj-changes

default-information originate always

!

route outside 0.0.0.0 0.0.0.0 [Public IP] 1 track 1

route outside 172.16.240.159 255.255.255.255 *.*.*.* 1

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

dynamic-access-policy-record DfltAccessPolicy

aaa-server SBS-RADIUS protocol radius

reactivation-mode depletion deadtime 1

max-failed-attempts 2

aaa-server SBS-RADIUS (inside) host SBS

key *

radius-common-pw *

aaa authentication ssh console SBS-RADIUS LOCAL

aaa authentication enable console SBS-RADIUS LOCAL

aaa authentication http console SBS-RADIUS LOCAL

aaa authorization exec authentication-server

http server enable

http inside-nets 255.255.0.0 inside

snmp-server host inside 172.16.250.135 community * version 2c

no snmp-server location

no snmp-server contact

snmp-server community *

snmp-server enable traps snmp authentication linkup linkdown coldstart

sla monitor 123

type echo protocol ipIcmpEcho *.*.*.* interface outside

num-packets 3

frequency 10

sla monitor schedule 123 life forever start-time now

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac

crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac

crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac

crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac

crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac

crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac

crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac

crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac

crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac

crypto ipsec security-association lifetime seconds 28800

crypto ipsec security-association lifetime kilobytes 4608000

crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set pfs group1

crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-

192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5

crypto map outside_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP

crypto map outside_map interface outside

crypto isakmp enable outside

crypto isakmp policy 65535

authentication pre-share

encryption 3des

hash sha

group 2

lifetime 86400

no crypto isakmp nat-traversal

!

track 1 rtr 123 reachability

telnet timeout 5

ssh inside-nets 255.255.0.0 inside

ssh timeout 60

ssh version 2

console timeout 0

management-access inside

dhcpd address 172.16.30.100-172.16.30.200 GuestWireless

dhcpd dns *.*.*.* *.*.*.* interface GuestWireless

dhcpd option 3 ip 172.16.30.1 interface GuestWireless

dhcpd enable GuestWireless

!

threat-detection basic-threat

threat-detection statistics port

threat-detection statistics protocol

threat-detection statistics access-list

threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200

ntp server SBS

webvpn

group-policy remote-users internal

group-policy remote-users attributes

dns-server value 172.16.250.15

vpn-idle-timeout none

vpn-session-timeout none

vpn-tunnel-protocol IPSec

split-tunnel-policy tunnelspecified

split-tunnel-network-list value remote-users_splitTunnelAcl

default-domain value ***.local

tunnel-group remote-users type remote-access

tunnel-group remote-users general-attributes

address-pool remote-user-pool

authentication-server-group SBS-RADIUS

default-group-policy remote-users

tunnel-group remote-users ipsec-attributes

pre-shared-key ***

!

class-map inspection_default

match default-inspection-traffic

!

!

policy-map type inspect dns preset_dns_map

parameters

  message-length maximum client auto

  message-length maximum 512

policy-map global_policy

class inspection_default

  inspect dns preset_dns_map

  inspect ftp

  inspect h323 h225

  inspect h323 ras

  inspect ip-options

  inspect netbios

  inspect rsh

  inspect rtsp

  inspect skinny 

  inspect sqlnet

  inspect sunrpc

  inspect tftp

  inspect sip 

  inspect xdmcp

  inspect icmp

  inspect snmp

class class-default

  set connection decrement-ttl

!

service-policy global_policy global

prompt hostname context

call-home

profile CiscoTAC-1

  no active

  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService

  destination address email callhome@cisco.com

  destination transport-method http

  subscribe-to-alert-group diagnostic

  subscribe-to-alert-group environment

  subscribe-to-alert-group inventory periodic monthly

  subscribe-to-alert-group configuration periodic monthly

  subscribe-to-alert-group telemetry periodic daily

[Jouni Forss]

Hi,

Is there something wrong with the routing?

The logs indicate that its trying to create a Dynamic PAT for traffic from "inside" to "inside". The connections should be going to "outside" I would imagine?

The reason you see the "portmap" log message is that the traffic is trying to head out through the "inside" interface and "inside" interface doesnt have any matching "global" command. (Which it shouldnt have I imagine)

So why is the traffic heading to "inside"?

Could you perhaps share the output of

show route

- Jouni

[Jouni Forss]

Also,

One thing that can cause the ASA to forward traffic to wrong place are "static" NAT configurations.

But I would imagine that is not the case here.

But I still wonder how/why the traffic is forwarded to "inside" according to the above log messages.

You do have static default route pointing out of the ASA so to my understanding that should be prefered for the public destination IP address shown in the logs no matter what routes the dynamic routing protocol might advertice.

You seem to be using OSPF for the routing. I have never to this day used any dynamic routing protocol on the ASA. It has always been static routing.

- Jouni

[skoch1skoch]

The routing from inside to inside is confusing me also.  Here is the routing table:

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP

       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area

       * - candidate default, U - per-user static route, o - ODR

       P - periodic downloaded static route

Gateway of last resort is [ISP Gateway] to network 0.0.0.0

C    [Public Network] 255.255.255.240 is directly connected, outside

O    172.16.180.0 255.255.255.0 [110/11] via 172.16.20.2, 45:27:17, inside

O    172.16.170.0 255.255.255.0 [110/11] via 172.16.20.2, 45:27:17, inside

O    172.16.250.0 255.255.255.0 [110/11] via 172.16.20.2, 45:27:17, inside

O    172.16.251.0 255.255.255.0 [110/11] via 172.16.20.2, 45:27:17, inside

O    172.16.240.0 255.255.255.0 [110/11] via 172.16.20.2, 45:27:17, inside

O    172.16.230.0 255.255.255.0 [110/11] via 172.16.20.2, 45:27:17, inside

O    172.16.220.0 255.255.255.0 [110/11] via 172.16.20.2, 45:27:17, inside

C    172.16.30.0 255.255.255.0 is directly connected, GuestWireless

C    172.16.20.0 255.255.255.248 is directly connected, inside

S    172.16.240.159 255.255.255.255 [1/0] via [ISP Gateway], outside

O    172.16.99.0 255.255.255.0 [110/11] via 172.16.20.2, 45:27:17, inside

O    10.8.0.0 255.255.0.0 [110/11] via 172.16.20.2, 45:27:17, inside

O    10.5.0.0 255.255.0.0 [110/11] via 172.16.20.2, 45:27:17, inside

S*   0.0.0.0 0.0.0.0 [1/0] via [ISP Gateway], outside

Thanks for the help!

[turbo_engine26]

Hi,

Can you launch a packet tracer on the outside interface toward a host in the 10.8.0.0/16 network ?

packet-tracer input outside src-public_ip 1234 10.8.2.2 1234

Regards,

AM

[skoch1skoch]

Here's the output of the packet-tracer:

ASA5510# packet-tracer input outside tcp public_ip 1234 10.8.2.2 12$

Phase: 1

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

MAC Access list

Phase: 2

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 3

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   10.8.0.0        255.255.0.0     inside

Phase: 4

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Result:

input-interface: outside

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

...so, it seems that an ACL is blocking the return traffic somehow.  I added these two statements to the ACL:

access-list outside_access_in extended permit tcp any 10.8.0.0 255.255.0.0 eq 1234

access-list outside_access_in extended permit ip any 10.8.0.0 255.255.0.0

I reran the packet-tracer and the connection was still blocked by an acl-drop.

ASA5510(config)# packet-tracer input outside tcp public-ip 1234 10.$

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   10.8.0.0        255.255.0.0     inside

Phase: 3

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Result:

input-interface: outside

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

I cleared the ACL counter on the outside_access_in ACL and re-ran the packet-tracer:

ASA5510# clear access-list outside_access_in counters

ASA5510# show access-list

access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)

            alert-interval 300

access-list outside_access_in; 16 elements; name hash: 0x6892a938

access-list outside_access_in line 1 extended deny ip 150.70.0.0 255.255.0.0 any (hitcnt=0) 0x247e4f19

access-list outside_access_in line 2 remark ICMP type 11 for Windows Traceroute

access-list outside_access_in line 3 extended permit icmp any any time-exceeded (hitcnt=0) 0x03690eb3

access-list outside_access_in line 4 remark ICMP type 3 for Cisco and Linux

access-list outside_access_in line 5 extended permit icmp any any unreachable (hitcnt=1) 0x5c2fa603

access-list outside_access_in line 10 extended permit tcp any host public_ip eq 993 (hitcnt=0) 0x445ffc7a

access-list outside_access_in line 11 extended permit tcp any host public_ip eq imap4 (hitcnt=0) 0xc10d6b17

access-list outside_access_in line 12 extended permit tcp any host public_ip eq 4125 (hitcnt=0) 0xf9904a7e

access-list outside_access_in line 13 extended permit tcp any host public_ip eq https (hitcnt=2) 0x933075d4

access-list outside_access_in line 14 extended permit tcp any host public_ip eq smtp (hitcnt=0) 0xbbbe8a1e

access-list outside_access_in line 15 extended permit tcp any host public_ip eq 6699 (hitcnt=0) 0x67c9552f

access-list outside_access_in line 16 extended permit tcp any host public_ip eq https (hitcnt=0) 0xda2d6a77

access-list outside_access_in line 17 extended permit tcp any 10.8.0.0 255.255.0.0 eq 1234 (hitcnt=0) 0x88dd37e5

access-list outside_access_in line 18 extended permit ip any 10.8.0.0 255.255.0.0 (hitcnt=0) 0xa9169af9

access-list remote-users_splitTunnelAcl; 1 elements; name hash: 0x19f43771

access-list remote-users_splitTunnelAcl line 1 standard permit inside-nets 255.255.0.0 (hitcnt=0) 0x9cd507cd

access-list inside_nat0_outbound; 5 elements; name hash: 0x467c8ce4

access-list inside_nat0_outbound line 1 extended permit ip 172.16.180.0 255.255.255.0 172.16.200.0 255.255.255.0 (hitcnt=0) 0xbc5b1ea5

access-list inside_nat0_outbound line 2 extended permit ip 172.16.180.0 255.255.255.0 172.16.210.0 255.255.255.0 (hitcnt=0) 0xbff6a7ce

access-list inside_nat0_outbound line 3 extended permit ip 172.16.180.0 255.255.255.0 172.16.190.0 255.255.255.0 (hitcnt=0) 0x19b50b54

access-list inside_nat0_outbound line 4 extended permit ip tascet-inside-nets 255.255.0.0 172.16.100.0 255.255.255.0 (hitcnt=0) 0xba71f6d5

access-list inside_nat0_outbound line 5 extended permit ip 172.16.180.0 255.255.255.0 172.16.240.0 255.255.255.0 (hitcnt=0) 0x00a7b064

ASA5510# packet-tracer input outside tcp 66.170.26.242 1234 8.0.2.2 123$

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   0.0.0.0         0.0.0.0         outside

Phase: 3

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Result:

input-interface: outside

input-status: up

input-line-status: up

output-interface: outside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

ASA5510# show access-list

access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)

            alert-interval 300

access-list outside_access_in; 16 elements; name hash: 0x6892a938

access-list outside_access_in line 1 extended deny ip 150.70.0.0 255.255.0.0 any (hitcnt=0) 0x247e4f19

access-list outside_access_in line 2 remark ICMP type 11 for Windows Traceroute

access-list outside_access_in line 3 extended permit icmp any any time-exceeded (hitcnt=0) 0x03690eb3

access-list outside_access_in line 4 remark ICMP type 3 for Cisco and Linux

access-list outside_access_in line 5 extended permit icmp any any unreachable (hitcnt=1) 0x5c2fa603

access-list outside_access_in line 10 extended permit tcp any host public_ip eq 993 (hitcnt=0) 0x445ffc7a

access-list outside_access_in line 11 extended permit tcp any host public_ip eq imap4 (hitcnt=0) 0xc10d6b17

access-list outside_access_in line 12 extended permit tcp any host public_ip eq 4125 (hitcnt=0) 0xf9904a7e

access-list outside_access_in line 13 extended permit tcp any host public_ip eq https (hitcnt=2) 0x933075d4

access-list outside_access_in line 14 extended permit tcp any host public_ip eq smtp (hitcnt=0) 0xbbbe8a1e

access-list outside_access_in line 15 extended permit tcp any host public_ip eq 6699 (hitcnt=0) 0x67c9552f

access-list outside_access_in line 16 extended permit tcp any host public_ip eq https (hitcnt=0) 0xda2d6a77

access-list outside_access_in line 17 extended permit tcp any 10.8.0.0 255.255.0.0 eq 1234 (hitcnt=0) 0x88dd37e5

access-list outside_access_in line 18 extended permit ip any 10.8.0.0 255.255.0.0 (hitcnt=0) 0xa9169af9

access-list remote-users_splitTunnelAcl; 1 elements; name hash: 0x19f43771

access-list remote-users_splitTunnelAcl line 1 standard permit inside-nets 255.255.0.0 (hitcnt=0) 0x9cd507cd

access-list inside_nat0_outbound; 5 elements; name hash: 0x467c8ce4

access-list inside_nat0_outbound line 1 extended permit ip 172.16.180.0 255.255.255.0 172.16.200.0 255.255.255.0 (hitcnt=0) 0xbc5b1ea5

access-list inside_nat0_outbound line 2 extended permit ip 172.16.180.0 255.255.255.0 172.16.210.0 255.255.255.0 (hitcnt=0) 0xbff6a7ce

access-list inside_nat0_outbound line 3 extended permit ip 172.16.180.0 255.255.255.0 172.16.190.0 255.255.255.0 (hitcnt=0) 0x19b50b54

access-list inside_nat0_outbound line 4 extended permit ip inside-nets 255.255.0.0 172.16.100.0 255.255.255.0 (hitcnt=0) 0xba71f6d5

access-list inside_nat0_outbound line 5 extended permit ip 172.16.180.0 255.255.255.0 172.16.240.0 255.255.255.0 (hitcnt=0) 0x00a7b064

I am not seeing the ACL that is blocking traffic here - I'm baffled.  any ideas?

[Jouni Forss]

Hi,

There is not much point in trying to simulate a connection coming from the "outside" to your "inside" network with the private IP address as the destination as no such packet will ever reach your firewall.

Can you try doing a "packet-tracer" to simulate a new connection being formed from the new LAN Network

packet-tracer input inside tcp 10.8.2.2 12345 8.8.8.8 80

And see if it gives us any information.

The original log messages still seem really strange to me.

- Jouni

[skoch1skoch]

That seemed to work just fine:

ASA5510# packet-tracer input inside tcp 10.8.2.2 12345 8.8.8.8 80

Phase: 1

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

MAC Access list

Phase: 2

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 3

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   0.0.0.0         0.0.0.0         outside

Phase: 4

Type: CONN-SETTINGS

Subtype:

Result: ALLOW

Config:

class-map class-default

match any

policy-map global_policy

class class-default

  set connection decrement-ttl

service-policy global_policy global

Additional Information:

Phase: 5

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 6

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (inside) 101 0.0.0.0 0.0.0.0

  match ip inside any outside any

    dynamic translation to pool 101 (public_ip [Interface PAT])

    translate_hits = 964843, untranslate_hits = 39941

Additional Information:

Dynamic translate 10.8.2.2/12345 to public_ip/51088 using netmask 255.255.255.255

Phase: 7

Type: NAT

Subtype: host-limits

Result: ALLOW

Config:

nat (inside) 101 0.0.0.0 0.0.0.0

  match ip inside any outside any

    dynamic translation to pool 101 (public_ip [Interface PAT])

    translate_hits = 964843, untranslate_hits = 39941

Additional Information:

Phase: 8

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 9

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 1300556, packet dispatched to next module

Result:

input-interface: inside

input-status: up

input-line-status: up

output-interface: outside

output-status: up

output-line-status: up

Action: allow

[Jouni Forss]

Hi,

I would suggest going through the whole path of the network from the host in the new network all the way to its default Internet gateway where its default route leads to and again back to the host from that Internet gateway viewing routing table on each step. I would imagine that are atleast some routers involved as you decided to use a routing protocol instead of static routing.

I am just wondering if this would have anything to do with you possibly having another Internet gateway in addition to the ASA and the traffic taking a "wrong turn" somewhere along the way in your network?

At first glance the log message would seem to point to a situation where a connection from the host might have gone to the Internet through some other Internet gateway (other than the ASA) and the return traffic has come back to the ASA from that Internet gateway before reaching the actual host.

- Jouni

[skoch1skoch]

I've got layer 3 switches doing routing in my network.  The traffic from the 10.8.0.0 network is coming from the switch below.  The default route is to the inside interface of the firewall (172.16.20.1).  The ASA's routing table (shown above) then has a default gateway set to the ISP's gateway address.  Should the default gateway on the switch be set to the ISP's Gateway address?

Edit:  There are no other paths out of the network.

"switch1"#sh ip ro

Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route

Gateway of last resort is 172.16.20.1 to network 0.0.0.0

     172.16.0.0/16 is variably subnetted, 11 subnets, 3 masks

C       172.16.180.0/24 is directly connected, Vlan180

C       172.16.170.0/24 is directly connected, Vlan170

C       172.16.250.0/24 is directly connected, Vlan250

C       172.16.251.0/24 is directly connected, Vlan251

C       172.16.240.0/24 is directly connected, Vlan240

C       172.16.230.0/24 is directly connected, Vlan230

C       172.16.220.0/24 is directly connected, Vlan220

O       172.16.30.0/24 [110/11] via 172.16.20.1, 6d23h, GigabitEthernet1/0/48

C       172.16.20.0/29 is directly connected, GigabitEthernet1/0/48

S       172.16.240.159/32 [1/0] via 172.16.20.1

C       172.16.99.0/24 is directly connected, Vlan99

     10.0.0.0/16 is subnetted, 2 subnets

C       10.8.0.0 is directly connected, Vlan108

C       10.5.0.0 is directly connected, Vlan105

S*   0.0.0.0/0 [1/0] via 172.16.20.1

[Jouni Forss]

Hi,

Routing seems correct. Default gateway should be pointing to the ASA "inside" interface IP address like it is at the moment.

I don't know why we would see the ASA at any point trying to create a translation from "inside" to "inside".

The most common reason I have seen for an ASA to forward traffic through wrong interface has been some NAT configuration but I cant see anything in your configuration that would point to such a problem.

I can't remember ever running into such a situation myself so I dont even have that advantage when trying to see what this problem is about.

You could always consider rebooting the ASA and see if it has any effect on the current problem (remember to save configuration if you device to do so)

Other options would be to further try to get logs from connection attempts from a single host.

There is also the option of trying to capture some traffic on the ASA if it would tell anything about the problem.

I just dont see a reason why this network wouldnt work if all other networks behind "inside" work at the moment.

- Jouni

[skoch1skoch]

Jouni,

Exactly right, all other inside networks (except the 10.5.0.0, which is also new) are routing and NATing just fine.  Also, I'm curious as to why ICMP traffic is routed properly to and from the 10.8.0.0 network, but not http or other tcp traffic.  I will bounce the ASA, but I'll have to wait until the evening.  Thanks for your help thus far.

-Shane

[Jouni Forss]

Hi,

In some cases where routing is the problem ICMP might act a bit differently than TCP connections for example.

ICMP might work and TCP not because of asymmetric routing. (especially when ASA is in the picture)

Looking at your L3 Switch routing table I can only see ONE network adverticed with OSPF and its from the ASA. And since the ASA is the default gateway it wouldnt even need to advertice that network as the traffic would always come to the ASA with default route.

So I am not sure where the OSPF is needed in this case since it seems to only run between the L3 switch and ASA. I would personally handle it with Static routing at both ASA and L3 switch since the OSPF doesnt really bring that much into the picture.

You would only need static default route from L3 Switch towards ASA and ASA would need route for the LAN network towards its "inside" interface.

It just somehow seems to me like something to do with routing but doesnt make sense especially when the ASA should be the only route out of your network.

- Jouni

[turbo_engine26]

Shane,

There is one obvious mistake in your OSPF configuration that may lead to this confusion. You are advertising networks that are not directly connected to your ASA. As shown, these networks are directly connected to the switch through Vlans 108 and 105. This means that the switch need to advertise them to the ASA not the vice-versa. After all, ASA want to reach these networks in order to serve them so they must get advertised by the switch. Remember the "network" command is used to define which networks the device knows to advertise them to other devices.

Can you just create static routes to these destinations with the switch as the default gateway and see what will happen?

Regards,

AM

[turbo_engine26]

Also, i noticed that there is an object group defined called "inside-nets" in OSPF config. Is this the 172.16.0.0 major network behind the switch? ... When you said that there is no problem with traffic coming from/to 172.16.0.0, did you mean the 172.16 networks behind the switch OR the 172.16.30.0 and 172.16.20.0 that are directly connected to ASA? Did you try to initiate traffic from any of 172.16 networks that are behind the switch?

Regards,

AM