Re: Route or NAT problem?

ken.montgomery · ‎11-07-2013

Hi Everyone,

We have an ASA 5540 at our data center, with ASA 5505's at most remote sites.

At the sites without layer 3 switches behind the ASA 5505's, we can't reach the data center internal network through the ASA for flow-export, etc.

So, what I'm basically saying is, even though the tunnel is up and everything behind the branch ASA can reach the data center networks fine, the ASA itself cannot reach hosts on the data center network.

I'm hoping to configure these ASA 5505's so I can do flow export and SNMP logging from them, but without this routing or nat problem resolved, they just won't do it.

Doing a packet tracer from the ASA 5505 to the data center server I'm most focused on, reveals this:

BRANCH5505f01# packet input inside icmp 10.15.16.1 8 0 10.1.1.15 detailed

Phase: 1

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0xcb0b6698, priority=1, domain=permit, deny=false

hits=1004755, user_data=0x0, cs_id=0x0, l3_type=0x8

src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0100.0000.0000

input_ifc=inside, output_ifc=any

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 10.1.1.15 255.255.255.255 outside

Phase: 3

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 0.0.0.0 0.0.0.0 outside

Result:

input-interface: inside

input-status: up

input-line-status: up

output-interface: outside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (rpf-violated) Reverse-path verify failed

I am thinking the problem is NAT related, but with the new ASA NAT rule format due to v9.1... struggling to get a grip on where it is... any thoughts/help are appreciated.

Ken

Here is the relevant config for the Branch ASA and also the relevant config from the data center ASA:

Branch ASA Config Parts:

: Saved

:

ASA Version 9.1(2)

!

hostname BRANCHASA5505

xlate per-session deny tcp any4 any4

xlate per-session deny tcp any4 any6

xlate per-session deny tcp any6 any4

xlate per-session deny tcp any6 any6

xlate per-session deny udp any4 any4 eq domain

xlate per-session deny udp any4 any6 eq domain

xlate per-session deny udp any6 any4 eq domain

xlate per-session deny udp any6 any6 eq domain

names

!

interface Ethernet0/0

switchport access vlan 2

!

interface Ethernet0/1

speed 100

duplex full

!

interface Ethernet0/2

!

interface Ethernet0/3

!

interface Ethernet0/4

!

interface Ethernet0/5

!

interface Ethernet0/6

!

interface Ethernet0/7

!

interface Vlan1

description LAN_NETWORK

nameif inside

security-level 100

ip address 10.15.6.1 255.255.254.0

!

interface Vlan2

nameif outside

security-level 0

ip address <outside ip> 255.255.255.248

!

same-security-traffic permit inter-interface

same-security-traffic permit intra-interface

object network obj_any

subnet 0.0.0.0 0.0.0.0

object-group network BRANCH_NETWORKS

description BRANCH LOCAL NETWORKS

network-object 10.15.6.0 255.255.254.0

object-group network LAN_NETWORKS

network-object 10.0.0.0 255.0.0.0

network-object 134.200.131.0 255.255.255.0

network-object 134.200.220.0 255.255.255.0

network-object 134.201.2.0 255.255.255.0

network-object 163.243.195.0 255.255.255.0

network-object 172.16.0.0 255.240.0.0

network-object 192.168.0.0 255.255.0.0

network-object 10.1.3.0 255.255.255.0

network-object 10.31.2.0 255.255.255.0

network-object 10.1.1.0 255.255.255.0

network-object 172.26.1.0 255.255.255.0

object-group network NETWORK_MGMT

network-object 10.0.0.0 255.0.0.0

access-list DATACENTER_VPN_ACL remark *******************************************************************

access-list DATACENTER_VPN_ACL remark * FOR VPN CONNECTION TO DATACENTER/VEYANCE NETWORKS *

access-list DATACENTER_VPN_ACL remark *******************************************************************

access-list DATACENTER_VPN_ACL extended permit ip host <outside ip> host <outside ip datacenter asa>

access-list DATACENTER_VPN_ACL extended permit ip object-group BRANCH_NETWORKS object-group LAN_NETWORKS

access-list INSIDE_NONAT extended permit ip object-group BRANCH_NETWORKS object-group LAN_NETWORKS

access-list INSIDE_FILTER extended permit tcp any4 any4 eq www

access-list INSIDE_FILTER extended permit tcp any4 any4 eq 8080

logging host inside 10.1.1.15

flow-export destination inside 10.1.1.15 2055

ip verify reverse-path interface inside

ip verify reverse-path interface outside

nat (inside,outside) source static LAN_NETWORKS LAN_NETWORKS destination static BRANCH_NETWORKS BRANCH_NETWORKS route-lookup

nat (inside,outside) source static BRANCH_NETWORKS BRANCH_NETWORKS destination static NETWORK_MGMT NETWORK_MGMT route-lookup

nat (inside,outside) source dynamic any interface

!

object network obj_any

nat (inside,outside) dynamic interface

access-group FROM_OUTSIDE in interface outside

route outside 0.0.0.0 0.0.0.0 <outside ip gateway> 1

route outside 10.1.1.15 255.255.255.255 <outside ip datacenter asa> 1

management-access inside

threat-detection basic-threat

threat-detection statistics access-list

no threat-detection statistics tcp-intercept

tunnel-group <outside ip datacenter asa> type ipsec-l2l

tunnel-group <outside ip datacenter asa> ipsec-attributes

ikev1 pre-shared-key *****

class-map type regex match-any DomainBlockList

match regex DomainList-Netflix

class-map type inspect http match-all BlockDomainsClass

match request header host regex class DomainBlockList

class-map inspection_default

match default-inspection-traffic

class-map httptraffic

match access-list INSIDE_FILTER

!

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto

message-length maximum 512

policy-map type inspect http http_inspection_policy

parameters

protocol-violation action log

class BlockDomainsClass

reset log

policy-map URL-filter-policy

class httptraffic

inspect http http_inspection_policy

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect rsh

inspect rtsp

inspect esmtp

inspect sqlnet

inspect skinny

inspect sunrpc

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

inspect ip-options

inspect http

class class-default

flow-export event-type all destination 10.1.1.15

!

service-policy URL-filter-policy interface inside

prompt hostname context

Datacenter ASA Config Parts:

ASA Version 9.0(1)

!

hostname DATACENTERASA5540

xlate per-session deny tcp any4 any4

xlate per-session deny tcp any4 any6

xlate per-session deny tcp any6 any4

xlate per-session deny tcp any6 any6

xlate per-session deny udp any4 any4 eq domain

xlate per-session deny udp any4 any6 eq domain

xlate per-session deny udp any6 any4 eq domain

xlate per-session deny udp any6 any6 eq domain

xlate per-session deny tcp any4 any4

xlate per-session deny tcp any4 any6

xlate per-session deny tcp any6 any4

xlate per-session deny tcp any6 any6

xlate per-session deny udp any4 any4 eq domain

xlate per-session deny udp any4 any6 eq domain

xlate per-session deny udp any6 any4 eq domain

xlate per-session deny udp any6 any6 eq domain

names

interface GigabitEthernet0/0

description *** TO OUTSIDE NETWORK AT DATACENTER ***

speed 100

duplex full

nameif OUTSIDE

security-level 0

ip address <outside ip>

!

interface GigabitEthernet0/1

description *** TO INSIDE NETWORK ***

nameif INSIDE

security-level 100

ip address 10.1.3.2 255.255.255.0

!

same-security-traffic permit inter-interface

same-security-traffic permit intra-interface

object-group network LAN_NETWORKS

network-object 10.0.0.0 255.0.0.0

network-object 134.200.131.0 255.255.255.0

network-object 134.200.220.0 255.255.255.0

network-object 134.201.2.0 255.255.255.0

network-object 163.243.195.0 255.255.255.0

network-object 172.16.0.0 255.240.0.0

network-object 192.168.0.0 255.255.0.0

network-object 10.1.3.0 255.255.255.0

network-object 10.31.2.0 255.255.255.0

network-object 10.1.1.0 255.255.255.0

network-object 172.26.1.0 255.255.255.0

object-group network DATACENTER_NETWORKS

network-object 10.1.0.0 255.255.0.0

object-group network BRANCH_NETWORKS

network-object 10.15.6.0 255.255.254.0

access-list BRANCH_VPN_ACL remark ****************************************************

access-list BRANCH_VPN_ACL remark * FOR SITE TO SITE VPN TO BRANCH WV USA *

access-list BRANCH_VPN_ACL remark ****************************************************

access-list BRANCH_VPN_ACL extended permit ip host <outside ip> host <outside ip branch asa>

access-list BRANCH_VPN_ACL extended permit ip object-group LAN_NETWORKS object-group BRANCH_NETWORKS

flow-export destination INSIDE 10.1.1.15 2055

flow-export template timeout-rate 1

flow-export delay flow-create 180

ip verify reverse-path interface OUTSIDE

ip verify reverse-path interface INSIDE

no failover

nat (INSIDE,OUTSIDE) source static LAN_NETWORKS LAN_NETWORKS destination static BRANCH_NETWORKS BRANCH_NETWORKS route-lookup

access-group FROM_OUTSIDE in interface OUTSIDE

route OUTSIDE 0.0.0.0 0.0.0.0 <outside ip> 1

route INSIDE 10.0.0.0 255.0.0.0 10.1.3.1 1

route OUTSIDE 10.15.6.0 255.255.254.0 <outside ip branch asa> 1

crypto map OUTSIDE-MAP 156 match address BRANCH_VPN_ACL

crypto map OUTSIDE-MAP 156 set pfs

crypto map OUTSIDE-MAP 156 set peer <outside ip branch asa>

crypto map OUTSIDE-MAP 156 set ikev1 transform-set ESP-3DES-MD5 ESP-3DES-SHA

tunnel-group <outside ip branch asa> type ipsec-l2l

tunnel-group <outside ip branch asa> ipsec-attributes

ikev1 pre-shared-key *****

!

class-map inspection_default

match default-inspection-traffic

!

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect rsh

inspect rtsp

inspect esmtp

inspect sqlnet

inspect skinny

inspect sunrpc

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

inspect ip-options

class class-default

flow-export event-type all destination 10.1.1.15

user-statistics accounting

!

service-policy global_policy global

smtp-server 172.19.1.137

prompt hostname context

call-home reporting anonymous

Again, any help you can provide is appreciated... will vote for best...

Jouni Forss · ‎11-07-2013

Hi,

First off it seems to me that you are using the wrong IP address in the "packet-tracer" command as the source.

Atleast I presume you meant to use the interface IP address of 10.15.6.1 but you are using 10.15.16.1.

Then again if you were to use the actual interface IP address of the "inside" interface of the ASA5505 then that "packet-tracer" should also fail since you are using the interface IP address as the source.

This whole situation reminds me of a similiar setup I tested for someone. Though in that case we built a L2L VPN connection between Remote Site and Central Site and used it to send the remote ASA Syslogs through the VPN and also use SNMP for the remote ASA through the L2L VPN. That worked great.

If this traffic generated from the remote ASA itself is anything like Syslog and SNMP then it would seem to me that the problem is in the Crypto ACL you are using.

I notice that you have used an ACL line that specifies the external IP addresses of the ASA firewalls (Central and Remote Site). In your case since the flow export destination is 10.1.1.15 shouldnt you be specifying the remote site ASAs Crypto ACL as the following?

Remote Site

access-list DATACENTER_VPN_ACL extended permit ip host host 10.1.1.15

Central Site

access-list BRANCH_VPN_ACL extended permit ip host 10.1.1.15 host

In addtion to that I think the remote ASA would not need any NAT configurations related to this as its traffic generated from the device itself. On the other hand I think the Central Site would required a NAT0 configuration addition as at that side the the host involved in the flow export is not the ASA but a device behind that.

So I think you would need

object network FLOW-EXPORT

host 10.1.1.15

object network REMOTE-SITE-ASA-PUBLIC

host

nat (INSIDE,OUTSIDE) 1 source static FLOW-EXPORT FLOW-EXPORT destination static REMOTE-SITE-ASA-PUBLIC REMOTE-SITE-ASA-PUBLIC

The above NAT configuration would naturally mean that ALL this internal hosts traffic towards the remote ASA public IP address would use the L2L VPN connection rather than the public Internet (wihout VPN)

Hope this helps

Let me know what the situation is

- Jouni

ken.montgomery · ‎11-11-2013

Hmm... that is interesting considering we are using similar setups to what I originally posted, with a layer 3 switch behind them and working fine. let me take a look at it... I'm not sure that what you're proposing will resolve it or not...

harshisi_2 · ‎11-07-2013

Hi Ken,

can you please run the packet tracer command as follows :

packet input inside icmp 10.15.16.2 8 0 10.1.1.15 detailed

and share results.

Regards,

~Harry

ken.montgomery · ‎11-11-2013

I ran it, with the source IP corrected (it is 10.15.6.2):

BRANCHASA# packet input inside icmp 10.15.6.2 8 0 10.1.1.15 detailed

Phase: 1

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

Forward Flow based lookup yields rule:

in id=0xcb0b6698, priority=1, domain=permit, deny=false

hits=1203279, user_data=0x0, cs_id=0x0, l3_type=0x8

src mac=0000.0000.0000, mask=0000.0000.0000

dst mac=0000.0000.0000, mask=0100.0000.0000

input_ifc=inside, output_ifc=any

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 0.0.0.0 0.0.0.0 outside

Phase: 3

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

nat (inside,outside) source static BRANCH_NETWORKS BRANCH_NETWORKS destination static NETWORK_MGMT NETWORK_MGMT route-lookup

Additional Information:

NAT divert to egress interface outside

Untranslate 10.1.1.15/0 to 10.1.1.15/0

Phase: 4

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 10.15.6.0 255.255.254.0 inside

Phase: 5

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (inside,outside) source static BRANCH_NETWORKS BRANCH_NETWORKS destination static NETWORK_MGMT NETWORK_MGMT route-lookup

Additional Information:

Static translate 10.15.6.2/0 to 10.15.6.2/0

Forward Flow based lookup yields rule:

in id=0xcb12f2f0, priority=6, domain=nat, deny=false

hits=15824, user_data=0xcb0fdef8, cs_id=0x0, flags=0x0, protocol=0

src ip/id=10.15.6.0, mask=255.255.254.0, port=0, tag=0

dst ip/id=10.0.0.0, mask=255.0.0.0, port=0, tag=0, dscp=0x0

input_ifc=inside, output_ifc=outside

Phase: 6

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xcaa712e0, priority=0, domain=nat-per-session, deny=true

hits=77610, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0

src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0

input_ifc=any, output_ifc=any

Phase: 7

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xcb0bc128, priority=0, domain=inspect-ip-options, deny=true

hits=91404, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0

input_ifc=inside, output_ifc=any

Phase: 8

Type: INSPECT

Subtype: np-inspect

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0xcb0bbc28, priority=66, domain=inspect-icmp-error, deny=false

hits=4585, user_data=0xcb0bb238, cs_id=0x0, use_real_addr, flags=0x0, protocol=1

src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0

dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0

input_ifc=inside, output_ifc=any

Phase: 9

Type: VPN

Subtype: encrypt

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

out id=0xcb0c1218, priority=70, domain=encrypt, deny=false

hits=708, user_data=0xbf63c, cs_id=0xcb9ad918, reverse, flags=0x0, protocol=0

src ip/id=10.15.6.0, mask=255.255.254.0, port=0, tag=0

dst ip/id=10.0.0.0, mask=255.0.0.0, port=0, tag=0, dscp=0x0

input_ifc=any, output_ifc=outside

Phase: 10

Type: NAT

Subtype: rpf-check

Result: ALLOW

Config:

nat (inside,outside) source static BRANCH_NETWORKS BRANCH_NETWORKS destination static NETWORK_MGMT NETWORK_MGMT route-lookup

Additional Information:

Forward Flow based lookup yields rule:

out id=0xcb12fb00, priority=6, domain=nat-reverse, deny=false

hits=15837, user_data=0xcb124438, cs_id=0x0, use_real_addr, flags=0x0, protocol=0

src ip/id=10.15.6.0, mask=255.255.254.0, port=0, tag=0

dst ip/id=10.0.0.0, mask=255.0.0.0, port=0, tag=0, dscp=0x0

input_ifc=inside, output_ifc=outside

Phase: 11

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 143081, packet dispatched to next module

Module information for forward flow ...

snp_fp_tracer_drop

snp_fp_inspect_ip_options

snp_fp_translate

snp_fp_adjacency

snp_fp_encrypt

snp_fp_fragment

snp_ifc_stat

Module information for reverse flow ...

Result:

input-interface: inside

input-status: up

input-line-status: up

output-interface: outside

output-status: up

output-line-status: up

Action: allow

ken.montgomery · ‎11-12-2013

Turns out the problem is neither.

The actions below, discovered by one of my staff members, repairs it:

The fix is to apply the default inspection globally

service-policy global_policy global

On old codes, default inspection is enabled by default thus it does not show up on show run. So on the new versions although we have the default policy maps, this will not take effect unless we “explicitly” apply the service policy:

.

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

.

class class-default

flow-export event-type all destination 10.1.1.15

!

service-policy global_policy global

So, thank you all for attempting, but that seems to fix it