ASA 5505 LAN to LAN Issues

[poppascotch]

Hi everyone, I was hoping that I could get some help for an issue we are having and i'm about to rip my hair out.

Out setup consists of two locations, one ASA 5505 (security license) \ at our main site and the other is a remote site with a cisco 1921 acting as the edge device.  

The ASA 5505 acting as an edge device at our main site.  From our ISP, two vlans come into the ASA (one for public internet traffic, one for the remote site, set up as a VC).  The main site and the remote site are both separate LAN subnets, with a third subnet acting as a serial between the two locations.

At our main site, the ASA can access the public internet just fine, it can also ping to the gateway address on the 1921 (for their lan10.34.60.245: below) and receive a reply.  This tells me that the Route commands are all set up fine as well as NAT translations to the public internet.  The idea in this case is to have the remote site send all data back to the ASA5505 (think of the VC as one long cable connecting the two) and the ASA will handle the actual public internet connectivity as well as allowing connectivity to their private LAN (to access servers).  I.E both LANs need to be able to talk to eachother. 

the remote site as we stand now is able to ping the other end of the serial IP (10.1.1.1) but that's it.  It can't ping the main site LAN gateway and it can't ping anything on the public internet. 

I've narrowed the problem down to something on this ASA that isn't allowing these private LANs to communicate however I have no idea what it is.  Any help would be very much appreciated.

here is some info to help

REMOTE SITE LAN =

10.34.60.0/24 (gateway is 10.34.60.245)

Serial IP on the VC: 10.1.1.0/30

ASA sh run:

interface Vlan1

nameif inside

security-level 100

ip address 10.25.102.245 255.255.255.0

!

interface Vlan783

nameif Internet

security-level 0

ip address X.X.X.X 255.255.255.252

!

interface Vlan789

nameif remotesite

security-level 100

ip address 10.1.1.1 255.255.255.252

!

interface Ethernet0/0

switchport trunk allowed vlan 783,789

switchport mode trunk

speed 10

duplex full

ftp mode passive

clock timezone EST -5

clock summer-time EDT recurring

dns server-group DefaultDNS

domain-name workgroup

same-security-traffic permit inter-interface

same-security-traffic permit intra-interface

access-list ingress extended deny ip 0.0.0.0 0.255.255.255 any

access-list ingress extended deny ip 127.0.0.0 255.255.255.0 any

access-list ingress extended deny ip 169.254.0.0 255.255.0.0 any

access-list ingress extended deny ip 172.16.0.0 255.255.240.0 any

access-list ingress extended deny ip 192.0.2.0 255.255.255.0 any

access-list ingress extended deny ip 192.168.0.0 255.255.0.0 any

access-list ingress extended deny ip 224.0.0.0 255.255.255.224 any

access-list ingress extended permit ip any any

access-list inside_nat0_outbound extended permit ip 10.25.102.0 255.255.255.0 10.34.60.0 255.255.255.0

access-list egress extended permit ip any any

access-list remotesite extended permit ip any any

pager lines 24

logging buffer-size 40960

mtu inside 1500

mtu Internet 1500

mtu remotesite1500

no failover

icmp unreachable rate-limit 1 burst-size 1

icmp permit any inside

icmp permit any Internet

icmp permit any remotesite

no asdm history enable

arp timeout 14400

global (inside) 11 interface

global (Internet) 1 interface

global (remotesite) 10 interface

nat (inside) 0 access-list inside_nat0_outbound

nat (inside) 1 10.25.102.0 255.255.255.0

nat (inside) 10 0.0.0.0 0.0.0.0

nat (remotesite) 1 10.34.60.0 255.255.255.0

nat (remotesite) 11 0.0.0.0 0.0.0.0

static (inside,remotesite) 10.25.102.100 10.25.102.100 netmask 255.255.255.255

static (inside,remotesite) 10.1.1.100 10.1.1.100 netmask 255.255.255.255

access-group egress in interface inside

access-group ingress in interface Internet

access-group remotesite in interface remotesite

route Internet 0.0.0.0 0.0.0.0 204.186.244.193 1

route inside 10.25.102.0 255.255.255.0 10.25.102.145 1

route remotesite10.34.60.0 255.255.255.0 10.1.1.2 1

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

dynamic-access-policy-record DfltAccessPolicy

aaa authentication ssh console LOCAL

no snmp-server location

no snmp-server contact

snmp-server enable traps snmp authentication linkup linkdown coldstart

crypto ipsec security-association lifetime seconds 28800

crypto ipsec security-association lifetime kilobytes 4608000

ssh timeout 5

console timeout 0

[Julio Carvajal]

Hello John,

Can you provide the following output :

-packet-tracer input remote site tcp 10.34.60.15 1025 10.25.102.15 80

Regards,

Julio

[poppascotch]

Sure, here it is

Phase: 1

Type: ACCESS-LIST

Subtype:

Result: ALLOW

Config:

Implicit Rule

Additional Information:

MAC Access list

Phase: 2

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 3

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   10.25.102.0     255.255.255.0   inside

Phase: 4

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Config:

access-group remotesite in interface remotesite

access-list remotesite extended permit ip any any

Additional Information:

Phase: 5

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 6

Type: NAT-EXEMPT

Subtype: rpf-check

Result: ALLOW

Config:

Additional Information:

Phase: 7

Type: NAT

Subtype:

Result: DROP

Config:

nat (remotesite) 1 10.34.60.0 255.255.255.0

  match ip remotesite 10.34.60.0 255.255.255.0 inside any

    dynamic translation to pool 1 (No matching global)

    translate_hits = 1, untranslate_hits = 0

Additional Information:

Result:

input-interface: remotesite

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

[Julio Carvajal]

Hello John,

Thanks for the information. We are almost there

Please add the following and let me know the result

No nat (remotesite) 1 10.34.60.0 255.255.255.0

Then try it and if it does not work, use the packet tracer one more time and let me know the result.

Regards,

Do please rate helpful posts

Julio

[poppascotch]

I'll give it a shot tomorrow and report what I find, man I hope this fixes it.  Spent too many hours trying to figure it out.....

EDIT: I'm feeling confident about this, tried the packet-tracer command again and got the following:

Phase: 12

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 10026, packet dispatched to next module

Result:

input-interface: remotesite

input-status: up

input-line-status: up

output-interface: inside

output-status: up

output-line-status: up

Action: allow

Won't know tomorrow until i actually deploy it, but so far it looks good.  I'll let you know what happens.

and also, thank you very much

[Julio Carvajal]

Hello John,

Sure, let me know.

The packet tracer show us that the packets comming from the Remote site are not maching any global statement, so that should do it.

Edit: That is great! Seems that its gonna work.

Regards,

Julio

[poppascotch]

Looks like I'm still having issues.

the ASA can't ping the router's interface IP of 10.1.1.2 over the VC anymore, previously it could

Here is the packet tracer command, showing it is getting dropped

Phase: 1

Type: FLOW-LOOKUP

Subtype:

Result: ALLOW

Config:

Additional Information:

Found no matching flow, creating a new flow

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   10.25.102.0     255.255.255.0   inside

Phase: 3

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Config:

access-group remotesitein interface remotesite

access-list remotesite extended permit ip any any

Additional Information:

Phase: 4

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 5

Type: NAT

Subtype:

Result: DROP

Config:

nat (remotesite) 5 10.1.1.0 255.255.255.252

  match ip remotesite 10.1.1.0 255.255.255.252 inside any

    dynamic translation to pool 1 (No matching global)

    translate_hits = 1, untranslate_hits = 0

Additional Information:

Result:

input-interface: remotesite

input-status: up

input-line-status: up

output-interface: inside

Here are the NAT and global commands:

global (inside) 11 interface

global (Internet) 1 interface

global (remotesite) 10 interface

nat (inside) 0 access-list inside_nat0_outbound

nat (inside) 1 10.25.102.0 255.255.255.0

nat (inside) 10 0.0.0.0 0.0.0.0

nat (remotesite) 1 10.1.1.0 255.255.255.252

nat (remotesite) 1 10.34.60.0 255.255.255.0

nat (remotesite) 11 0.0.0.0 0.0.0.0

static (inside,remotesite) 10.25.102.100 10.25.102.100 netmask 255.255.255.255

static (inside,remotesite) 10.1.1.100 10.1.1.100 netmask 255.255.255.255

[Julio Carvajal]

Hello Jonh,

Are those the only nat statements we have in there:

because packet tracer shows this one:

nat (remotesite) 5 10.1.1.0 255.255.255.252

And the nats you provide are the following:

global (inside) 11 interface

global (Internet) 1 interface

global (remotesite) 10 interface

nat (inside) 0 access-list inside_nat0_outbound

nat (inside) 1 10.25.102.0 255.255.255.0

nat (inside) 10 0.0.0.0 0.0.0.0

nat (remotesite) 1 10.1.1.0 255.255.255.252

nat (remotesite) 1 10.34.60.0 255.255.255.0

nat (remotesite) 11 0.0.0.0 0.0.0.0

static (inside,remotesite) 10.25.102.100 10.25.102.100 netmask 255.255.255.255

static (inside,remotesite) 10.1.1.100 10.1.1.100 netmask 255.255.255.255

I do not see Nat (remotesite) 5 in there.

From the ASA we have not change something that could affect that the ASA could ping the other site, so seems like something else change on the behavior of the router because it is not allowing pings anymore.

Please take out the following statements:

no nat (remotesite) 1 10.1.1.0 255.255.255.252

no nat (remotesite) 1 10.34.60.0 255.255.255.0

Regards,

Julio

[poppascotch]

nat (remotesite) 5 10.1.1.0 255.255.255.252 was set up when I was troublshooting and has since been removed.

so at this point, I put in the following commands:

nat (remotesite) 1 10.1.1.0 255.255.255.252

nat (remotesite) 1 10.34.60.0 255.255.255.0

Obviously, now the packettracer command is showing drops on both the networks (serial and other LAN)

Phase: 5

Type: NAT

Subtype:

Result: DROP

Config:

nat (remotesite) 1 10.1.1.0 255.255.255.252

  match ip remotesite 10.1.1.0 255.255.255.252 inside any

    dynamic translation to pool 1 (No matching global)

    translate_hits = 1, untranslate_hits = 0

Phase: 6

Type: NAT

Subtype:

Result: DROP

Config:

nat (remotesite) 1 10.34.60.0 255.255.255.0

  match ip remotesite 10.34.60.0 255.255.255.0 inside any

    dynamic translation to pool 1 (No matching global)

    translate_hits = 1, untranslate_hits = 0

Additional Information:

Here are the current NAT settings I have enabled.

global (inside) 11 interface

global (Internet) 1 interface

global (remotesite) 10 interface

nat (inside) 0 access-list inside_nat0_outbound

nat (inside) 1 10.25.102.0 255.255.255.0

nat (inside) 10 0.0.0.0 0.0.0.0

nat (remotesite) 1 10.1.1.0 255.255.255.252

nat (remotesite) 1 10.34.60.0 255.255.255.0

nat (remotesite) 11 0.0.0.0 0.0.0.0

static (inside,remotesite) 10.25.102.100 10.25.102.100 netmask 255.255.255.255

static (inside,remotesite) 10.1.1.100 10.1.1.100 netmask 255.255.255.255

I tried taking removing the no nat commands for the 10. networks and thought that maybe I could make a sweeping no nat command of just 10.0.0.0 255.0.0.0, but I got the error "ERROR: nat element not found"

I checked with someone on site and the router is on and working, so its not that.  Also in the routing table, 10.1.1.0 shows up as directly connected, no there has to be a link there at the moment.

[Julio Carvajal]

Hello John,

I want you to remove the nat (remotesite) 1 as I asked on previous post

no nat (remotesite) 1 10.1.1.0 255.255.255.252

no nat (remotesite) 1 10.34.60.0 255.255.255.0

The ASA is directly connected to the router so they should be able to ping each other,

capture test interface remotesite match icmp x.x.x.x ( router other site) x.x.x.x (this asa remote site interface)

Now try to ping from the router to the asa

Please provide show capture test, packet tracer again and show nat and global

Regards,

[poppascotch]

Sorry, I misunderstood you.

Anywho,

I reinstated the no nat commands and whenever I try to do that

capture test interface remotesite match icmp x.x.x.x ( router other site) x.x.x.x (this asa remote site interface) command, it reports with error :"ERROR: ERROR: IP address,mask doesn't pair", which doesn't made sense to me at all

I'm beginning to think that this is an issue with the router opposite the ASA.  I'm about 99% sure that the config changes I've made in the past few days haven't altered anything that would have denied access to that router.  the only thing I can think at this point is that the router lost the link, or for some reason lost its config.  This would be unfortunate since its about 20 miles away from me right now.

I'm including the sh run here to see if there are any issues that I may be missing that wouldn't allow the ping requests to 10.1.1.2, but it looks like i'll be taking a drive

!

hostname ciscoasa

domain-name workgroup

names

!

interface Vlan1

nameif inside

security-level 100

ip address 10.25.102.245 255.255.255.0

!

interface Vlan783

nameif Internet

security-level 0

ip address 204.186.244.194 255.255.255.252

!

interface Vlan789

nameif remotesite

security-level 100

ip address 10.1.1.1 255.255.255.252

!

interface Ethernet0/0

switchport trunk allowed vlan 783,789

switchport mode trunk

speed 10

duplex full

!

interface Ethernet0/1

!

banner motd -----------------------------------------------------------------------------------------------------------

banner motd -------------------------------------------------------------------------

boot system disk0:/asa822-k8.bin

ftp mode passive

clock timezone EST -5

clock summer-time EDT recurring

dns server-group DefaultDNS

domain-name workgroup

same-security-traffic permit inter-interface

same-security-traffic permit intra-interface

access-list ingress extended deny ip 0.0.0.0 0.255.255.255 any

access-list ingress extended deny ip 127.0.0.0 255.255.255.0 any

access-list ingress extended deny ip 169.254.0.0 255.255.0.0 any

access-list ingress extended deny ip 172.16.0.0 255.255.240.0 any

access-list ingress extended deny ip 192.0.2.0 255.255.255.0 any

access-list ingress extended deny ip 192.168.0.0 255.255.0.0 any

access-list ingress extended deny ip 224.0.0.0 255.255.255.224 any

access-list ingress extended permit ip any any

access-list inside_nat0_outbound extended permit ip 10.25.102.0 255.255.255.0 10.34.60.0 255.255.255.0

access-list egress extended permit ip any any

access-list remotesite extended permit ip any any

pager lines 24

logging buffer-size 40960

mtu inside 1500

mtu Internet 1500

mtu remotesite1500

no failover

icmp unreachable rate-limit 1 burst-size 1

icmp permit any inside

icmp permit any Internet

icmp permit any remotesite

no asdm history enable

arp timeout 14400

global (inside) 11 interface

global (Internet) 1 interface

global (remotesite) 10 interface

nat (inside) 0 access-list inside_nat0_outbound

nat (inside) 1 10.25.102.0 255.255.255.0

nat (inside) 10 0.0.0.0 0.0.0.0

nat (remotesite) 11 0.0.0.0 0.0.0.0

static (inside,remotesite) 10.25.102.100 10.25.102.100 netmask 255.255.255.255

static (inside,remotesite) 10.1.1.100 10.1.1.100 netmask 255.255.255.255

access-group egress in interface inside

access-group ingress in interface Internet

access-group remotesite in interface remotesite

route Internet 0.0.0.0 0.0.0.0 204.186.244.193 1

route inside 10.25.102.0 255.255.255.0 10.25.102.145 1

route remotesite 10.34.60.0 255.255.255.0 10.1.1.2 1

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

dynamic-access-policy-record DfltAccessPolicy

aaa authentication ssh console LOCAL

no snmp-server location

no snmp-server contact

snmp-server enable traps snmp authentication linkup linkdown coldstart

crypto ipsec security-association lifetime seconds 28800

crypto ipsec security-association lifetime kilobytes 4608000

console timeout 0

threat-detection basic-threat

threat-detection statistics access-list

no threat-detection statistics tcp-intercept

!

class-map inspection_default

match default-inspection-traffic

!

!

policy-map type inspect dns preset_dns_map

parameters

  message-length maximum 512

policy-map global_policy

class inspection_default

  inspect ftp

  inspect h323 h225

  inspect h323 ras

  inspect rsh

  inspect rtsp

  inspect esmtp

  inspect sqlnet

  inspect skinny

  inspect sunrpc

  inspect xdmcp

  inspect sip

  inspect netbios

  inspect tftp

[Julio Carvajal]

Hello John,

That is correct, and you are pinging from the ASA right?

That should work.. Regards,

Julio

[poppascotch]

yeah, i was pinging from the ASA to 10.1.1.2 the entire time.

I'll be going out tomorrow to check it out and fix the issue, I'll report back as to how it went.

Thank you very much for your help jcarvaja, I really appreciate it

[Julio Carvajal]

Hello John,

Sure let me know.

My pleasure!

Julio

[poppascotch]

Still isn't working.  the remote site router can ping the IP of the ASA interface at 10.1.1.1 with no problem.  Whenever it tries to ping the LAN of the ASA or the public internet, it fails. 

The ASA can ping botht the serial IP of 10.1.1.2 and the LAN of the router with no problem.