ASA5510 - 1 site can access internet , 2 others cannot , why?

[cadek1fraen]

I have 3 locations that are interconnected with an MPLS type of cloud provided by an ISP , it is transparent to me , currently I have all inter company traffic working but only site 1 is able to reach the internet. I'm running out of ideas and could use some more things to look at or troubleshooting steps.

this is the network diagram

site 3 uses 192.168.3.0/24
site 2 is 2.0/24
site 1 is 1.0/24
(just FYI so diagram makes more sense)

each PC in each site has its gateway set to its local router, so 2.100 (PC) has a gateway of 2.1 (its router in site 2) , 3.100 (PC) has a gateway of 3.1 (its router in site 3) etc..

All sites can reach all other sites on private subnets
for example: 192.168.3.1 can ping 2.1 and 1.1
or 2.1 can ping 3.1 and 1.1 , 100% connectivity seems to exist there.

but... only the 1.0/24 site can get out to the internet!

more examples:

1.100 (PC) can ping 1.1 (Firewall)
2.100 (PC) cannot ping 1.1 (firewall)
2.100 (PC) can ping 1.100 (PC)
1.100 (PC) can ping outside ip on internet

2.100 (PC) cannot ping outside ip on internet

there is only 1 firewall for all 3 sites, all internet traffic should go out through this one firewall, all inter-company traffic does not need to be inspected by the firewall. In theory it is a good setup (in theory, lol)

I need basic ideas of what to try at this point as I'm out of ideas.

My only route is one static route of 0.0.0.0 0.0.0.0 next_hop_IP , clearly this works for my "connected subnet" as internet access is working, why this does not work for my other two subnets is beyond me.

should I somehow specify in the firewall config that traffic from 2.0/24 and 3.0/24 is allowed?

I am trying to configure traceroutes to pass through, I did add inspect icmp to the global config and I can ping from 1.0/24 everywhere, I'm *assuming* this should allow a PC in 2.0/24 or 3.0/24 to also ping and get a reply but that's just an assumption on my part.

I don't know for sure if packets (lets say ping) from 2.100 is actually getting to 1.1 (firewall) , I'm not sure how to test that either at this point. It may just be the firewall dropping the ICMP replies to the other 2 subnets or maybe the packets don't even get there.

any futher help will once again be greatly appreciated! Thank you

[varrao]

Hi Martin,

Could you please provide a configuration from your firewall, that would be really helpful, difficult tos ay why its not working, but yes if yolu do not have any nat command for the two networks, they wont be able to access internet.

Thanks,

Varun

[cadek1fraen]

Here it is, that's good news about the NAT, I didn't think I had to do that so that might be my solution. Please advise with what commands I should add.

: Saved
:
ASA Version 8.2(1)
!
hostname ciscoasa
enable password
passwd
names
!
interface Ethernet0/0
nameif EXTERNAL
security-level 0
ip address 123.123.123.147 255.255.255.240
!
interface Ethernet0/1
nameif PROTECTED
security-level 100
ip address 192.168.1.1 255.255.255.0
!
interface Ethernet0/2
shutdown
no nameif
no security-level
no ip address
!
interface Ethernet0/3
shutdown
no nameif
no security-level
no ip address
!
interface Management0/0
nameif management
security-level 100
ip address 192.168.10.1 255.255.255.0
management-only
!
ftp mode passive
pager lines 24
logging asdm informational
mtu management 1500
mtu PROTECTED 1500
mtu EXTERNAL 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
icmp permit any PROTECTED
icmp permit any EXTERNAL
no asdm history enable
arp timeout 14400
global (EXTERNAL) 101 interface
nat (PROTECTED) 101 0.0.0.0 0.0.0.0
route EXTERNAL 0.0.0.0 0.0.0.0 123.123.123.145 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
dynamic-access-policy-record DfltAccessPolicy
http server enable
http 192.168.10.0 255.255.255.0 management
http 192.168.1.0 255.255.255.0 PROTECTED
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
telnet 0.0.0.0 0.0.0.0 management
telnet 192.168.1.0 255.255.255.0 PROTECTED
telnet timeout 5
ssh timeout 5
console timeout 0
dhcpd address 192.168.1.100-192.168.1.250 PROTECTED
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
  message-length maximum 512
policy-map global_policy
class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect rsh
  inspect rtsp
  inspect esmtp
  inspect sqlnet
  inspect skinny 
  inspect sunrpc
  inspect xdmcp
  inspect sip 
  inspect netbios
  inspect tftp
  inspect icmp
!
service-policy global_policy global
prompt hostname context
Cryptochecksum
: end
no asdm history enable

[varrao]

Hi Martin,

There is nothing wrong with nat:

global (EXTERNAL) 101 interface

nat (PROTECTED) 101 0.0.0.0 0.0.0.0

all the suers behind the PROTECTED interface aere allowed internet access. Two things you need to check now, first captures and sercond logs.

Collected the captures on the ASA and take the logs as well when the traffic gets denied by the ASA.

For taking captures:

https://supportforums.cisco.com/docs/DOC-1222

What you need to chek in the captures is, if traffic from the two subnets is reaching the firewall and if yes, does it leave out from the EXTERNAL interface.

Thanks,

Varun

[cadek1fraen]

thanks for the link, I will do this right now and will post results. Maybe the traffic isn't even reaching the firewall which would at least set me on the right path.

[varrao]

Thats right, I suspect that too.

[cadek1fraen]

well now I am thoroughly confused, lol

I did the capture for ICMP traffic only , which worked...

locally I pinged from 1.123 to a public IP

16: 11:54:07.164847 192.168.1.123 > xxx.228: icmp: echo request
17: 11:54:07.264924 xxx.228 > 192.168.1.123: icmp: echo reply

ok good I thought, then..

from site 3 , 3.1 (telneted into the router) I used routers ping utility to ping 1.1 (firewall)

1: 11:51:40.014662 xxx.150 > 192.168.1.1: icmp: echo request
2: 11:51:42.014220 xxx.150 > 192.168.1.1: icmp: echo request

no replies??? but at least it got there

here's the real confusing part

from site 3, 3.1 pinged outside ip (same as in first test above)

NOTHING in the logs at all

[varrao]

Hi Martin,

Try pinging the ip address 4.2.2.2 from a machine in site 3 and take the captures on the ASA, do not get confused with any thing, just check on the routers, whether you have a route pointing towards the ASA PROTECTED interface, thats it, do not check anything, this simple thing would clear out things for us:

access-list cap permit ip host 192.168.3.1 host 4.2.2.2

access-list cap permit ip host 4.2.2.2 host 192.168.3.1

access-list cap permit ip host 123.123.123.147 host 4.2.2.2

access-list cap permit ip host 4.2.2.2 host 123.123.123.147

capture capin access-list cap interface PROTECTED

capture capout access-list cap interface EXTERNAL

after applying these, ping 4.2.2.2

and check "show capture capin" and show capture capout"

does it show anything????

Thanks,

Varun

[cadek1fraen]

no I get 0 captures, but (there's always a but )

a direct ping to 192.168.1.1 from 3.1 results in 5 icmp: echo requests showing up in the logs, but 0 replies! that part I don't understand

but pinging 4.2.2.2. results in 0 packets captured. My ISP is continuously saying that there's nothing wrong and that the firewall is dropping packets for the 2 subnets in question , 3.0/24 and 2.0/24

[mvsheik123]

Hi,

1. As Varun mentioned above, make sure that you have defaulr route statements at remote sites pointing to ASA.

2. Also, as the ASA not running any dynamic routing protocol, you need to add route statements for remote sites subnets on ASA pointing back to routers. route PROTECTED 192.168.2.0 255.255.255.0 1

hth

MS

[cadek1fraen]

I will check on both and will add the routes you mentioned, that might make a big difference. I'll report back

[lusandi]

Marin,

I hope you are doing great,

Also I would like to remind you that in order to verify if the ASA is dropping packets you can do a:

capture asp-drop type asp-drop all

and after that you can do:

show capture asp-drop | include (ip address receiving the ICMP packets)

I hope this will be helpful.

Regards,

Luis Sandi

[cadek1fraen]

well.. I'm leaning towards this being just a routing issue, but I cannot figure it out for the life of me.

I added the routes back to 2.0/24 and 3.0/24 , the gateway for those was 192.168.1.253 which is the internal port of the router (orange in site 1 router in diagram above). This should work as that router has all the other necessary routes to get elsewhere which I can confirm works, but it accomplished nothing. I also tried using the public IP of the remote sites as the gateway for these routes, same result.

does this make sense?

from a PC (1.100) I can ping the public IP of router in site 2 or 3

from the firewall (1.1) using the ADSM ping tool I cannot get to either of those IPs.

something is very wrong, if I only knew what, lol

[mvsheik123]

Hi,

Please try by enabling enable 'same-security-traffic permit intra-interface' on ASA. Also, I don't think it is required but you can enable same-security-traffic permit inter-interface as well. If you still have issues, try to ping inside ip of the ASA from remote site.

if this fails (thats what iam expecting ;-)): then go a step back- ping the site 1 router IP (192.168.1.253) ...go backwards and see where you get the reply.

if you can ping ASA inside IP: enable debug icmp trace and try to ping public IPs. you should see on ASA the logs.

hth

MS

[cadek1fraen]

I've made some changes and some progress

right now a PC 3.100 can ping firewall's PROTECTED interface (inside) , so 3.100 -> 1.1 works , now I even get replies back for the pings (not just see echo requests on firewall).

but still no internet traffic

when the same PC 3.100 tries going to 4.2.2.2 I see 0 traffic on the fw and of course it does not work.

I want to call cisco and use my smartnet, but I fear they'll just tell me to call the ISP who in turn blames everything on the firewall , oh the joys of IT!