topic ASA5510 - 1 site can access internet , 2 others cannot , why? in Network Security

ASA5510 - 1 site can access internet , 2 others cannot , why?

cadek1fraen — Mon, 11 Mar 2019 20:57:55 GMT

I have 3 locations that are interconnected with an MPLS type of cloud provided by an ISP , it is transparent to me , currently I have all inter company traffic working but only site 1 is able to reach the internet. I'm running out of ideas and could use some more things to look at or troubleshooting steps.

this is the network diagram

site 3 uses 192.168.3.0/24
site 2 is 2.0/24
site 1 is 1.0/24
(just FYI so diagram makes more sense)

each PC in each site has its gateway set to its local router, so 2.100 (PC) has a gateway of 2.1 (its router in site 2) , 3.100 (PC) has a gateway of 3.1 (its router in site 3) etc..

All sites can reach all other sites on private subnets
for example: 192.168.3.1 can ping 2.1 and 1.1
or 2.1 can ping 3.1 and 1.1 , 100% connectivity seems to exist there.

but... only the 1.0/24 site can get out to the internet!

more examples:

1.100 (PC) can ping 1.1 (Firewall)
2.100 (PC) cannot ping 1.1 (firewall)
2.100 (PC) can ping 1.100 (PC)
1.100 (PC) can ping outside ip on internet

2.100 (PC) cannot ping outside ip on internet

there is only 1 firewall for all 3 sites, all internet traffic should go out through this one firewall, all inter-company traffic does not need to be inspected by the firewall. In theory it is a good setup (in theory, lol)

I need basic ideas of what to try at this point as I'm out of ideas.

My only route is one static route of 0.0.0.0 0.0.0.0 next_hop_IP , clearly this works for my "connected subnet" as internet access is working, why this does not work for my other two subnets is beyond me.

should I somehow specify in the firewall config that traffic from 2.0/24 and 3.0/24 is allowed?

I am trying to configure traceroutes to pass through, I did add inspect icmp to the global config and I can ping from 1.0/24 everywhere, I'm *assuming* this should allow a PC in 2.0/24 or 3.0/24 to also ping and get a reply but that's just an assumption on my part.

I don't know for sure if packets (lets say ping) from 2.100 is actually getting to 1.1 (firewall) , I'm not sure how to test that either at this point. It may just be the firewall dropping the ICMP replies to the other 2 subnets or maybe the packets don't even get there.

any futher help will once again be greatly appreciated! Thank you

ASA5510 - 1 site can access internet , 2 others cannot , why?

varrao — Tue, 12 Jul 2011 18:15:36 GMT

Hi Martin,

Could you please provide a configuration from your firewall, that would be really helpful, difficult tos ay why its not working, but yes if yolu do not have any nat command for the two networks, they wont be able to access internet.

Thanks,

Varun

ASA5510 - 1 site can access internet , 2 others cannot , why?

cadek1fraen — Tue, 12 Jul 2011 18:25:03 GMT

Here it is, that's good news about the NAT, I didn't think I had to do that so that might be my solution. Please advise with what commands I should add.

: Saved
:
ASA Version 8.2(1)
!
hostname ciscoasa
enable password
passwd
names
!
interface Ethernet0/0
nameif EXTERNAL
security-level 0
ip address 123.123.123.147 255.255.255.240
!
interface Ethernet0/1
nameif PROTECTED
security-level 100
ip address 192.168.1.1 255.255.255.0
!
interface Ethernet0/2
shutdown
no nameif
no security-level
no ip address
!
interface Ethernet0/3
shutdown
no nameif
no security-level
no ip address
!
interface Management0/0
nameif management
security-level 100
ip address 192.168.10.1 255.255.255.0
management-only
!
ftp mode passive
pager lines 24
logging asdm informational
mtu management 1500
mtu PROTECTED 1500
mtu EXTERNAL 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
icmp permit any PROTECTED
icmp permit any EXTERNAL
no asdm history enable
arp timeout 14400
global (EXTERNAL) 101 interface
nat (PROTECTED) 101 0.0.0.0 0.0.0.0
route EXTERNAL 0.0.0.0 0.0.0.0 123.123.123.145 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
dynamic-access-policy-record DfltAccessPolicy
http server enable
http 192.168.10.0 255.255.255.0 management
http 192.168.1.0 255.255.255.0 PROTECTED
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
telnet 0.0.0.0 0.0.0.0 management
telnet 192.168.1.0 255.255.255.0 PROTECTED
telnet timeout 5
ssh timeout 5
console timeout 0
dhcpd address 192.168.1.100-192.168.1.250 PROTECTED
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect esmtp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect icmp
!
service-policy global_policy global
prompt hostname context
Cryptochecksum
: end
no asdm history enable

ASA5510 - 1 site can access internet , 2 others cannot , why?

varrao — Tue, 12 Jul 2011 18:31:49 GMT

Hi Martin,

There is nothing wrong with nat:

global (EXTERNAL) 101 interface

nat (PROTECTED) 101 0.0.0.0 0.0.0.0

all the suers behind the PROTECTED interface aere allowed internet access. Two things you need to check now, first captures and sercond logs.

Collected the captures on the ASA and take the logs as well when the traffic gets denied by the ASA.

For taking captures:

https://supportforums.cisco.com/docs/DOC-1222

What you need to chek in the captures is, if traffic from the two subnets is reaching the firewall and if yes, does it leave out from the EXTERNAL interface.

Thanks,

Varun

ASA5510 - 1 site can access internet , 2 others cannot , why?

cadek1fraen — Tue, 12 Jul 2011 18:33:42 GMT

thanks for the link, I will do this right now and will post results. Maybe the traffic isn't even reaching the firewall which would at least set me on the right path.

ASA5510 - 1 site can access internet , 2 others cannot , why?

varrao — Tue, 12 Jul 2011 18:36:47 GMT

Thats right, I suspect that too.

ASA5510 - 1 site can access internet , 2 others cannot , why?

cadek1fraen — Tue, 12 Jul 2011 19:06:56 GMT

well now I am thoroughly confused, lol

I did the capture for ICMP traffic only , which worked...

locally I pinged from 1.123 to a public IP

16: 11:54:07.164847 192.168.1.123 > xxx.228: icmp: echo request
17: 11:54:07.264924 xxx.228 > 192.168.1.123: icmp: echo reply

ok good I thought, then..

from site 3 , 3.1 (telneted into the router) I used routers ping utility to ping 1.1 (firewall)

1: 11:51:40.014662 xxx.150 > 192.168.1.1: icmp: echo request
2: 11:51:42.014220 xxx.150 > 192.168.1.1: icmp: echo request

no replies??? but at least it got there

here's the real confusing part

from site 3, 3.1 pinged outside ip (same as in first test above)

NOTHING in the logs at all

ASA5510 - 1 site can access internet , 2 others cannot , why?

varrao — Tue, 12 Jul 2011 19:17:30 GMT

Hi Martin,

Try pinging the ip address 4.2.2.2 from a machine in site 3 and take the captures on the ASA, do not get confused with any thing, just check on the routers, whether you have a route pointing towards the ASA PROTECTED interface, thats it, do not check anything, this simple thing would clear out things for us:

access-list cap permit ip host 192.168.3.1 host 4.2.2.2

access-list cap permit ip host 4.2.2.2 host 192.168.3.1

access-list cap permit ip host 123.123.123.147 host 4.2.2.2

access-list cap permit ip host 4.2.2.2 host 123.123.123.147

capture capin access-list cap interface PROTECTED

capture capout access-list cap interface EXTERNAL

after applying these, ping 4.2.2.2

and check "show capture capin" and show capture capout"

does it show anything????

Thanks,

Varun

ASA5510 - 1 site can access internet , 2 others cannot , why?

cadek1fraen — Tue, 12 Jul 2011 20:02:29 GMT

no I get 0 captures, but (there's always a but )

a direct ping to 192.168.1.1 from 3.1 results in 5 icmp: echo requests showing up in the logs, but 0 replies! that part I don't understand

but pinging 4.2.2.2. results in 0 packets captured. My ISP is continuously saying that there's nothing wrong and that the firewall is dropping packets for the 2 subnets in question , 3.0/24 and 2.0/24

ASA5510 - 1 site can access internet , 2 others cannot , why?

mvsheik123 — Tue, 12 Jul 2011 20:31:51 GMT

Hi,

1. As Varun mentioned above, make sure that you have defaulr route statements at remote sites pointing to ASA.

2. Also, as the ASA not running any dynamic routing protocol, you need to add route statements for remote sites subnets on ASA pointing back to routers. route PROTECTED 192.168.2.0 255.255.255.0 1

hth

ASA5510 - 1 site can access internet , 2 others cannot , why?

cadek1fraen — Tue, 12 Jul 2011 20:35:03 GMT

I will check on both and will add the routes you mentioned, that might make a big difference. I'll report back

ASA5510 - 1 site can access internet , 2 others cannot , why?

lusandi — Tue, 12 Jul 2011 23:45:55 GMT

Marin,

I hope you are doing great,

Also I would like to remind you that in order to verify if the ASA is dropping packets you can do a:

capture asp-drop type asp-drop all

and after that you can do:

show capture asp-drop | include (ip address receiving the ICMP packets)

I hope this will be helpful.

Regards,

Luis Sandi

ASA5510 - 1 site can access internet , 2 others cannot , why?

cadek1fraen — Wed, 13 Jul 2011 12:50:50 GMT

well.. I'm leaning towards this being just a routing issue, but I cannot figure it out for the life of me.

I added the routes back to 2.0/24 and 3.0/24 , the gateway for those was 192.168.1.253 which is the internal port of the router (orange in site 1 router in diagram above). This should work as that router has all the other necessary routes to get elsewhere which I can confirm works, but it accomplished nothing. I also tried using the public IP of the remote sites as the gateway for these routes, same result.

does this make sense?

from a PC (1.100) I can ping the public IP of router in site 2 or 3

from the firewall (1.1) using the ADSM ping tool I cannot get to either of those IPs.

something is very wrong, if I only knew what, lol

ASA5510 - 1 site can access internet , 2 others cannot , why?

mvsheik123 — Wed, 13 Jul 2011 13:29:49 GMT

Hi,

Please try by enabling enable 'same-security-traffic permit intra-interface' on ASA. Also, I don't think it is required but you can enable same-security-traffic permit inter-interface as well. If you still have issues, try to ping inside ip of the ASA from remote site.

if this fails (thats what iam expecting ;-)): then go a step back- ping the site 1 router IP (192.168.1.253) ...go backwards and see where you get the reply.

if you can ping ASA inside IP: enable debug icmp trace and try to ping public IPs. you should see on ASA the logs.

hth

ASA5510 - 1 site can access internet , 2 others cannot , why?

cadek1fraen — Wed, 13 Jul 2011 18:49:33 GMT

I've made some changes and some progress

right now a PC 3.100 can ping firewall's PROTECTED interface (inside) , so 3.100 -> 1.1 works , now I even get replies back for the pings (not just see echo requests on firewall).

but still no internet traffic

when the same PC 3.100 tries going to 4.2.2.2 I see 0 traffic on the fw and of course it does not work.

I want to call cisco and use my smartnet, but I fear they'll just tell me to call the ISP who in turn blames everything on the firewall , oh the joys of IT!

ASA5510 - 1 site can access internet , 2 others cannot , why?

mvsheik123 — Wed, 13 Jul 2011 19:00:21 GMT

Hi,

Please post the current config from ASA. Als, do you see any o/p on ASA for 'debug icmp trace' (I guees you can give options 128 or 255 - don't recall exactly) while trying to ping 4.2.2.1 or 4.2.2.2 etc from remote site?

Thx

ASA5510 - 1 site can access internet , 2 others cannot , why?

varrao — Wed, 13 Jul 2011 19:02:57 GMT

Hi Martin,

Do you have any route on the routers that says, all the internet traffic request coming from the internal subnets needs to be routed to ASA inside interface, plz check that.

Moreover, yes I would suggest opening a TAC case for it, and plz be rest assured, we are not an organization who would just shrug off our responsibilities, I assure you we would definitely assist you in resolving the issue, if the issue is not on the firewall, we woudl definitely let you know how to troubleshoot it. You can open a TAC case with me as well, my shift timings are 01:30 am - 10:30 PM EDT, actually i work in Brussels timezone. I would love to assist you with this issue. Cisco has always been a customer centric organization.

Thanks,

Varun

ASA5510 - 1 site can access internet , 2 others cannot , why?

cadek1fraen — Wed, 13 Jul 2011 19:58:35 GMT

that is great to hear Varun. Thank you for the offer and I definitely wasn't implying that cisco would leave us hanging, but if it's a routing problem then it enters that gray territory of whose problem it is

The remote routers utilize BGP so it's pretty confusing as there are BGP and connected routes showing, but as far as I can tell they have a default route 0.0.0.0 0.0.0.0 their_next_hop_public_ip so everything is sent to the cloud in the diagram above. For internal traffic this works and arrives in its correct place, for internet traffic not so much.

The more I think about it I think it's time to bug the ISP again.

ASA5510 - 1 site can access internet , 2 others cannot , why?

mvsheik123 — Thu, 14 Jul 2011 02:07:05 GMT

Hi,

It definitely appears to be routing issue. Internal network reachability works fine as the network being learned via BGP/any other routing protocol in use. The default route- when the packet reaches the 'next hop public ip', I believe it is dropping as the next hop is carrier cloud and carrier peer unable to send it to main location (again this is MPLS and all I have is theoritical knowledge in MPLS ;-)). Although it sounds against the basic routing policy, as the network is transperant ,did you try to add the the default route netxt hop on remote location router as Main location router or ASA IP? Can you test it?

Also, if you can post current config from ASA, remote end router and main router any MPLS experts may be able to help you.

Thx