ASA 5505 - *ALL* Outside Traffic In

[jdlampard]

At a branch MPLS site I have an ASA 5505 (Security Plus license and provides DHCP) with about a dozen devices behind it (PCs, phones, and a network printer).  The MPLS is the only connection outside of the office until it is replaced with a dedicated Internet and site-to-site VPN connection in a couple months.

I need all traffic from the main office to flow through the ASA (without statics) to the branch systems and I don't want outbound traffic translated.

I removed the NAT and global and thought/hoped simply changing vlan2 (outside) security-level to 100 would complete the objectives.  No luck.  I added same-security-traffic permit inter-interface - again, no luck.

Is there a way to achieve the goal other than using firewall transparent (it's remote and from what I understand once changed I'll lose access to it and it would have to be changed back when MPLS is replaced with Internet).

The only option I see is to set up site-to-site VPN across MPLS.  Hopefully I'm missing something and there's a better/easy way.

Thanks for your help.

Regards,

JD

[jdlampard]

Error Message Decoder (support tool on site) didn't help with 106001 - I understand the problem but don't know how to fix it.  The tool suggested nothing needs to be done which is obviously wrong.

[Jennifer Halim]

Depending on when the log error is, it is seeing traffic from the HQ (assuming 10.99.21.21 is your HQ host) towards your print server (10.30.20.20) on the inside interface, which is wrong. If traffic is coming from HQ, it should be seen on the outside interface.

So, yes, definitely a routing issue. How does traffic coming from outside appears on the inside interface? Traffic from the HQ should appear on the outside interface of the ASA, right?

Defintely not configuration issue on the ASA. You would need to check hop by hop and make sure that the ASA has been correctly connected, ie: correct VLAN on the switch, etc.

You can run packet capture on the ASA itself, and that can be performed remotely.

On the ASA, assuming that traffic is coming from 10.99.21.0/24 subnet towards 10.30.20.0/24 subnet:

access-list cap permit ip 10.99.21.0 255.255.255.0 10.30.20.0 255.255.255.0

access-list cap permit ip 10.30.20.0 255.255.255.0 10.99.21.0 255.255.255.0

cap cap-out access-list cap interface outside

cap cap-in access-list cap interface inside

Then send the traffic across, then share the output of the packet capture:

show cap cap-out

show cap cap-in

[jdlampard]

Set up capture...

access-list cap line 1 extended permit ip 10.30.20.0 255.255.255.0 10.17.21.0 255.255.255.0 (hitcnt=0) 0xf179b88e
access-list cap line 2 extended permit ip 10.17.21.0 255.255.255.0 10.30.20.0 255.255.255.0 (hitcnt=0) 0x088dd972
access-list cap line 3 extended permit icmp any any (hitcnt=8) 0xc6bea65a

Note line 3 has hits and they will be shown later.

capture cap-out type raw-data access-list cap interface outside [Capturing - 920 bytes]
capture cap-in type raw-data access-list cap interface inside [Capturing - 0 bytes]

cap-out...

8 packets captured

   1: 06:54:49.803821 802.1Q vlan#2 P0 10.99.14.2 > 10.30.8.2: icmp: echo request
   2: 06:54:49.804401 802.1Q vlan#2 P0 10.30.8.2 > 10.99.14.2: icmp: echo reply
   3: 07:07:19.468985 802.1Q vlan#2 P0 10.99.21.11 > 10.30.8.2: icmp: echo request
   4: 07:07:19.469427 802.1Q vlan#2 P0 10.30.8.2 > 10.99.21.11: icmp: echo reply
   5: 07:07:19.496266 802.1Q vlan#2 P0 10.99.21.11 > 10.30.8.2: icmp: echo request
   6: 07:07:19.496556 802.1Q vlan#2 P0 10.30.8.2 > 10.99.21.11: icmp: echo reply
   7: 07:07:19.523257 802.1Q vlan#2 P0 10.99.21.11 > 10.30.8.2: icmp: echo request
   8: 07:07:19.523547 802.1Q vlan#2 P0 10.30.8.2 > 10.99.21.11: icmp: echo reply
8 packets shown

cap-in...

0 packet captured

0 packet shown

Some running-config...

...

interface Vlan1
nameif inside
security-level 100
ip address 10.30.20.2 255.255.255.0
!
interface Vlan2
nameif outside
security-level 100
ip address 10.30.8.2 255.255.255.0
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
shutdown
!
interface Ethernet0/3
shutdown
!
interface Ethernet0/4
shutdown
!
interface Ethernet0/5
shutdown
!
interface Ethernet0/6
shutdown
!
interface Ethernet0/7
shutdown
...

same-security-traffic permit inter-interface

...

nat (inside) 0 access-list nonat

access-group nonat in interface inside

route outside 0.0.0.0 0.0.0.0 10.30.8.1 1

..

ssh 0.0.0.0 0.0.0.0 outside

There is only one physical and logical path in/out of Remote and that is through ethernet0/0 (outside, vlan2).  I am obviously SSHed to this interface to manage the device and can ping the firewall from various MPLS sites (10.99.0.0 is HQ - 14.2 is a firewall and 21.11 is a server) which appears in the capture.  What doesn't sppear in the capture is that I can't ping anything behind the 5505 at Remote.

The default route on 5505 is 10.30.8.1 - the provider-managed MPLS router.

This isn't a routing issue - I can get to the firewall and *everything* worked with an unmanaged switch before replacing it with the firewall.

Now for some more supporting info, this is from the 5505 to the printer server...

ping 10.30.20.20
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.30.20.20, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

sh route

...

Gateway of last resort is 10.30.8.1 to network 0.0.0.0

C    10.30.20.0 255.255.255.0 is directly connected, inside
C    10.30.8.0 255.255.255.0 is directly connected, outside
S*   0.0.0.0 0.0.0.0 [1/0] via 10.30.8.1, outside

Definately something with the firewall.  Again, if I need to do site-to-site VPN or firewall transparent that so be it, although the VPN seems like a heavy-handed band-aid and firewall transparent will be a huge nuisance.  Certainly there is a non-VPN, routed-mode solution.

One last look in the log praying for something of value and the best I could come up with is...

Jan 15 2011 07:42:39: %ASA-6-302016: Teardown UDP connection 2917 for outside:139.78.133.139/123 to identity:10.30.8.2/65535 duration 0:02:01 bytes 48

The firewall has NTP configured sourced from the Internet and it too works (although this is outbound which is not my problem but it highlights that from the firewall across MPLS and then to the Internet from 10.99 HQ site works).

I have my log set as follows...

logging enable
logging timestamp
logging buffered informational

I appreciate you working with me on this!

[Jennifer Halim]

Your capture result is showing that you are actually pinging the ASA outside interface (10.30.8.2), so that would not help us in identifying the issue.

Pls kindly test the following:

1) Clear the capture - clear cap cap-out && clear cap cap-in

2) Ping from HQ towards hosts in 10.30.20.0/24 network, eg: try to ping the print server and your XP machine

3) Browse to your print server on port 80 as tested earlier.

4) Then grab the output of "show cap cap-out" and "show cap cap-in"

I fail to understand how VPN will change anything base on the description provided so far.

[jdlampard]

Cleared the captures and did the following:

1) Ping 5505 (10.30.8.2 - outside interface) from HQ host 10.99.21.11 - replies

2) Ping print server at Branch (10.30.20.20) from HQ host 10.99.21.11 - times out

3) Traceroute to 5505 (10.30.8.2 - outside interface) from HQ host 10.99.21.11 - completes

4) Traceroute to print server at Branch (10.30.20.20) from HQ host 10.99.21.11 - times out beginning at 5505

5) Telnet to print server port 9100 at Branch (10.30.20.20) from HQ host 10.99.21.11 - can't open connection

C:\Documents and Settings\jlampard>ping 10.30.8.2

Pinging 10.30.8.2 with 32 bytes of data:

Reply from 10.30.8.2: bytes=32 time=26ms TTL=248
Reply from 10.30.8.2: bytes=32 time=26ms TTL=248
Reply from 10.30.8.2: bytes=32 time=26ms TTL=248
Reply from 10.30.8.2: bytes=32 time=26ms TTL=248

Ping statistics for 10.30.8.2:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 26ms, Maximum = 26ms, Average = 26ms

C:\Documents and Settings\jlampard>ping 10.30.20.20

Pinging 10.30.20.20 with 32 bytes of data:

Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 10.30.20.20:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

C:\Documents and Settings\jlampard>tracert -d 10.30.8.2

Tracing route to 10.30.8.2 over a maximum of 30 hops

  1     1 ms    <1 ms    <1 ms  10.99.14.3
  2     1 ms     1 ms     2 ms  12.84.226.113
  3    31 ms    28 ms    28 ms  12.122.100.202
  4    38 ms    28 ms    28 ms  12.123.18.250
  5    28 ms    28 ms    28 ms  12.122.31.90
  6    22 ms    22 ms    22 ms  12.84.171.173
  7    26 ms    26 ms    26 ms  12.84.171.174
  8    28 ms    27 ms    27 ms  10.30.8.2

Trace complete.

C:\Documents and Settings\jlampard>tracert -d 10.30.20.20

Tracing route to 10.30.20.20 over a maximum of 30 hops

  1     1 ms    <1 ms    <1 ms  10.99.14.3
  2     1 ms     1 ms     1 ms  12.84.226.113
  3    29 ms    28 ms    28 ms  12.122.100.202
  4    29 ms    28 ms    29 ms  12.123.18.250
  5    28 ms    28 ms    27 ms  12.122.31.90
  6    22 ms    22 ms    22 ms  12.84.171.173
  7    26 ms    26 ms    26 ms  12.84.171.174
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
10     *        *        *     Request timed out.
11     *        *        *     Request timed out.
12     *        *        *     Request timed out.
13     *        *        *     Request timed out.
14     *        *        *     Request timed out.
15     *        *        *     ^C

Note in the two traceroutes above hops seven are the ISP-managed MPLS router - all traffic is taking the same path to get to this point - these results show the breakdown is at the firewall.

sh cap cap-out

14 packets captured

   1: 16:44:17.665050 802.1Q vlan#2 P0 10.99.21.11 > 10.30.8.2: icmp: echo request
   2: 16:44:17.665462 802.1Q vlan#2 P0 10.30.8.2 > 10.99.21.11: icmp: echo reply
   3: 16:44:18.669887 802.1Q vlan#2 P0 10.99.21.11 > 10.30.8.2: icmp: echo request
   4: 16:44:18.670222 802.1Q vlan#2 P0 10.30.8.2 > 10.99.21.11: icmp: echo reply
   5: 16:44:19.670070 802.1Q vlan#2 P0 10.99.21.11 > 10.30.8.2: icmp: echo request
   6: 16:44:19.670451 802.1Q vlan#2 P0 10.30.8.2 > 10.99.21.11: icmp: echo reply
   7: 16:44:20.669795 802.1Q vlan#2 P0 10.99.21.11 > 10.30.8.2: icmp: echo request
   8: 16:44:20.670161 802.1Q vlan#2 P0 10.30.8.2 > 10.99.21.11: icmp: echo reply
   9: 16:45:00.982538 802.1Q vlan#2 P0 10.99.21.11 > 10.30.8.2: icmp: echo request
  10: 16:45:00.982966 802.1Q vlan#2 P0 10.30.8.2 > 10.99.21.11: icmp: echo reply
  11: 16:45:01.010207 802.1Q vlan#2 P0 10.99.21.11 > 10.30.8.2: icmp: echo request
  12: 16:45:01.010512 802.1Q vlan#2 P0 10.30.8.2 > 10.99.21.11: icmp: echo reply
  13: 16:45:01.037595 802.1Q vlan#2 P0 10.99.21.11 > 10.30.8.2: icmp: echo request
  14: 16:45:01.037946 802.1Q vlan#2 P0 10.30.8.2 > 10.99.21.11: icmp: echo reply
14 packets shown

sh cap cap-in

0 packet captured

0 packet shown

Complete log...

Jan 15 2011 16:43:51: %ASA-5-111008: User 'enable_15' executed the 'clear capture cap-out' command.
Jan 15 2011 16:43:54: %ASA-5-111008: User 'enable_15' executed the 'clear capture cap-in' command.
Jan 15 2011 16:44:17: %ASA-6-302020: Built inbound ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:44:17: %ASA-6-302021: Teardown ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:44:18: %ASA-6-302020: Built inbound ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:44:18: %ASA-6-302021: Teardown ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:44:19: %ASA-6-302020: Built inbound ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:44:19: %ASA-6-302021: Teardown ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:44:20: %ASA-6-302020: Built inbound ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:44:20: %ASA-6-302021: Teardown ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:44:24: %ASA-6-302010: 1 in use, 132 most used
Jan 15 2011 16:45:00: %ASA-6-302020: Built inbound ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:45:00: %ASA-6-302021: Teardown ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:45:01: %ASA-6-302020: Built inbound ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:45:01: %ASA-6-302021: Teardown ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:45:01: %ASA-6-302020: Built inbound ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:45:01: %ASA-6-302021: Teardown ICMP connection for faddr 10.99.21.11/512 gaddr 10.30.8.2/0 laddr 10.30.8.2/0
Jan 15 2011 16:46:54: %ASA-6-302015: Built outbound UDP connection 3025 for outside:139.78.133.139/123 (139.78.133.139/123) to identity:10.30.8.2/65535 (10.30.8.2/65535)

How would a VPN help?  Well, it would bypass the 5505's outside interface.

[Jennifer Halim]

OK, the reason why it is not working towards the 10.30.20.0/24 subnet is because your MPLS provider is not routing the 10.30.20.0/24 subnet towards the ASA outside interface (10.30.8.2). This is clearly seen on your traceroute towards 10.30.20.20, as it stops at the MPLS provider ip address instead of the ASA outside interface. Further supporting evidence is on the packet capture that you have taken on the outside interface. Traffic towards the 10.30.20.20 is not even seen on the ASA outside interface. If it is blocked by the ASA, you would see the packet on the outside interface, however, you won't see it leave the inside interface. From the packet capture output, we don't even see that packet destined to 10.30.20.20 arrives on the ASA outside interface.

This confirms that the MPLS provider does not route traffic towards 10.30.20.0/24 towards the ASA outside interface.

So to conclude, it is not policy, or NAT configuration error on the ASA.

But you are absolutely correct, there are 2 ways we can fix this issue:

1) Request the MPLS provider to route the 10.30.20.0/24 towards the ASA outside interface (10.30.8.2)

OR/

2) Option number 1 is the easiest way, as all the MPLS provider needs to do is to add a route for 10.30.20.0/24 pointing towards the ASA outside interface. However, if this is too difficult (liasing with MPLS provider, etc), then you are absolutely correct. We can configure site-to-site VPN between the 2 sites, and as they will be encrypted into the ASA outside interface ip address, as from the MPLS provider point of view, they will only need to route towards the ASA outside ip address (and it is confirmed that it's already working just fine).

Hope that makes sense now.

[jdlampard]

No, again, this all worked before putting the firewall in.

In a perfect world, supposing the brainiest people did a brand new setup along the lines of what I'm trying to accomplish, how would the ASA be configured to pass traffic in and out without any filtering or translations?  A remote host can address a Branch host by its *real* address and connect and traffic leaving the branch arriving at HQ has the source host's *real* (not NATed) address as if it were simply passing through a Layer 2 switch.

All I did was take an unmanaged switch out and put the 5505 in place.  Things worked with the unmanaged switch and with the 5505 they don't.

Thanks,

Jonathan

[Jennifer Halim]

"Unmanaged switch" seems to me that you don't have L3 VLAN interface configured, and all ports are in the same VLAN. That also means that they were in the same subnet as the MPLS provider gateway, and as per your current setup, all the PC would be in the 10.30.8.0/24 subnet.

Please correct me if I am wrong.

When you have the ASA in place in routed mode, you have 2 interfaces, ie: outside - 10.30.8.0/24 and inside - 10.30.20.0/24.

So if you don't want to NAT your inside hosts, the MPLS provider needs to know how to route to 10.30.20.0/24, ie via 10.30.8.2 (which is confirmed that they know how to route to that subnet since ping and traceroute is successful).

On the ASA itself, if you do not want filtering policy and NAT translation, what I have advised earlier is all you need:

1) Security level needs to be the same on both vlan 1 and vlan 2.

2) Configure: same-security-traffic permit inter-interface

3) Remove all the NAT and global statement as well as the static statements.

4) Configure: no nat-control

5) Remove access-list applied to any of the interfaces (especially the outside interface).

I have worked with this setup plenty of times and it works just fine.

[Note to other forum community] Pls feel free to comment on this post..

[jdlampard]

Jennifer:

Thanks for taking the time to troubleshoot; I really appreciate your effort.

I have found the solutions...

interface Vlan1

...

security-level 100

...

interface Vlan2

...

security-level 100

...

same-security-traffic permit inter-interface

...

static (inside,outside) 10.30.20.0 10.30.20.0 netmask 255.255.255.0

static (outside,inside) 10.30.8.0 10.30.8.0 netmask 255.255.255.0

static (outside,inside) 10.99.21.0 10.99.21.0 netmask 255.255.255.0

...[additional statics as necessary for your envrionment - I have more than listed]...

* No nat statements

* No global statements

* No access-lists

In my network 10.30.8.0 is the outside on this ASA and 10.30.20.0 is the inside, 10.99.21.0 is a remote MPLS site.

The  configuration has been tested and passes traffic in and out of this  site without filtering and allows hosts on both sides to address each  other with their real addresses.  Furthermore, this proves there was no routing problem as suggested and as I explained.